I just posted a new version of the function threshDIC that allows users to use the deviance information criterion (DIC) to compare Brownian, OU, and λ models for evolution of the liability on the tree in ancThresh. I implemented threshDIC primarily to compare alternative sequences for the threshold characters on the liability axis - but (theoretically) the same approach could be useful in choosing among alternative models for the liability.
DIC is similar to the better known AIC, except that it can be used for Bayesian approaches in which we are unable to maximize the likelihood and all we have is a sample from the posterior distribution obtained by MCMC. DIC estimates the effective parameterization of our model by computing the difference between the mean likelihoods evaluated for each of our samples from the posterior and the likelihood for our mean parameter values. The logic is that this difference will be increase with the effective number of parameters in the model (because when the number of parameters is large we tend to spend most of the time during MCMC far from the parameter values that maximize the likelihood).
This code is also in a new version of phytools here. (In fact, should you try to run it from source without updating phytools it will not work because it has an internal dependency only present in the last couple of non-CRAN versions.)