3. Model Fit and Selection

Laud-Ibrahim Criteria

Laud-Ibrahim Criteria is another model comparison tool.

Consider \(M\) family of models, and \(\theta^{(m)}\) are the \(m\) model’s parameters. By Bayes theorem, the posterior of \(\theta^{(m)}\) is

\[ \pi(\theta^{(m)} | m, y) = \frac{f(y | \theta^{(m)},m) \pi(\theta^{(m)})}{\int f(y | \theta^{(m)},m) \pi(\theta^{(m)}) d \theta^{(m)}} \]

The posterior predictive distribution of \(Z\) is

\[ f(z|y,m) = \int f(z|\theta^{(m)},m) \pi(\theta^{(m)}|y,m)d\theta^{(m)} \]

So for a fixed \(m\), Laud-Ibrahim Criteria is

\[ L_m^2 = \text{E}(Z - y)'(Z-y) = \sum_{i=1}^n \left[(\text{E} Z_i - y_i)^2 + \text{Var}(Z_i)\right] \]

where \(Z \sim f(z|y,m)\).

\(\text{L}_m = \sqrt{\text{L}_m^2} \) is essentially a discrepancy measure, and is in the same units as the response \(y\). The model that minimizes \(\text{L}_m\) is preferred. On the next page, we can see an example of comparing models with Laud-Ibrahim Criteria in PyMC.

Modern approach

The Laud–Ibrahim criterion measures posterior predictive loss by combining the difference between observed and predicted values with the uncertainty in those predictions. Our understanding is that modern practice has moved towards using more visual checks posterior predictive checks (PPCs). We can generate posterior predictive samples using pm.sample_posterior_predictive() and compare their distributions with the observed data using ArviZ. ArviZ has functions like az.plot_ppc_dist() and az.plot_ppc_tstat().

When the goal is to compare models based on their expected predictive accuracy on new observations, we generally prefer PSIS-LOO CV, as mentioned in the U9L1 notes.