loo_compare {loo} | R Documentation

## Model comparison

### Description

Compare fitted models on LOO or WAIC.

### Usage

    loo_compare(x, ...)

    ## Default S3 method:
    loo_compare(x, ...)

    ## S3 method for class 'compare.loo'
    print(x, ..., digits = 1, simplify = TRUE)

### Arguments

`x` | An object of class `"loo"` or a list of such objects.
`...` | Additional objects of class `"loo"`.
`digits` | For the print method only, the number of digits to use when printing.
`simplify` | For the print method only, should only the essential columns of the summary matrix be printed? The entire matrix is always returned, but by default only the most important columns are printed.

### Details

When comparing two fitted models, we can estimate the difference in their expected predictive accuracy by the difference in `elpd_loo` or `elpd_waic` (or multiplied by -2, if desired, to be on the deviance scale).

When using `loo_compare()`, the returned matrix will have one row per model and several columns of estimates. The values in the `elpd_diff` and `se_diff` columns of the returned matrix are computed by making pairwise comparisons between each model and the model with the largest ELPD (the model in the first row). For this reason the `elpd_diff` column will always have the value `0` in the first row (i.e., the difference between the preferred model and itself) and negative values in subsequent rows for the remaining models.

To compute the standard error of the difference in ELPD — which should not be expected to equal the difference of the standard errors — we use a paired estimate to take advantage of the fact that the same set of N data points was used to fit both models. These calculations should be most useful when N is large, because then non-normality of the distribution is not such an issue when estimating the uncertainty in these sums. These standard errors, for all their flaws, should give a better sense of uncertainty than what is obtained using the current standard approach of comparing differences of deviances to a Chi-squared distribution, a practice derived for Gaussian linear models or asymptotically, and which only applies to nested models in any case.

### Value

A matrix with class `"compare.loo"` that has its own print method. See the Details section for more .

### References

Vehtari, A., Gelman, A., and Gabry, J. (2017a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4. ( journal, preprint arXiv:1507.04544).

Vehtari, A., Gelman, A., and Gabry, J. (2017b). Pareto smoothed importance sampling. arXiv preprint: http://arxiv.org/abs/1507.02646/

### Examples

    ## Not run:
    loo1 <- loo(log_lik1)
    loo2 <- loo(log_lik2)
    print(loo_compare(loo1, loo2), digits = 3)
    print(loo_compare(x = list(loo1, loo2)))

    waic1 <- waic(log_lik1)
    waic2 <- waic(log_lik2)
    loo_compare(waic1, waic2)

    ## End(Not run)

[Package loo version 2.1.0 Index]