pareto-k-diagnostic {loo}R Documentation

Diagnostics for Pareto smoothed importance sampling (PSIS)

Description

Print a diagnostic table summarizing the estimated Pareto shape parameters and PSIS effective sample sizes, find the indexes of observations for which the estimated Pareto shape parameter k is larger than some threshold value, or plot observation indexes vs. diagnostic estimates. The Details section below provides a brief overview of the diagnostics, but we recommend consulting Vehtari, Gelman, and Gabry (2017a, 2017b) for full details.

Usage

pareto_k_table(x)

pareto_k_ids(x, threshold = 0.5)

pareto_k_values(x)

psis_n_eff_values(x)

mcse_loo(x, threshold = 0.7)

## S3 method for class 'psis_loo'
plot(x, diagnostic = c("k", "n_eff"), ...,
  label_points = FALSE, main = "PSIS diagnostic plot")

## S3 method for class 'psis'
plot(x, diagnostic = c("k", "n_eff"), ...,
  label_points = FALSE, main = "PSIS diagnostic plot")

Arguments

x

An object created by loo or psis.

threshold

For pareto_k_ids, threshold is the minimum k value to flag (default is 0.5). For mcse_loo, if any k estimates are greater than threshold the MCSE estimate is returned as NA (default is 0.7).

diagnostic

For the plot method, which diagnostic should be plotted? The options are "k" for Pareto k estimates (the default) or "n_eff" for PSIS effective sample size estimates.

label_points, ...

For the plot method, if label_points is TRUE the observation numbers corresponding to any values of k greater than 0.5 will be displayed in the plot. Any arguments specified in ... will be passed to text and can be used to control the appearance of the labels.

main

For the plot method, a title for the plot.

Details

The reliability and approximate convergence rate of the PSIS-based estimates can be assessed using the estimates for the shape parameter k of the generalized Pareto distribution:

If the estimated tail shape parameter k exceeds 0.5, the user should be warned, although in practice we have observed good performance for values of k up to 0.7. (If k is greater than 0.5 then WAIC is also likely to fail, but WAIC lacks its own diagnostic.)

If using PSIS in the context of approximate LOO-CV, even if the PSIS estimate has a finite variance the user should consider sampling directly from p(θ^s | y_{-i}) for any problematic observations i, use K-fold cross-validation, or use a more robust model. Importance sampling is likely to work less well if the marginal posterior p(θ^s | y) and LOO posterior p(θ^s | y_{-i}) are much different, which is more likely to happen with a non-robust model and highly influential observations. A robust model may reduce the sensitivity to highly influential observations.

Effective sample size and error estimates

In the case that we obtain the samples from the proposal distribution via MCMC we can also compute estimates for the Monte Carlo error and the effective sample size for importance sampling, which are more accurate for PSIS than for IS and TIS (see Vehtari et al (2017b) for details). However, the PSIS effective sample size estimate will be over-optimistic when the estimate of k is greater than 0.7.

We can also compute estimates for the Monte Carlo error and the effective sample size for importance sampling. However, the PSIS effective sample size estimate will be over-optimistic when the estimate of k is greater than 0.7. In the case that we obtain the samples from the proposal distribution via MCMC, we need to take into account also the relative efficiency of MCMC sampling (see Vehtari et al (2017b) for details). Following the notation in Stan, the PSIS effective sample size is denoted here with n_{eff}, instead of S_{eff} used by Vehtari et al (2017b).

Value

pareto_k_table returns an object of class "pareto_k_table", which is a matrix with columns "Count", "Proportion", and "Min. n_eff", and has its own print method.

pareto_k_ids returns an integer vector indicating which observations have Pareto k estimates above threshold.

pareto_k_values returns a vector of the estimated Pareto k parameters.

psis_n_eff_values returns a vector of the estimated PSIS effective sample sizes.

mcse_loo returns the Monte Carlo standard error (MCSE) estimate for PSIS-LOO. MCSE will be NA if any Pareto k values are above threshold.

The plot method is called for its side effect and does not return anything. If x is the result of a call to loo or psis then plot(x, diagnostic) produces a plot of the estimates of the Pareto shape parameters (diagnostic = "k") or estimates of the PSIS effective sample sizes (diagnostic = "n_eff").

References

Vehtari, A., Gelman, A., and Gabry, J. (2017a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4. ( journal, preprint arXiv:1507.04544).

Vehtari, A., Gelman, A., and Gabry, J. (2017b). Pareto smoothed importance sampling. arXiv preprint: http://arxiv.org/abs/1507.02646/

See Also

psis for the implementation of the PSIS algorithm.


[Package loo version 2.1.0 Index]