Skip to contents

Estimates out-of-sample predictive performance, by refitting the model with each fold of the data held out in turn or, for a julia Laplace fit, by Pareto-smoothed importance sampling without refitting.

Usage

ctLOO(
  fit,
  folds = 10,
  cores = getOption("mc.cores", 2L),
  parallelFolds = FALSE,
  tol = NULL,
  subjectwise = ifelse(length(unique(.ctFitRowSubject(fit))) >= folds, TRUE, FALSE),
  keepfirstobs = FALSE,
  leaveOutN = NA,
  refit = TRUE,
  casewiseApproximation = FALSE,
  method = c("kfold", "psis"),
  nsamples = 1000
)

Arguments

fit

A fit from ctFit, on either backend.

folds

Number of cross validation splits to use – 10 folds implies that the model is re-fit 10 times, each time to a data set with 1/10 of the observations randomly removed.

cores

Number of processor cores to use.

parallelFolds

compute folds in parallel or use cores to finish single folds faster. parallelFolds will use folds times as much memory.

tol

objective tolerance for each fold's refit. On a stan fit it defaults to 1e-5. On a julia fit a fold by default stops once a step is predicted to gain less than 1e-4 in log likelihood, which keeps each held-out subject's error to about 0.014 times the square root of its influence (see casewiseApproximation); an objective tolerance stops it while the gradient can still be large, and at 1e-5 moved one 40-subject model's out-of-sample log likelihood by 1 in total.

subjectwise

drop random subjects instead of data rows? With method = 'psis' this chooses the level instead – leave one subject (unit) out when TRUE, the default there, and leave one row out when FALSE.

keepfirstobs

do not drop first observation (more stable estimates)

leaveOutN

if a positive integer is given, the folds argument is ignored and instead the folds are calculated by leaving out every Nth row from the data when fitting. Leaving 2 out would result in 3 folds (starting at rows 1,2,3), each containing one third of the data.

refit

if FALSE, do not optimise parameters for the new data set, just compute the likelihoods etc from the original parameters

casewiseApproximation

TRUE replaces each fold's refit by one Newton step from the full estimate: the held-out data's score times the fit's covariance. Much faster than refitting, and close to it when no fold moves the estimate far. On a julia fit each fold's influence, the step's squared length in the covariance's metric, is returned in foldinfluence, and a number instead of TRUE steps only the folds whose influence is below it and refits the rest (see Details). Needs an optimized fit with a covariance; on a stan fit, subject folds and TRUE or FALSE only.

method

'kfold' (the default) refits the model once per fold. 'psis' is refit-free Pareto-smoothed importance sampling leave-one-out for backend = 'julia' fits (see Details); it needs the loo package. The K-fold arguments folds, parallelFolds, tol, keepfirstobs, leaveOutN, refit and casewiseApproximation mean nothing to it and are refused if given.

nsamples

Number of importance sampling draws for method = 'psis'. A sampled fit's posterior draws are used instead: all of them, or thinned to nsamples when it is given. Fewer draws estimate the Pareto k of a heavy-tailed correction too low: on one fit with an outlying subject the proposal's k read 0.54 at 1000 draws and 0.83 at 4000.

Value

For method = 'kfold', a list with foldrows, foldpars, insampleLogLikRow, LogLikRowFolds, outsampleLogLikRow, insampleLogLik, outsampleLogLik and entropy and standard deviation summaries, and processes, whether the folds ran in worker processes. When a julia fit's folds are refit, foldoptim has a row per fold: method ('refit' or 'step'), and for a refit the optimiser's iterations, gradients, converged, gain (what it was predicted still to gain, in log likelihood, when it stopped) and seconds; a warning counts the refits that stopped without converging. With casewiseApproximation, foldinfluence and foldstepped give each fold's influence and whether it was stepped rather than refit. scoring is added whenever the folds were refit, naming what was scored and that they are uncertified (see Details); an intoverpop = 'laplace' fit also adds insampleLogLikSubject and outsampleLogLikSubject. For method = 'psis', an object of class ctLOOpsis: elpd_loo, se_elpd_loo, p_loo, looic, pointwise (a data frame of elpd_loo, lpd, p_loo and pareto_k per unit or row, and at the unit level refined), n_high_k, ndraws, ndropped, proposal_k (the Pareto k of the correction from proposal to posterior: one value at the unit level, NA for a sampled fit, one per unit at the row level), level and note.

Details

Works for backend='julia' fits as well as stan ones. There, a held-out row is withheld by setting its manifest observations to missing and re-preparing, which is a row the filter propagates through without an update and without a likelihood contribution – the same thing standata$dokalmanrows does for the stan backend. For julia fits parallelFolds = TRUE runs up to cores folds at once, each in a background R session with a one-thread Julia, started and compiled for the model before the first fold (it needs the future package; the sessions persist for the next call, and ctJuliaKill('workers') releases them). Without it the folds run one at a time on cores threads. Folds are separate fits, so they gain more from processes than from threads, except where each fold takes only seconds and starting the sessions costs more than it saves. $processes on the result says which ran.

Be aware that each fold is an independent re-optimisation, and neither backend's optimizer restarts from a flat direction. On a weakly identified model, or one where withholding a fold leaves a parameter poorly determined, two folds – or two backends – can converge to noticeably different raw parameters for a similar likelihood.

K fold on an intoverpop = 'laplace' fit. The random effects are integrated out, so what is scored depends on the fold type. With row folds, a held-out row is scored with each subject's random effects at their mode given the fold's training rows, at the fold's parameters – the held-out row never informs the mode it is scored at. With subject folds a held-out subject's rows are not scored at all: the out-of-sample quantity is the subject's Laplace marginal likelihood at the fold's parameters, returned as outsampleLogLikSubject, and outsampleLogLikRow is NA because the marginal does not decompose into rows. In a multilevel model a subject's marginal is conditional on the rest of its outermost group, as the difference of that group's marginal with and without the held-out subjects; several subjects of one group held out together share that joint value evenly. insampleLogLik is correspondingly the Laplace marginal at the estimate for subject folds, and the sum of the rows at the full-data modes for row folds.

casewiseApproximation. Each fold's refit is replaced by one Newton step from the full estimate, est - cov %*% s, where s is the score of the held-out data at the estimate: each unit the fold empties (a subject, or under the Laplace layer an outermost group) contributes its likelihood score, and a unit it leaves partly observed – a row fold, or a subject inside a group – the difference between its score on the full and on the training data, at one gradient of the training objective per fold. The prior stays, as it does in a refit. This is the one-step (infinitesimal jackknife) approximation to the refit; its error grows faster than the step. A fold's influence t(s) %*% cov %*% s measures the step in posterior standard deviations squared, and a number given as casewiseApproximation refits exactly every fold whose influence is at or above it, so the approximation is used only where the step is short. foldstepped says which folds were stepped. On a julia fit a stepped fold scores its held-out rows only, together with every other stepped fold in one pass over the data, so its column of LogLikRowFolds is NA elsewhere: the training rows at each step would cost a pass a fold, which for leave-one-subject-out is the whole cost.

method = 'psis'. Pareto-smoothed importance sampling (Vehtari, Gelman and Gabry, 2017), without refitting. It estimates the Bayesian leave-one-out predictive density – a unit's likelihood averaged over the posterior without it – where K fold scores each fold at its refit estimate, so the two differ by more than Monte Carlo error, most for a unit that moves the estimate. Two levels:

  • subjectwise = TRUE: leave one unit out, where a unit is the outermost level of random effect – a subject in a one-level model, a group in a multilevel one, since subjects sharing a group effect are not independent given the population parameters. A unit's term is its likelihood with its random effects integrated out: by the filter under intoverpop = 'augmented', by the Laplace approximation otherwise. On a sampled fit (optimize = FALSE, or ctFitUncertainty(fit, 'sample')) the population parameters are its posterior draws. On an optimized fit they are drawn from its normal approximation (fit$estimate$cov, so the fit must carry one), widened by a factor of 1.5 and importance corrected to the posterior. A unit whose Pareto k is above 0.7 is estimated again with nsamples more draws, centred where leaving it out moves the estimate – the step casewiseApproximation takes – pooled with draws around the estimate, and the estimate with the lower k is kept (pointwise$refined). That needs the fit's objective to be the posterior it describes, so it is not done for a fit sampled jointly over parameters and random effects (intoverpop = 'none'), whose unit terms are Laplace approximations to the sampled marginal. When the normal approximation itself misses the posterior (proposal_k, printed, above 0.7) every unit's estimate is unreliable, and a warning says to sample the fit and use its draws instead. No warning is not proof the approximation is good: k is estimated too low from few draws, so a value approaching 0.7, or a posterior you expect to be skewed (an outlying subject, a variance near zero, binary indicators), is worth rerunning with more nsamples or on a sampled fit.

  • subjectwise = FALSE, on an optimized intoverpop = 'laplace' fit only: leave one row out, conditional on the estimated population parameters and integrating over each unit's random effects, drawn from the Gaussian the Laplace approximation fits at their mode, widened by a factor of 1.5. This is pointwise leave-one-out of the filter's one-step-ahead terms – each row's density given the rows before it – not exact time series leave-one-out, and it does not propagate uncertainty in the population parameters.

Draws whose inner mode solve fails are dropped and counted. A Pareto k above 0.7 flags a unit or row whose estimate is unreliable; K fold is then the better tool for it.

Both methods cross-validate the Laplace objective. A Laplace fit's estimate can be refitted with its units integrated (intoverpop = 'quadrature', in ctFit or ctFitUncertainty). The folds are refits of the Laplace objective, scored by it, and are not corrected – each would need its own Hessian and quadrature gradient – so the in-sample side, the folds' starting point and the PSIS proposal all use the Laplace optimum, fit$laplace$correction$laplace_estimate, rather than the corrected estimate. scoring says so when the fit was corrected.

A fold's estimate is never certified. Whatever the route, a refit fold is optimised with optimcontrol$certify = FALSE: certifying and correcting a fold the way the full fit is would cost each fold its own Hessian, which is the expense K-fold exists to avoid. scoring says so for any refit (refit = TRUE, the default, and not casewiseApproximation), on both routes.

References

Vehtari, A., Gelman, A. and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27, 1413–1432.

Examples

# \donttest{
ctLOO(ctstantestfit)
# }