Cross validation for ctsem fits: K fold by refitting, or PSIS for Laplace fits
Source:R/ctLOO.R
ctLOO.RdEstimates out-of-sample predictive performance, by refitting the model with each fold of the data held out in turn or, for a julia Laplace fit, by Pareto-smoothed importance sampling without refitting.
Usage
ctLOO(
fit,
folds = 10,
cores = getOption("mc.cores", 2L),
parallelFolds = FALSE,
tol = NULL,
subjectwise = ifelse(length(unique(.ctFitRowSubject(fit))) >= folds, TRUE, FALSE),
keepfirstobs = FALSE,
leaveOutN = NA,
refit = TRUE,
casewiseApproximation = FALSE,
method = c("kfold", "psis"),
nsamples = 1000
)Arguments
- fit
A fit from
ctFit, on either backend.- folds
Number of cross validation splits to use – 10 folds implies that the model is re-fit 10 times, each time to a data set with 1/10 of the observations randomly removed.
- cores
Number of processor cores to use.
- parallelFolds
compute folds in parallel or use cores to finish single folds faster. parallelFolds will use folds times as much memory.
- tol
objective tolerance for each fold's refit. On a stan fit it defaults to 1e-5. On a julia fit a fold by default stops once a step is predicted to gain less than 1e-4 in log likelihood, which keeps each held-out subject's error to about 0.014 times the square root of its influence (see
casewiseApproximation); an objective tolerance stops it while the gradient can still be large, and at 1e-5 moved one 40-subject model's out-of-sample log likelihood by 1 in total.- subjectwise
drop random subjects instead of data rows? With
method = 'psis'this chooses the level instead – leave one subject (unit) out whenTRUE, the default there, and leave one row out whenFALSE.- keepfirstobs
do not drop first observation (more stable estimates)
- leaveOutN
if a positive integer is given, the folds argument is ignored and instead the folds are calculated by leaving out every Nth row from the data when fitting. Leaving 2 out would result in 3 folds (starting at rows 1,2,3), each containing one third of the data.
- refit
if FALSE, do not optimise parameters for the new data set, just compute the likelihoods etc from the original parameters
- casewiseApproximation
TRUEreplaces each fold's refit by one Newton step from the full estimate: the held-out data's score times the fit's covariance. Much faster than refitting, and close to it when no fold moves the estimate far. On a julia fit each fold's influence, the step's squared length in the covariance's metric, is returned infoldinfluence, and a number instead ofTRUEsteps only the folds whose influence is below it and refits the rest (see Details). Needs an optimized fit with a covariance; on a stan fit, subject folds andTRUEorFALSEonly.- method
'kfold'(the default) refits the model once per fold.'psis'is refit-free Pareto-smoothed importance sampling leave-one-out forbackend = 'julia'fits (see Details); it needs the loo package. The K-fold argumentsfolds,parallelFolds,tol,keepfirstobs,leaveOutN,refitandcasewiseApproximationmean nothing to it and are refused if given.- nsamples
Number of importance sampling draws for
method = 'psis'. A sampled fit's posterior draws are used instead: all of them, or thinned tonsampleswhen it is given. Fewer draws estimate the Pareto k of a heavy-tailed correction too low: on one fit with an outlying subject the proposal's k read 0.54 at 1000 draws and 0.83 at 4000.
Value
For method = 'kfold', a list with foldrows,
foldpars, insampleLogLikRow, LogLikRowFolds,
outsampleLogLikRow, insampleLogLik, outsampleLogLik
and entropy and standard deviation summaries, and processes,
whether the folds ran in worker processes. When a julia fit's folds are
refit, foldoptim has a row per fold: method ('refit'
or 'step'), and for a refit the optimiser's iterations,
gradients, converged, gain (what it was predicted
still to gain, in log likelihood, when it stopped) and seconds; a
warning counts the refits that stopped without converging. With
casewiseApproximation, foldinfluence and
foldstepped give each fold's influence and whether it was stepped
rather than refit. scoring is added
whenever the folds were refit, naming what was scored and that they are
uncertified (see Details); an intoverpop = 'laplace' fit also adds
insampleLogLikSubject and outsampleLogLikSubject. For
method = 'psis', an object of class
ctLOOpsis: elpd_loo, se_elpd_loo, p_loo,
looic, pointwise (a data frame of elpd_loo,
lpd, p_loo and pareto_k per unit or row, and at
the unit level refined),
n_high_k, ndraws, ndropped, proposal_k (the
Pareto k of the correction from proposal to posterior: one value at the
unit level, NA for a sampled fit, one per unit at the row level),
level and note.
Details
Works for backend='julia' fits as well as stan ones. There,
a held-out row is withheld by setting its manifest observations to missing
and re-preparing, which is a row the filter propagates through without an
update and without a likelihood contribution – the same thing
standata$dokalmanrows does for the stan backend. For julia fits
parallelFolds = TRUE runs up to cores folds at once, each in a
background R session with a one-thread Julia, started and compiled for the
model before the first fold (it needs the future package; the
sessions persist for the next call, and ctJuliaKill('workers')
releases them). Without it the folds run one at a time on cores
threads. Folds are separate fits, so they gain more from processes than
from threads, except where each fold takes only seconds and starting the
sessions costs more than it saves. $processes on the result says
which ran.
Be aware that each fold is an independent re-optimisation, and neither backend's optimizer restarts from a flat direction. On a weakly identified model, or one where withholding a fold leaves a parameter poorly determined, two folds – or two backends – can converge to noticeably different raw parameters for a similar likelihood.
K fold on an intoverpop = 'laplace' fit. The random effects
are integrated out, so what is scored depends on the fold type. With row
folds, a held-out row is scored with each subject's random effects at their
mode given the fold's training rows, at the fold's parameters – the
held-out row never informs the mode it is scored at. With subject folds a
held-out subject's rows are not scored at all: the out-of-sample quantity is
the subject's Laplace marginal likelihood at the fold's parameters,
returned as outsampleLogLikSubject, and outsampleLogLikRow is
NA because the marginal does not decompose into rows. In a multilevel
model a subject's marginal is conditional on the rest of its outermost group,
as the difference of that group's marginal with and without the held-out
subjects; several subjects of one group held out together share that joint
value evenly. insampleLogLik is correspondingly the Laplace marginal at
the estimate for subject folds, and the sum of the rows at the full-data
modes for row folds.
casewiseApproximation. Each fold's refit is replaced by one
Newton step from the full estimate, est - cov %*% s, where
s is the score of the held-out data at the estimate: each unit the
fold empties (a subject, or under the Laplace layer an outermost group)
contributes its likelihood score, and a unit it leaves partly observed – a
row fold, or a subject inside a group – the difference between its score on
the full and on the training data, at one gradient of the training objective
per fold. The prior stays, as it does in a refit. This is the one-step
(infinitesimal jackknife) approximation to the refit; its error grows faster
than the step. A fold's influence t(s) %*% cov %*% s measures the step in
posterior standard deviations squared, and a number given as
casewiseApproximation refits exactly every fold whose influence is at
or above it, so the approximation is used only where the step is short.
foldstepped says which folds were stepped. On a julia fit a stepped
fold scores its held-out rows only, together with every other stepped fold
in one pass over the data, so its column of LogLikRowFolds is
NA elsewhere: the training rows at each step would cost a pass a
fold, which for leave-one-subject-out is the whole cost.
method = 'psis'. Pareto-smoothed importance sampling
(Vehtari, Gelman and Gabry, 2017), without refitting. It estimates the
Bayesian leave-one-out predictive density – a unit's likelihood averaged
over the posterior without it – where K fold scores each fold at its refit
estimate, so the two differ by more than Monte Carlo error, most for a unit
that moves the estimate. Two levels:
subjectwise = TRUE: leave one unit out, where a unit is the outermost level of random effect – a subject in a one-level model, a group in a multilevel one, since subjects sharing a group effect are not independent given the population parameters. A unit's term is its likelihood with its random effects integrated out: by the filter underintoverpop = 'augmented', by the Laplace approximation otherwise. On a sampled fit (optimize = FALSE, orctFitUncertainty(fit, 'sample')) the population parameters are its posterior draws. On an optimized fit they are drawn from its normal approximation (fit$estimate$cov, so the fit must carry one), widened by a factor of 1.5 and importance corrected to the posterior. A unit whose Pareto k is above 0.7 is estimated again withnsamplesmore draws, centred where leaving it out moves the estimate – the stepcasewiseApproximationtakes – pooled with draws around the estimate, and the estimate with the lower k is kept (pointwise$refined). That needs the fit's objective to be the posterior it describes, so it is not done for a fit sampled jointly over parameters and random effects (intoverpop = 'none'), whose unit terms are Laplace approximations to the sampled marginal. When the normal approximation itself misses the posterior (proposal_k, printed, above 0.7) every unit's estimate is unreliable, and a warning says to sample the fit and use its draws instead. No warning is not proof the approximation is good: k is estimated too low from few draws, so a value approaching 0.7, or a posterior you expect to be skewed (an outlying subject, a variance near zero, binary indicators), is worth rerunning with morensamplesor on a sampled fit.subjectwise = FALSE, on an optimizedintoverpop = 'laplace'fit only: leave one row out, conditional on the estimated population parameters and integrating over each unit's random effects, drawn from the Gaussian the Laplace approximation fits at their mode, widened by a factor of 1.5. This is pointwise leave-one-out of the filter's one-step-ahead terms – each row's density given the rows before it – not exact time series leave-one-out, and it does not propagate uncertainty in the population parameters.
Draws whose inner mode solve fails are dropped and counted. A Pareto k above 0.7 flags a unit or row whose estimate is unreliable; K fold is then the better tool for it.
Both methods cross-validate the Laplace objective. A Laplace fit's
estimate can be refitted with its units integrated (intoverpop =
'quadrature', in ctFit or ctFitUncertainty).
The folds are
refits of the Laplace objective, scored by it, and are not corrected – each
would need its own Hessian and quadrature gradient – so the in-sample side,
the folds' starting point and the PSIS proposal all use the Laplace optimum,
fit$laplace$correction$laplace_estimate, rather than the corrected
estimate. scoring says so when the fit was corrected.
A fold's estimate is never certified. Whatever the route, a refit
fold is optimised with optimcontrol$certify = FALSE: certifying and
correcting a fold the way the full fit is would cost each fold its own
Hessian, which is the expense K-fold exists to avoid. scoring says so
for any refit (refit = TRUE, the default, and not
casewiseApproximation), on both routes.