Write a folder of summary, diagnostic and interpretation output for a fit
Source:R/ctReport.R
ctReport.RdRuns the package's reporting and fit-checking functions over a fitted model and writes the results into one folder, with a markdown index that says what each file is and how to read it. Works for either backend; components a backend or a model does not support are recorded in the index as not produced, rather than stopping the run.
Arguments
- fit
A fit from
ctFit(classctStanFitorctJuliaFit).- folder
Directory to write into; created if it does not exist. Files from a previous
ctReportrun in the same folder are removed first, so the folder always matches its index.- components
Either one of
'quick'(nothing that needs generated data or further model evaluations),'default','all'(adds cross validation and LaTeX), or a character vector of component names:summary,parmatrices,identification,profile,discretepars,network,predictions,residuals,adequacy(ctFitAdequacy),scoretests(ctScoreTest, julia fits),covcheck,postpred,tipredeffects,crossval,equations.- nsamples
Draws used by the components that sample from the fit's uncertainty, and by data generation for
adequacy,covcheckandpostpred.- cores
Cores for the components that can use more than one (
discretepars,adequacy,scoretests,covcheck,postpred,crossval).- quantiles
Quantiles for the parameter matrices and the discrete parameter plots.
- times
Interval grid for
discretepars.'auto'uses zero to the 90th percentile of the observed per-subject time span.- timeinterval
Time interval at which the networks, the discrete-time parameter matrices and the covariate effects are reported.
- nsubjects
Number of subjects plotted by
predictions, chosen as those with the most observations.- lags
Lags for the lagged covariance check.
- folds
Folds for
crossval.- profilepoints
Grid points per parameter in the likelihood profile.
- verbose
Report each component as it runs.
Value
Invisibly, a list with the folder and, per component, whether it was written, skipped or failed. Called for the files it writes.
Details
The expensive parts are adequacy, covcheck and
postpred, which generate data from the fit (once, shared),
scoretests, which evaluates the model extended with new terms, and
crossval, which refits. components = 'quick' excludes them
all.
profile walks the log probability along each raw parameter, holding
the others at the estimate, over a grid measured in the standard error that
parameter's own curvature implies. It flags parameters whose slice peaks away
from the estimate (the optimizer did not arrive), whose surface is not the
shape a standard error assumes, and whose reported interval is much wider
than their own curvature (they sit on a ridge with other parameters). It is a
conditional slice, not a profile likelihood, which is why it is cheap enough
to run by default; identification names the ridge directions.
Checking a fit – which function answers what
- Can the data inform every parameter?
ctIdentifybefore fitting;fit$identifiabilityandctFitProfileafter.- Did the optimiser, or the sampler, get there?
print(fit)says whether it converged,summary(fit)adds R-hat and effective sample sizes for a sampled fit, andctTracePlotdraws the optimiser's trace or each chain's draws.- How far did the data move each parameter from its prior?
- Does data generated from the fit look like the data?
ctPostPredPlots, andctFitCheckCovfor the lagged covariances, split by a group or predictor if wanted.- Are the one-step-ahead errors what the model says they are?
ctFitAdequacytests them by family, allowing for the estimation;ctResidualsandctACFresidualsshow them.- What would relaxing the model change?
ctScoreTest, without refitting;ctModelProposalslists the changes andctModelImproverefits them.- Does a richer model fit better, or predict better?
anovafor nested models,ctLOOfor out-of-sample prediction.- How much do the julia engine's approximations cost?
ctLaplaceCheckfor the Laplace approximation of random effects,ctParticleLikfor the filter's Gaussian approximation of the states.- Do the intervals cover as often as they claim?
ctCoverageCheck, by simulation.
Examples
# \donttest{
f <- ctReport(ctstantestfit, folder = file.path(tempdir(), 'ctReport'),
components = c('summary', 'identification', 'profile'), profilepoints = 5)
list.files(f$folder)
# }