Improve a model step by step, from score tests and predictive checks
Source:R/ctModelImprove.R
ctModelImprove.RdStarting from a fitted model, repeatedly: screens it with
ctScoreTest; refits it with the most strongly indicated
relaxations, one at a time, warm started from the current estimates; keeps
the best of them if the refit's likelihood confirms it; and stops when no
relaxation is indicated and the model is adequate by
ctFitAdequacy. When nothing is indicated but the model is not
adequate, a slow process driving a latent
(ctModelSlowComponent), a cycle such as a daily rhythm
(ctModelPeriodicComponent), a process of its own for an item
(ctModelUniqueProcess), raising the order of a latent process
(ctModelHigherOrder) and any models given in
candidates are tried the same way, each ranked by the adequacy
check's evidence for it: a slow process for a latent whose updates stay
correlated over time gaps (the latent family), a cycle at the period
the periodic family points to, a process of its own for an item
whose errors stay correlated with its own (the residuals family).
Usage
ctModelImprove(
fit,
covariates = NULL,
relax = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed",
"level", "slow", "unique", "periodic", "order"),
changes = NULL,
candidates = NULL,
alpha = 0.05,
maxsteps = 10L,
ntry = 3L,
criterion = c("LR", "AIC", "BIC"),
adequacy = TRUE,
prune = TRUE,
close = NULL,
holdout = 0,
form = "auto",
periods = NULL,
nsamples = 100L,
projection = "auto",
cores = getOption("mc.cores", 2L),
checkpoint = NULL,
verbose = TRUE
)Arguments
- fit
A fit from
ctFitwithbackend = 'julia'.- covariates
Optional subject-level covariates the model may add, as for
ctScoreTest.- relax
Which relaxations may be made.
- changes
Optionally rows of
ctModelProposals: the first round refits with these, in the order given, instead of screening, and keeps the first that improves the fit bycriterion. Withmaxsteps = 1, one step of a search made by hand; later rounds screen as usual.- candidates
Optional list of alternative models (from
ctModel) to try when the screen is clean but the checks fail.- alpha
Level for the screen (Benjamini-Hochberg within each family), for the likelihood ratio decision, and for adequacy.
- maxsteps
Most refit rounds.
- ntry
Proposals from the score tests refitted in a round, at most: one at a time, the strongest first, keeping the first that improves the fit. Proposals from the adequacy check are tried the same way until one improves or none is left, since the check does not change until the model does. Pruning and a held-out confirmation refit this many side by side.
- criterion
'LR','AIC'or'BIC'.- adequacy
Use
ctFitAdequacyto decide when to stop.- prune
After the search, try dropping each kept change in turn and drop it if the likelihood does not need it given the others: a change kept early can be made redundant by one kept later.
- close
Optional root mean square residual correlation regarded as a close enough fit to stop at; see Details.
- holdout
Share of subjects, between 0 and 0.5, held out of the search to confirm its changes on.
- form
How covariates are screened, as for
ctScoreTest;'threshold'lets the search add a covariate as a step.- periods
Periods for the adequacy check's
periodicfamily, as forctFitAdequacy; by default it scans.- nsamples
Generated datasets for the adequacy checks.
- projection
How the adequacy checks remove the estimation, as for
ctFitAdequacy.- cores
Cores for fitting and testing. With two or more, once a refit has taken longer than five minutes, the refits of pruning and of a held-out confirmation run side by side in worker processes,
ntryat a time, as sampling chains do, each with an equal share ofcores: they are independent, one fit uses its cores only partly, and a worker takes about half a minute to start.options(ctsem.improve.processes = FALSE)keeps them in this session.- checkpoint
Optional file the search saves its state to after every round and after pruning. Called again with the same file, the same fit and the same settings, it resumes after the last round saved; a file from a different search is refused. A long search on a cluster otherwise keeps nothing until it ends. With
verbose, the refit running is also saved every ten minutes beside it (<checkpoint>.refit.rds): the round, the change, the iteration and the raw parameters it had reached, and, where a change's starts are refitted side by side, the start.- verbose
Report progress: each round's steps, each refit's time, and while a refit runs, its iteration, objective and gradient every ten minutes.
Value
An object of class ctModelImprove: fit (the final
fit), history (every refit tried: the round, the relaxation, its
screening p-value, the change in log likelihood and parameters, the
criterion's verdict, and whether it was kept; a pruning row, family
'prune', says whether the change was dropped, and a 'rank' row
whether the lower rank was kept), screens (the score
tests of each round), adequacy (the final checks),
confirmation (with holdout: each kept change's likelihood
ratio test on the held-out subjects), and stopped (why the search
ended: 'adequate', no adequacy family detected misfit;
'close fit', those that did were shown within close;
'not adequate, no candidate improved' or 'not adequate,
nothing left to try', the checks still detect misfit; 'nothing
indicated', with adequacy = FALSE, the screen was clean;
'maxsteps'). A refit that failed, or ended below the model it
extends, is in history with a note saying so. With holdout,
the held-out subjects take no part in the search; each kept change is
tested once on them, at the end, at alpha. fit is the model with the confirmed
changes refitted to every subject, and selected the search's own
choice fitted to the subjects it searched on.
Details
The relaxations, in the language of the model:
- heterogeneity
indvarying = TRUEfor a parameter and theT0MEANSparameters, whose covariances with it the test took in; for a fixedMANIFESTMEANScell, a parameter there varying about the fixed value as its stated mean.- covariates
a covariate added as a time independent predictor, with effects on the parameters its tests single out.
- statedependence
a term
d * eta_k * eta_min a latent's drift, written intoCINTwithdaPARSparameter: a drift coefficient that varies with a state.- volatility
a latent's innovation sd scaled by
exp(v * eta_m): innovations whose size depends on a state.- fixed
a cell fixed at zero freed: a drift cross effect, a diffusion covariance, a cross loading.
- level
a parameter of the dynamics or error variances that depends on a subject's level (a random intercept) let vary over subjects, its random effect correlated with the intercepts (
ctScoreTest's level terms).- slow
a slow process driving a latent through its drift, its effect started at one and its persistence at a tenth of the latent's; where the model has a slow process for another latent, that process driving this one too is tried first.
- periodic
a cycle of the period the adequacy check points to, driving every latent with innovations of its own, each subject's phase set from the clock at its first row and free to deviate; the phase predictors are added to the data.
- unique
a process of its own, persistent, loading on one item; and, where the
pairsfamily finds two items' errors correlated within an occasion, a process the two share (ctModelUniqueProcess(shared = TRUE)): an error is meant to be random, and co-movement within an occasion that the latents' innovation covariances do not explain is read as a process faster than the occasions. Like the other structural changes it is tried once the score tests indicate nothing, ranked against them by the check's evidence, so that it does not absorb misfit that belongs to a cycle or to one item's own process.- errorcov
not by default: a measurement error covariance the score test flags is freed, for designs with a reason for correlated error, such as a method the items share, and no shared process is proposed.
- order
a latent process made second order.
On a reduced-rank level (poprank) a heterogeneity result for a
parameter that already varies through the shared dimensions gives it a
variance of its own (popunique); without reduced rank, a
significant dimension test makes the parameters its direction
loads on vary together. Relaxations the score tests cannot tell apart (the same
class) are not refitted in the same round.
Each random effect the search adds brings its correlations with the others,
and with many effects for few subjects the data no longer determine them.
When a kept change leaves the fit's curvature weak or negative along the
population covariance (fit$identifiability), the covariance is
refitted at one rank lower (poprank), starting from its leading
dimensions, and the lower rank kept unless the likelihood ratio test needs
the dropped dimension; it steps down until the fit is identified. Later
random effects then load on the shared dimensions, and later heterogeneity
results give own variances. Single-level models only.
When the search stops, each change kept is tried out again, by refitting
without it (prune), and dropped if the others make it redundant.
A relaxation is kept when its refit improves on the current model by
criterion: a likelihood ratio test at alpha divided by the
number of refits tried at that step ('LR'), or a lower AIC or BIC
(subjects as the sample size). The likelihood ratio test of a new random
effect against the chi-square is conservative, as its variance starts on a
boundary.
Error rates
The screen and the refit read the same data, so a
refit confirms most of what the screen proposes, and the search's error
rate is close to alpha per decision. Measured on simulated data (one
latent, 80 subjects, 20 irregular rows, 40 searches each): with nothing
missing, alpha = 0.05 kept something in 5 per cent of searches and
alpha = 0.01 in none; an omitted covariate shifting a random
intercept by half its sd was added in 75 and 55 per cent; an unrelated
covariate never. criterion = 'BIC' kept something in 8 per cent,
so at that size it is not the stricter choice. A drift that varied over
subjects was never added as such at that size; a random diffusion or
measurement error variance stood in for it in 10 per cent, since what the
data showed was subjects differing in variance. A search makes a decision
per family it screens, so its error rate grows with the families: on two
processes with three items each (60 subjects, about 48 rows each), where
the screen also tests state dependence and volatility in each process,
alpha = 0.05 kept something in 4 of 20 searches with nothing
missing (a volatility term twice, state dependence once, a random effect
once), and alpha = 0.01 is the stricter choice. With an omitted
cross effect, a process of an item's own and a daily cycle in the same
data, the search kept the cross effect in 19 of 20 searches, the cycle in
16 and the item's process in all 20.
The result is a model chosen by the data it is fitted to, and its
standard errors and tests do not account for that choice (MacCallum,
Roznowski & Necowitz, 1992). holdout sets aside a share of the
subjects: the search runs on the rest, and each change it keeps is then
tested on the held-out subjects alone, by refitting the final model without
it there; confirmation says which held, and the final model, refitted
to every subject, keeps only those. Every step, and every refit tried and
rejected, is in history.
With many observations a model can be rejected by the checks for
departures that are small on these summaries. With close, the search
also stops when every check that fails is a residual or within-row
correlation family
whose size is shown below close (ctFitAdequacy).
Each refit is judged by criterion; with 'LR', the likelihood
ratio against a chi-square on the parameters the change adds, at
alpha divided by the refits the round tries. That reference is
exact for none of the changes the search makes: a variance added at zero
sits on its boundary, where the chi-square is conservative, and a process
or cycle added brings parameters that mean nothing when its coupling is
zero, where it can err either way. Screening, rounds and pruning add
further choices the level does not count. The search is a way of
developing a model, and its p-values describe the steps it took rather than
testing the result. A refit that ends below the model it extends has not
reached an optimum, and is recorded as such.