Skip to contents

Starting from a fitted model, repeatedly: screens it with ctScoreTest; refits it with the most strongly indicated relaxations, one at a time, warm started from the current estimates; keeps the best of them if the refit's likelihood confirms it; and stops when no relaxation is indicated and the model is adequate by ctFitAdequacy. When nothing is indicated but the model is not adequate, a slow process driving a latent (ctModelSlowComponent), a cycle such as a daily rhythm (ctModelPeriodicComponent), a process of its own for an item (ctModelUniqueProcess), raising the order of a latent process (ctModelHigherOrder) and any models given in candidates are tried the same way, each ranked by the adequacy check's evidence for it: a slow process for a latent whose updates stay correlated over time gaps (the latent family), a cycle at the period the periodic family points to, a process of its own for an item whose errors stay correlated with its own (the residuals family).

Usage

ctModelImprove(
  fit,
  covariates = NULL,
  relax = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed",
    "level", "slow", "unique", "periodic", "order"),
  changes = NULL,
  candidates = NULL,
  alpha = 0.05,
  maxsteps = 10L,
  ntry = 3L,
  criterion = c("LR", "AIC", "BIC"),
  adequacy = TRUE,
  prune = TRUE,
  close = NULL,
  holdout = 0,
  form = "auto",
  periods = NULL,
  nsamples = 100L,
  projection = "auto",
  cores = getOption("mc.cores", 2L),
  checkpoint = NULL,
  verbose = TRUE
)

Arguments

fit

A fit from ctFit with backend = 'julia'.

covariates

Optional subject-level covariates the model may add, as for ctScoreTest.

relax

Which relaxations may be made.

changes

Optionally rows of ctModelProposals: the first round refits with these, in the order given, instead of screening, and keeps the first that improves the fit by criterion. With maxsteps = 1, one step of a search made by hand; later rounds screen as usual.

candidates

Optional list of alternative models (from ctModel) to try when the screen is clean but the checks fail.

alpha

Level for the screen (Benjamini-Hochberg within each family), for the likelihood ratio decision, and for adequacy.

maxsteps

Most refit rounds.

ntry

Proposals from the score tests refitted in a round, at most: one at a time, the strongest first, keeping the first that improves the fit. Proposals from the adequacy check are tried the same way until one improves or none is left, since the check does not change until the model does. Pruning and a held-out confirmation refit this many side by side.

criterion

'LR', 'AIC' or 'BIC'.

adequacy

Use ctFitAdequacy to decide when to stop.

prune

After the search, try dropping each kept change in turn and drop it if the likelihood does not need it given the others: a change kept early can be made redundant by one kept later.

close

Optional root mean square residual correlation regarded as a close enough fit to stop at; see Details.

holdout

Share of subjects, between 0 and 0.5, held out of the search to confirm its changes on.

form

How covariates are screened, as for ctScoreTest; 'threshold' lets the search add a covariate as a step.

periods

Periods for the adequacy check's periodic family, as for ctFitAdequacy; by default it scans.

nsamples

Generated datasets for the adequacy checks.

projection

How the adequacy checks remove the estimation, as for ctFitAdequacy.

cores

Cores for fitting and testing. With two or more, once a refit has taken longer than five minutes, the refits of pruning and of a held-out confirmation run side by side in worker processes, ntry at a time, as sampling chains do, each with an equal share of cores: they are independent, one fit uses its cores only partly, and a worker takes about half a minute to start. options(ctsem.improve.processes = FALSE) keeps them in this session.

checkpoint

Optional file the search saves its state to after every round and after pruning. Called again with the same file, the same fit and the same settings, it resumes after the last round saved; a file from a different search is refused. A long search on a cluster otherwise keeps nothing until it ends. With verbose, the refit running is also saved every ten minutes beside it (<checkpoint>.refit.rds): the round, the change, the iteration and the raw parameters it had reached, and, where a change's starts are refitted side by side, the start.

verbose

Report progress: each round's steps, each refit's time, and while a refit runs, its iteration, objective and gradient every ten minutes.

Value

An object of class ctModelImprove: fit (the final fit), history (every refit tried: the round, the relaxation, its screening p-value, the change in log likelihood and parameters, the criterion's verdict, and whether it was kept; a pruning row, family 'prune', says whether the change was dropped, and a 'rank' row whether the lower rank was kept), screens (the score tests of each round), adequacy (the final checks), confirmation (with holdout: each kept change's likelihood ratio test on the held-out subjects), and stopped (why the search ended: 'adequate', no adequacy family detected misfit; 'close fit', those that did were shown within close; 'not adequate, no candidate improved' or 'not adequate, nothing left to try', the checks still detect misfit; 'nothing indicated', with adequacy = FALSE, the screen was clean; 'maxsteps'). A refit that failed, or ended below the model it extends, is in history with a note saying so. With holdout, the held-out subjects take no part in the search; each kept change is tested once on them, at the end, at alpha. fit is the model with the confirmed changes refitted to every subject, and selected the search's own choice fitted to the subjects it searched on.

Details

The relaxations, in the language of the model:

heterogeneity

indvarying = TRUE for a parameter and the T0MEANS parameters, whose covariances with it the test took in; for a fixed MANIFESTMEANS cell, a parameter there varying about the fixed value as its stated mean.

covariates

a covariate added as a time independent predictor, with effects on the parameters its tests single out.

statedependence

a term d * eta_k * eta_m in a latent's drift, written into CINT with d a PARS parameter: a drift coefficient that varies with a state.

volatility

a latent's innovation sd scaled by exp(v * eta_m): innovations whose size depends on a state.

fixed

a cell fixed at zero freed: a drift cross effect, a diffusion covariance, a cross loading.

level

a parameter of the dynamics or error variances that depends on a subject's level (a random intercept) let vary over subjects, its random effect correlated with the intercepts (ctScoreTest's level terms).

slow

a slow process driving a latent through its drift, its effect started at one and its persistence at a tenth of the latent's; where the model has a slow process for another latent, that process driving this one too is tried first.

periodic

a cycle of the period the adequacy check points to, driving every latent with innovations of its own, each subject's phase set from the clock at its first row and free to deviate; the phase predictors are added to the data.

unique

a process of its own, persistent, loading on one item; and, where the pairs family finds two items' errors correlated within an occasion, a process the two share (ctModelUniqueProcess(shared = TRUE)): an error is meant to be random, and co-movement within an occasion that the latents' innovation covariances do not explain is read as a process faster than the occasions. Like the other structural changes it is tried once the score tests indicate nothing, ranked against them by the check's evidence, so that it does not absorb misfit that belongs to a cycle or to one item's own process.

errorcov

not by default: a measurement error covariance the score test flags is freed, for designs with a reason for correlated error, such as a method the items share, and no shared process is proposed.

order

a latent process made second order.

On a reduced-rank level (poprank) a heterogeneity result for a parameter that already varies through the shared dimensions gives it a variance of its own (popunique); without reduced rank, a significant dimension test makes the parameters its direction loads on vary together. Relaxations the score tests cannot tell apart (the same class) are not refitted in the same round.

Each random effect the search adds brings its correlations with the others, and with many effects for few subjects the data no longer determine them. When a kept change leaves the fit's curvature weak or negative along the population covariance (fit$identifiability), the covariance is refitted at one rank lower (poprank), starting from its leading dimensions, and the lower rank kept unless the likelihood ratio test needs the dropped dimension; it steps down until the fit is identified. Later random effects then load on the shared dimensions, and later heterogeneity results give own variances. Single-level models only.

When the search stops, each change kept is tried out again, by refitting without it (prune), and dropped if the others make it redundant.

A relaxation is kept when its refit improves on the current model by criterion: a likelihood ratio test at alpha divided by the number of refits tried at that step ('LR'), or a lower AIC or BIC (subjects as the sample size). The likelihood ratio test of a new random effect against the chi-square is conservative, as its variance starts on a boundary.

Error rates

The screen and the refit read the same data, so a refit confirms most of what the screen proposes, and the search's error rate is close to alpha per decision. Measured on simulated data (one latent, 80 subjects, 20 irregular rows, 40 searches each): with nothing missing, alpha = 0.05 kept something in 5 per cent of searches and alpha = 0.01 in none; an omitted covariate shifting a random intercept by half its sd was added in 75 and 55 per cent; an unrelated covariate never. criterion = 'BIC' kept something in 8 per cent, so at that size it is not the stricter choice. A drift that varied over subjects was never added as such at that size; a random diffusion or measurement error variance stood in for it in 10 per cent, since what the data showed was subjects differing in variance. A search makes a decision per family it screens, so its error rate grows with the families: on two processes with three items each (60 subjects, about 48 rows each), where the screen also tests state dependence and volatility in each process, alpha = 0.05 kept something in 4 of 20 searches with nothing missing (a volatility term twice, state dependence once, a random effect once), and alpha = 0.01 is the stricter choice. With an omitted cross effect, a process of an item's own and a daily cycle in the same data, the search kept the cross effect in 19 of 20 searches, the cycle in 16 and the item's process in all 20.

The result is a model chosen by the data it is fitted to, and its standard errors and tests do not account for that choice (MacCallum, Roznowski & Necowitz, 1992). holdout sets aside a share of the subjects: the search runs on the rest, and each change it keeps is then tested on the held-out subjects alone, by refitting the final model without it there; confirmation says which held, and the final model, refitted to every subject, keeps only those. Every step, and every refit tried and rejected, is in history.

With many observations a model can be rejected by the checks for departures that are small on these summaries. With close, the search also stops when every check that fails is a residual or within-row correlation family whose size is shown below close (ctFitAdequacy).

Each refit is judged by criterion; with 'LR', the likelihood ratio against a chi-square on the parameters the change adds, at alpha divided by the refits the round tries. That reference is exact for none of the changes the search makes: a variance added at zero sits on its boundary, where the chi-square is conservative, and a process or cycle added brings parameters that mean nothing when its coupling is zero, where it can err either way. Screening, rounds and pruning add further choices the level does not count. The search is a way of developing a model, and its p-values describe the steps it took rather than testing the result. A refit that ends below the model it extends has not reached an optimum, and is recorded as such.