Skip to contents

Tests, from one fitted model and without refitting it, whether the model would fit better with a relaxation it does not contain: parameters that differ between subjects (an unmodelled random effect), parameters that depend on a time independent covariate the model does not include, or dynamics that depend on the latent state (a missing nonlinearity). Each test is a score (Lagrange multiplier) test: the slope of the log likelihood in the direction of the relaxation, at the fitted model, with the fitted parameters projected out. Each also gives a one-step estimate of the relaxation's size, for screening: refit with the relaxations that matter before interpreting their sizes.

Usage

ctScoreTest(
  fit,
  covariates = NULL,
  tests = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed",
    "level", "instability"),
  form = c("auto", "linear", "spline", "factor", "threshold"),
  df = 3,
  parameters = NULL,
  order = 2L,
  variance = c("robust", "model"),
  nboot = 1999L,
  exact = 5L,
  cores = getOption("mc.cores", 2L),
  verbose = TRUE
)

Arguments

fit

A fit from ctFit with backend = 'julia', optimised.

covariates

Optional data frame of subject-level covariates to test: a column named as the fit's subject id (or id) and one column per covariate. Long data with repeated rows per subject is accepted when each covariate is constant within subject. Without an id column, one row per subject in the fit's subject order.

tests

Which families to run: any of 'heterogeneity', 'covariates', 'statedependence', 'volatility', 'fixed', 'level', 'instability'. By default all but 'instability'; covariates are tested only when covariates is given, and levels only where the model has a random intercept.

form

How a covariate enters: 'auto' (indicators for a factor, character or logical covariate; linear otherwise), 'linear', 'spline', 'factor' or 'threshold' (a step at an unknown cut). One value, or one per covariate.

df

Degrees of freedom of the spline basis.

parameters

Optional parameter names to restrict the heterogeneity and covariate tests to.

order

Highest degree of the state-dependence terms: 2, each drift coefficient varying linearly with each state; 3 adds variation with products and squares of states, which is what a symmetric dependence – regulation that strengthens at both extremes, say – shows up as, and which degree 2 cannot see.

variance

'robust' (the default) or 'model'; see the section on variance.

nboot

Bootstrap resamples for the robust p-values.

exact

For statedependence, volatility and fixed with the robust variance: how many of each family's terms have the fit's estimation removed exactly, which takes two evaluations of the extended model per term. Every term is first tested on its scores without that correction, which is conservative, and the strongest exact by that test are then corrected (projected in the tables). Inf corrects every term.

cores

Engine cores for the evaluations.

verbose

Report progress.

Value

An object of class ctScoreTest: a list with heterogeneity (a data frame, one row per parameter or fixed MANIFESTMEANS cell), dimension (the largest eigenvalue, its p-value and the direction, when two or more parameters are tested), covariates (one element per covariate, each a list of its joint test and a data frame parameters), statedependence (a list of terms, a data frame, and joint, the tests per equation and overall), volatility (the same, for the innovation sds), fixed (a list of cells and their joint test), instability (a list of parameters, one row per ordering and parameter, and joint, per ordering), and info (the route, the number of subjects and parameters, the variance used, and errors, why any family was not tested). Tables carry class where relaxations are alike, and the extended families' projected, whether a term's estimation was removed exactly (see exact).

Heterogeneity

For each parameter that is free, does not already vary over subjects, and is not an initial condition (T0MEANS and T0VAR, whose variation T0VAR already carries), the score test of letting it vary as ctFit would fit it: its variance over subjects together with its covariances with what already varies – each subject's initial state (the free T0MEANS parameters) and the parameters with random effects, named in with. Every subject has an initial state, so even a model without random effects has those covariances to test. They matter: an intercept in CINT that varies while the initial means are shared makes subjects drift apart from a common start, which data whose subjects differ stably from the first occasion reject, whereas the same intercept correlated with where each subject starts is a random intercept. The covariances are tested as ordinary score tests and the variance, given them, one sided (Stram & Lee, 1994): LM is referred to an equal mixture of chi-square on df - 1 and df. Where the new effect is independent of what varies, those covariances only cost power, so p is the stronger of that test and of the variance alone (z, one sided, its covariances held at zero), by their smaller p-value, calibrated by resampling subjects. A negative z beside a large LM says the parameter varies only together with what already varies, as an intercept does with where each subject starts. Nothing is assumed about how the initial states relate to the dynamics: neither that subjects start at their own equilibrium nor that they share a start. The statistic is the information matrix test in those coordinates, which is the score test for neglected heterogeneity (Chesher, 1984). It responds to any failure of the information equality there, of which subject differences are the usual cause but not the only one: a subject-varying drift also moves the diffusion's test, because both change how much a subject varies, and a missing nonlinearity moves several. sd is the one-step estimate of the subjects' standard deviation on the raw scale, with the covariances estimated beside it, and sd_natural approximately on the parameter's own; NA where the covariances carry the test and the variance given them reads zero, which happens for an intercept, and which only a refit can size. To act on a result, set indvarying = TRUE for the parameter and the T0MEANS parameters, refit, and test again; ctModelImprove does that.

Each MANIFESTMEANS cell the model fixes is tested too, as a random intercept on that indicator about its fixed value (rows named MANIFESTMEANS[...]), from the model with those cells written as parameters. With one indicator per latent such an intercept is a direction the CINT rows' covariances also reach; with more indicators it is the indicator's own; for a latent with no free intercept at all it is the only test of its level. To act on it, give the cell a parameter with the fixed value as its stated mean (ctPar(mean=)) and indvarying = TRUE.

On a reduced-rank level (poprank), a parameter that varies only through the shared dimensions, without a variance of its own, is tested too: for variation outside those dimensions. There the shared dimensions carry the covariances, and each test is of the variance alone. A parameter whose test the fitted model already determines – under one per cent of its spread left once estimation is taken out, as for an intercept that loads on the shared dimensions – is not tested (NA) and is left out of dimension.

dimension tests every direction at once: whether subjects vary along some combination of the tested parameters, a new dimension of person variation, its covariances with what already varies held at zero. It is not computed on a reduced-rank level, where in simulation it found a dimension the rank lacked 10 per cent of the time and the per-parameter tests of variation outside the shared dimensions 58 per cent; those are the test there. Adding variance along a direction u changes a subject's log likelihood, at zero, by (u's_i)^2/2 + u'H_iu/2, so the variances are the diagonal of one score matrix; its largest eigenvalue, scaled by each diagonal's spread, is the statistic, against a bootstrap over subjects, and its eigenvector is the direction, read as relative loadings (raw scale). It can detect variation spread over parameters that no single test singles out, and is less sensitive than the best single test when one parameter carries it.

Covariates

For each covariate, the effects it would have on every tested parameter if added as a time independent predictor (on the raw scale, as a predictor effect acts), including on the population standard deviations of the parameters that vary: a covariate moderating heterogeneity. The joint test of all of a covariate's effects is the screen. Per parameter, estimate is the one-step effect per unit of the covariate with the fitted parameters held at their estimates, jointly over the parameters, so a covariate that shifts one parameter is not read off another correlated with it; estimate_natural puts it approximately on the parameter's own scale, and p tests it. Holding the fit's parameters is what keeps the size near a refit's: a covariate the model lacks is absorbed by the parameters it has – a random effect's standard deviation grows to take in a covariate that shifts its mean – and letting them move in a one-step extrapolates the likelihood across that whole change. A covariate enters linearly (centred), as treatment indicators for a factor, or as a natural spline with df degrees of freedom, for which only p is given; subjects missing it contribute at the mean. To act on a result, add the covariate to the data and to TIpredNames and refit.

form = 'threshold' tests a step at an unknown cut instead: the joint test at each cut from the 15th to the 85th percentile, and the largest of them (Andrews, 1993) – the split a score-guided tree makes – against the same scan with the covariate permuted over subjects (Hothorn & Lausen, 2003), which takes the search into account exactly and refits nothing. The effects reported are those of being above the best cut, cut in the joint row.

On a reduced-rank level the spread of a varying parameter has no popsd_ coordinate, so a covariate's effect on it is tested along the direction that scales that parameter's row of the loadings and its own sd together, reported as scale(<parameter>): the effect on its log standard deviation per unit of the covariate, the correlations held. In simulation (rank 1) these rows rejected 6.5 to 8 per cent of correct models at .05 with 100 subjects and 5 to 6 with 300, where the covariate's joint test held its level: with few subjects the projection on the fitted parameters leaves a variance a little too small. Read them as attribution once the joint test has screened.

State dependence

For each latent equation and each pair of latent states k <= m, the score test of a term d * eta_k * eta_m in that equation's drift: a drift coefficient that varies linearly with a state, the second-order expansion of a drift that depends on the state (Luukkonen, Saikkonen & Terasvirta, 1988). A term eta_k * eta_m in the equation of eta_i is DRIFT[i,k] varying with eta_m – and, when k != m, equally DRIFT[i,m] varying with eta_k; at this order the data cannot tell those apart, and reads says both. estimate is the one-step change in that drift coefficient per unit of the state. It is NA when the fitted model's curvature cannot size the term, which happens when the departure is strong; the test itself is unaffected. order = 3 adds the terms of degree three. The terms are evaluated by the model's own filter at the state predicted from earlier observations, which keeps the test valid; a state estimate that has seen the observation being scored – the updated or smoothed state – would not. To act on a result, write the dependence into the drift cell (DRIFT[1,1] = 'dr11 + d * eta2' with PARS = c('dr11|-log1p_exp(-param)', 'd')) and refit.

Volatility

tests = 'volatility' asks whether the size of each latent's innovations depends on a state: the score test of DIFFUSION[i, i], the innovation sd, scaled by exp(v * eta_m), at v = 0, for every latent with innovations and every state. Variance that rises with a process's own level, as when negative affect is more variable when high, is the diagonal term. Evaluated by the filter at the predicted state, as for state dependence. To act on a result, write the scaling into the cell (DIFFUSION[2,2] = 'sd2 * exp(v * eta2)' with PARS = c('sd2|log1p_exp(param)', 'v')) and refit; ctModelImprove does that. The terms make the model nonlinear, so the test, and a refit, run the extended Kalman filter.

Level

tests = 'level' asks whether a parameter of the dynamics or of the error variances depends on a subject's level, where that level is a random intercept: for each free DRIFT, DIFFUSION and MANIFESTVAR parameter that does not vary over subjects, and each MANIFESTMEANS or CINT intercept that does, the score test of a term g * intercept added to the parameter, at g = 0. When subjects' levels sit in the intercepts, the latent states are deviations from them, so the state dependence and volatility tests, which read the states, see dependence on the deviation only; a parameter that depends on where a subject sits – persistence or innovation size that differs between people who are usually high and people who are usually low – shows there only as heterogeneity, in no particular direction. These terms ask about that direction. The score is taken over each subject's posterior for the intercept, from the extended model's own gradient, which the product of a subject's score and estimated level is not. estimate is the one-step change in the parameter, on its own scale, per unit of the intercept. To act on a result, let the parameter vary over subjects, so its random effect is estimated with its correlation with the intercept; ctModelImprove does that. A model with no random intercept has nothing to test.

Fixed cells

For each cell the model fixes at zero that could sensibly be freed – a DRIFT cross effect, a DIFFUSION or MANIFESTVAR covariance between two that have a variance, a LAMBDA cross loading, a TDPREDEFFECT – the score test of freeing it: the modification index (Sorbom, 1989), with estimate the expected parameter change. CINT and MANIFESTMEANS are not offered, since with the other free either is identified only through the initial transient. State dependence, volatility and fixed cells are tested on one extended model, so asking for several costs one preparation.

Stability

tests = 'instability', not run unless asked for, tests whether each parameter is stable across the data: ordered by time, and by each latent's state as predicted from earlier observations. It uses each data row's score – the derivative of its one-step log likelihood – whose partial sums, under the model, behave as a Brownian bridge in any order fixed in advance (Nyblom, 1989). A parameter that changes over the study, or differs at high and low levels of a state, bends them. Ordered by a state, that is a test of state dependence of unknown form, beside the polynomial terms of statedependence. The estimation effect is removed with the information accumulated along the ordering, and the p-values are from a wild bootstrap over rows. It costs a dual pass of the filter per eight parameters, and on the Laplace route one per truncated record of every subject.

Alike relaxations

Two relaxations whose per-subject projected contributions are nearly collinear (correlation of 0.9 or more) would each explain the same signal, and the data cannot tell them apart at this order (MacCallum et al., 1993). They share a letter in class; NA means alike to none. Refit with the most plausible, then test again.

Variance and p-values

The fitted model's curvature projects the fitted parameters out of each test and sets the one-step sizes; the variance of each test is, by default (variance = 'robust'), the spread over subjects of their projected contributions, and its p-value a bootstrap over subjects of the studentised statistic (nboot resamples, from a fixed stream, so a call is reproducible and leaves the session's random numbers alone). That keeps each test valid when the model is wrong in some other way, which is when a screen is used, and corrects for the skewness of a variance score. variance = 'model' uses the curvature for the variance and chi-square and normal references instead; it is what is used below 20 subjects, where a spread over subjects is too thin to estimate.

Reading the results

The tests are screens. A badly misspecified model lights up many at once, and their one-step sizes are local approximations – especially for drift parameters, which act through the matrix exponential. Relax what stands out most, refit, and test again. Within each family p_adj is the Benjamini-Hochberg adjustment over the tests shown. The tests use the fit's own objective, so they apply to the route it was fitted on (intoverpop); they are made at an optimum, and refuse a sampled fit. Heterogeneity and covariate tests need subjects as the independent units, so a model with a level above the subject can only be tested for state dependence. A family that cannot be tested on a model says why, as a warning and in info$errors, and the other families' results stand. With priors = TRUE the new terms of the extended model would carry the normal(0, 1) prior every parameter has; they are tested on the likelihood, without it.

References

Boos, D. D. (1992). On generalized score tests. The American Statistician, 46, 327-333.

Chesher, A. (1984). Testing for neglected heterogeneity. Econometrica, 52, 865-872.

Stram, D. O., & Lee, J. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50, 1171-1177.

Self, S. G., & Liang, K.-Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82, 605-610.

Hothorn, T., & Lausen, B. (2003). On the exact distribution of maximally selected rank statistics. Computational Statistics & Data Analysis, 43, 121-137.

Andrews, D. W. K. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61, 821-856.

Nyblom, J. (1989). Testing for the constancy of parameters over time. Journal of the American Statistical Association, 84, 223-230.

Hall, D. B., & Praestgaard, J. T. (2001). Order-restricted score tests for homogeneity in generalised linear and nonlinear mixed models. Biometrika, 88, 739-751.

MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological Bulletin, 114, 185-199.

Luukkonen, R., Saikkonen, P., & Terasvirta, T. (1988). Testing linearity against smooth transition autoregressive models. Biometrika, 75, 491-499.

Sorbom, D. (1989). Model modification. Psychometrika, 54, 371-384.

See also

vignette('score-tests') for three worked examples.

Examples

if (FALSE) { # \dontrun{
fit <- ctFit(dat, model, backend = 'julia')
st <- ctScoreTest(fit, covariates = subjectdata)
st
st$heterogeneity
} # }