Tests, from one fitted model and without refitting it, whether the model would fit better with a relaxation it does not contain: parameters that differ between subjects (an unmodelled random effect), parameters that depend on a time independent covariate the model does not include, or dynamics that depend on the latent state (a missing nonlinearity). Each test is a score (Lagrange multiplier) test: the slope of the log likelihood in the direction of the relaxation, at the fitted model, with the fitted parameters projected out. Each also gives a one-step estimate of the relaxation's size, for screening: refit with the relaxations that matter before interpreting their sizes.
Usage
ctScoreTest(
fit,
covariates = NULL,
tests = c("heterogeneity", "covariates", "statedependence", "volatility", "fixed",
"level", "instability"),
form = c("auto", "linear", "spline", "factor", "threshold"),
df = 3,
parameters = NULL,
order = 2L,
variance = c("robust", "model"),
nboot = 1999L,
exact = 5L,
cores = getOption("mc.cores", 2L),
verbose = TRUE
)Arguments
- fit
A fit from
ctFitwithbackend = 'julia', optimised.- covariates
Optional data frame of subject-level covariates to test: a column named as the fit's subject id (or
id) and one column per covariate. Long data with repeated rows per subject is accepted when each covariate is constant within subject. Without an id column, one row per subject in the fit's subject order.- tests
Which families to run: any of
'heterogeneity','covariates','statedependence','volatility','fixed','level','instability'. By default all but'instability'; covariates are tested only whencovariatesis given, and levels only where the model has a random intercept.- form
How a covariate enters:
'auto'(indicators for a factor, character or logical covariate; linear otherwise),'linear','spline','factor'or'threshold'(a step at an unknown cut). One value, or one per covariate.- df
Degrees of freedom of the spline basis.
- parameters
Optional parameter names to restrict the heterogeneity and covariate tests to.
- order
Highest degree of the state-dependence terms: 2, each drift coefficient varying linearly with each state; 3 adds variation with products and squares of states, which is what a symmetric dependence – regulation that strengthens at both extremes, say – shows up as, and which degree 2 cannot see.
- variance
'robust'(the default) or'model'; see the section on variance.- nboot
Bootstrap resamples for the robust p-values.
- exact
For
statedependence,volatilityandfixedwith the robust variance: how many of each family's terms have the fit's estimation removed exactly, which takes two evaluations of the extended model per term. Every term is first tested on its scores without that correction, which is conservative, and the strongestexactby that test are then corrected (projectedin the tables).Infcorrects every term.- cores
Engine cores for the evaluations.
- verbose
Report progress.
Value
An object of class ctScoreTest: a list with
heterogeneity (a data frame, one row per parameter or fixed
MANIFESTMEANS cell),
dimension (the largest eigenvalue, its p-value and the direction,
when two or more parameters are tested),
covariates (one element per covariate, each a list of its
joint test and a data frame parameters),
statedependence (a list of terms, a data frame, and
joint, the tests per equation and overall), volatility
(the same, for the innovation sds), fixed (a list
of cells and their joint test), instability (a list
of parameters, one row per ordering and parameter, and
joint, per ordering), and info (the
route, the number of subjects and parameters, the variance used, and
errors, why any family was not tested). Tables carry class
where relaxations are alike, and the extended families' projected,
whether a term's estimation was removed exactly (see exact).
Heterogeneity
For each parameter that is free, does not already
vary over subjects, and is not an initial condition (T0MEANS and T0VAR,
whose variation T0VAR already carries), the score test of letting it vary
as ctFit would fit it: its variance over subjects together with
its covariances with what already varies – each subject's initial state
(the free T0MEANS parameters) and the parameters with random
effects, named in with. Every subject has an initial state, so
even a model without random effects has those covariances to test. They
matter: an intercept in CINT that varies while the initial means
are shared makes subjects drift apart from a common start, which data
whose subjects differ stably from the first occasion reject, whereas the
same intercept correlated with where each subject starts is a random
intercept. The covariances are tested as ordinary score tests and the
variance, given them, one sided (Stram & Lee, 1994): LM is
referred to an equal mixture of chi-square on df - 1 and
df. Where the new effect is independent of what varies, those
covariances only cost power, so p is the stronger of that test
and of the variance alone (z, one sided, its covariances held at
zero), by their smaller p-value, calibrated by resampling subjects. A
negative z beside a large LM says the parameter varies only
together with what already varies, as an intercept does with where each
subject starts. Nothing is assumed about how the initial states relate
to the dynamics: neither that subjects start at their own equilibrium
nor that they share a start. The statistic is the information matrix
test in those coordinates, which is the score test for neglected
heterogeneity (Chesher, 1984). It responds to any failure of the
information equality there, of which subject differences are the usual
cause but not the only one: a subject-varying drift also moves the
diffusion's test, because both change how much a subject varies, and a
missing nonlinearity moves several. sd is the one-step estimate of
the subjects' standard deviation on the raw scale, with the covariances
estimated beside it, and sd_natural approximately on the
parameter's own; NA where the covariances carry the test and the
variance given them reads zero, which happens for an intercept, and
which only a refit can size. To act on a result, set indvarying = TRUE for the
parameter and the T0MEANS parameters, refit, and test again;
ctModelImprove does that.
Each MANIFESTMEANS cell the model fixes is tested too, as a
random intercept on that indicator about its fixed value (rows named
MANIFESTMEANS[...]), from the model with those cells written as
parameters. With one indicator per latent such an intercept is a
direction the CINT rows' covariances also reach; with more
indicators it is the indicator's own; for a latent with no free
intercept at all it is the only test of its level. To act on it, give
the cell a parameter with the fixed value as its stated mean
(ctPar(mean=)) and indvarying = TRUE.
On a reduced-rank level (poprank), a parameter that varies
only through the shared dimensions, without a variance of its own, is
tested too: for variation outside those dimensions. There the shared
dimensions carry the covariances, and each test is of the variance
alone. A parameter whose test the fitted model already determines –
under one per cent of its spread left once estimation is taken out, as
for an intercept that loads on the shared dimensions – is not tested
(NA) and is left out of dimension.
dimension tests every direction at once: whether subjects vary
along some combination of the tested parameters, a new dimension of
person variation, its covariances with what already varies held at
zero. It is not computed on a reduced-rank level, where in
simulation it found a dimension the rank lacked 10 per cent of the time
and the per-parameter tests of variation outside the shared dimensions 58
per cent; those are the test there. Adding
variance along a direction u changes a subject's log likelihood,
at zero, by (u's_i)^2/2 + u'H_iu/2, so the variances are
the diagonal of one score matrix; its largest eigenvalue, scaled by each
diagonal's spread, is the statistic, against a bootstrap over subjects,
and its eigenvector is the direction, read as relative loadings (raw
scale). It can detect variation spread over parameters that no single
test singles out, and is less sensitive than the best single test when
one parameter carries it.
Covariates
For each covariate, the effects it would have on every
tested parameter if added as a time independent predictor (on the raw
scale, as a predictor effect acts), including on the population standard
deviations of the parameters that vary: a covariate moderating
heterogeneity. The joint test of all of a covariate's effects is the
screen. Per parameter, estimate is the one-step effect per unit of
the covariate with the fitted parameters held at their estimates, jointly
over the parameters, so a covariate that shifts one parameter is not read
off another correlated with it; estimate_natural puts it
approximately on the parameter's own scale, and p tests it. Holding
the fit's parameters is what keeps the size near a refit's: a covariate
the model lacks is absorbed by the parameters it has – a random effect's
standard deviation grows to take in a covariate that shifts its mean –
and letting them move in a one-step extrapolates the likelihood across
that whole change. A covariate enters linearly (centred), as treatment
indicators for a factor, or as a natural spline with df degrees of
freedom, for which only p is given; subjects missing it contribute
at the mean. To act on a result, add the covariate to the data and to
TIpredNames and refit.
form = 'threshold' tests a step at an unknown cut instead: the
joint test at each cut from the 15th to the 85th percentile, and the
largest of them (Andrews, 1993) – the split a score-guided tree makes –
against the same scan with the covariate permuted over subjects (Hothorn
& Lausen, 2003), which takes the search into account exactly and refits
nothing. The effects reported are those of being above the best cut,
cut in the joint row.
On a reduced-rank level the spread of a varying parameter has no
popsd_ coordinate, so a covariate's effect on it is tested along
the direction that scales that parameter's row of the loadings and its own
sd together, reported as scale(<parameter>): the effect on its log
standard deviation per unit of the covariate, the correlations held. In
simulation (rank 1) these rows rejected 6.5 to 8 per cent of correct
models at .05 with 100 subjects and 5 to 6 with 300, where the
covariate's joint test held its level: with few subjects the projection
on the fitted parameters leaves a variance a little too small. Read them
as attribution once the joint test has screened.
State dependence
For each latent equation and each pair of
latent states k <= m, the score test of a term d * eta_k *
eta_m in that equation's drift: a drift coefficient that varies linearly
with a state, the second-order expansion of a drift that depends on the
state (Luukkonen, Saikkonen & Terasvirta, 1988). A term eta_k *
eta_m in the equation of eta_i is DRIFT[i,k] varying with
eta_m – and, when k != m, equally DRIFT[i,m] varying
with eta_k; at this order the data cannot tell those apart, and
reads says both. estimate is the one-step change in that
drift coefficient per unit of the state. It is NA when the fitted model's
curvature cannot size the term, which happens when the departure is
strong; the test itself is unaffected. order = 3 adds the terms
of degree three. The terms are evaluated by the model's own filter at the
state predicted from earlier observations, which keeps the test valid; a
state estimate that has seen the observation being scored – the updated
or smoothed state – would not. To act on a result, write the dependence
into the drift cell (DRIFT[1,1] = 'dr11 + d * eta2' with
PARS = c('dr11|-log1p_exp(-param)', 'd')) and refit.
Volatility
tests = 'volatility' asks whether the size of
each latent's innovations depends on a state: the score test of
DIFFUSION[i, i], the innovation sd, scaled by exp(v *
eta_m), at v = 0, for every latent with innovations and every
state. Variance that rises with a process's own level, as when negative
affect is more variable when high, is the diagonal term. Evaluated by the
filter at the predicted state, as for state dependence. To act on a
result, write the scaling into the cell (DIFFUSION[2,2] = 'sd2 *
exp(v * eta2)' with PARS = c('sd2|log1p_exp(param)', 'v')) and
refit; ctModelImprove does that. The terms make the model
nonlinear, so the test, and a refit, run the extended Kalman filter.
Level
tests = 'level' asks whether a parameter of the
dynamics or of the error variances depends on a subject's level, where
that level is a random intercept: for each free DRIFT,
DIFFUSION and MANIFESTVAR parameter that does not vary over
subjects, and each MANIFESTMEANS or CINT intercept that
does, the score test of a term g * intercept added to the
parameter, at g = 0. When subjects' levels sit in the intercepts,
the latent states are deviations from them, so the state dependence and
volatility tests, which read the states, see dependence on the deviation
only; a parameter that depends on where a subject sits – persistence or
innovation size that differs between people who are usually high and
people who are usually low – shows there only as heterogeneity, in no
particular direction. These terms ask about that direction. The score
is taken over each subject's posterior for the intercept, from the
extended model's own gradient, which the product of a subject's score
and estimated level is not. estimate is the one-step change in the
parameter, on its own scale, per unit of the intercept. To act on a
result, let the parameter vary over subjects, so its random effect is
estimated with its correlation with the intercept;
ctModelImprove does that. A model with no random intercept
has nothing to test.
Fixed cells
For each cell the model fixes at zero that could
sensibly be freed – a DRIFT cross effect, a DIFFUSION or
MANIFESTVAR covariance between two that have a variance, a
LAMBDA cross loading, a
TDPREDEFFECT – the score test of freeing it: the modification
index (Sorbom, 1989), with estimate the expected parameter change.
CINT and MANIFESTMEANS are not offered, since with the
other free either is identified only through the initial transient.
State dependence, volatility and fixed cells are tested on one extended
model, so asking for several costs one preparation.
Stability
tests = 'instability', not run unless asked
for, tests whether each parameter is stable across the data: ordered by
time, and by each latent's state as predicted from earlier observations.
It uses each data row's score – the derivative of its one-step log
likelihood – whose partial sums, under the model, behave as a Brownian
bridge in any order fixed in advance (Nyblom, 1989). A parameter that
changes over the study, or differs at high and low levels of a state,
bends them. Ordered by a state, that is a test of state dependence of
unknown form, beside the polynomial terms of statedependence. The
estimation effect is removed with the information accumulated along the
ordering, and the p-values are from a wild bootstrap over rows. It costs a
dual pass of the filter per eight parameters, and on the Laplace route one
per truncated record of every subject.
Alike relaxations
Two relaxations whose per-subject projected
contributions are nearly collinear (correlation of 0.9 or more) would
each explain the same signal, and the data cannot tell them apart at this
order (MacCallum et al., 1993). They share a letter in class;
NA means alike to none. Refit with the most plausible, then test
again.
Variance and p-values
The fitted model's curvature projects the
fitted parameters out of each test and sets the one-step sizes; the
variance of each test is, by default (variance = 'robust'), the
spread over subjects of their projected contributions, and its p-value a
bootstrap over subjects of the studentised statistic (nboot
resamples, from a fixed stream, so a call is reproducible and leaves the
session's random numbers alone). That keeps each test valid when the model
is wrong in some other way, which is when a screen is used, and corrects
for the skewness of a variance score. variance = 'model' uses the
curvature for the variance and chi-square and normal references instead;
it is what is used below 20 subjects, where a spread over subjects is too
thin to estimate.
Reading the results
The tests are screens. A badly misspecified
model lights up many at once, and their one-step sizes are local
approximations – especially for drift parameters, which act through the
matrix exponential. Relax what stands out most, refit, and test again.
Within each family p_adj is the Benjamini-Hochberg adjustment over
the tests shown. The tests use the fit's own objective, so they apply to
the route it was fitted on (intoverpop); they are made at an
optimum, and refuse a sampled fit. Heterogeneity and covariate tests need
subjects as the independent units, so a model with a level above the
subject can only be tested for state dependence. A family that cannot be
tested on a model says why, as a warning and in info$errors, and
the other families' results stand. With priors = TRUE the new terms
of the extended model would carry the normal(0, 1) prior every parameter
has; they are tested on the likelihood, without it.
References
Boos, D. D. (1992). On generalized score tests. The American Statistician, 46, 327-333.
Chesher, A. (1984). Testing for neglected heterogeneity. Econometrica, 52, 865-872.
Stram, D. O., & Lee, J. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50, 1171-1177.
Self, S. G., & Liang, K.-Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82, 605-610.
Hothorn, T., & Lausen, B. (2003). On the exact distribution of maximally selected rank statistics. Computational Statistics & Data Analysis, 43, 121-137.
Andrews, D. W. K. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61, 821-856.
Nyblom, J. (1989). Testing for the constancy of parameters over time. Journal of the American Statistical Association, 84, 223-230.
Hall, D. B., & Praestgaard, J. T. (2001). Order-restricted score tests for homogeneity in generalised linear and nonlinear mixed models. Biometrika, 88, 739-751.
MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological Bulletin, 114, 185-199.
Luukkonen, R., Saikkonen, P., & Terasvirta, T. (1988). Testing linearity against smooth transition autoregressive models. Biometrika, 75, 491-499.
Sorbom, D. (1989). Model modification. Psychometrika, 54, 371-384.
See also
vignette('score-tests') for three worked examples.
Examples
if (FALSE) { # \dontrun{
fit <- ctFit(dat, model, backend = 'julia')
st <- ctScoreTest(fit, covariates = subjectdata)
st
st$heterogeneity
} # }