What is the model missing? Score tests after a fit
Source:vignettes/score-tests.qmd
score-tests.RmdA fitted model can be wrong in ways its own estimates do not show.
Subjects may differ in a parameter the model holds fixed; a parameter
may depend on a covariate that was never put in the model; the dynamics
may depend on where the process is. ctScoreTest() asks of
one fitted model, without refitting it, whether any of these relaxations
would fit better, and if so where:
- heterogeneity: does a parameter differ between subjects, so that it should be individually varying?
- covariates: does a parameter depend on a subject-level covariate the model does not contain – including the size of a random effect, not only its mean?
- state dependence: does a drift coefficient change with the level of a latent state, so that the dynamics are nonlinear?
- volatility: does the size of a process’s random shocks change with the level of a state – negative affect more variable when it is high, say?
Each is a score (Lagrange multiplier) test: the slope of the log likelihood, at the fitted model, in the direction of the relaxation. It needs a few evaluations of the fitted model’s gradient rather than a new fit per candidate, so any number of candidates can be screened after one fit, and each test also says roughly how large the relaxation would be. The tests are screens: they point at what to relax, and the refit says how much it matters.
The three examples below each generate data with one thing the fitted
model lacks, fit the model without it, ask ctScoreTest(),
and then refit with what it found. The sections after them add a fourth
family of tests (cells the model fixes at zero), the posterior
predictive checks as tests (ctFitAdequacy()), and a search
that combines the two and does the refitting itself
(ctModelImprove()).
The process
Two latent processes, each measured by one indicator, observed 30
times at irregular intervals for 150 subjects. eta1
regulates towards its baseline at rate 0.6, eta2 at 0.4,
and eta1 pushes eta2 up (a cross effect of
0.25). Each subject has their own measurement intercepts. The generating
model is written once, and each example changes one cell of it.
generator <- function(DRIFT, MANIFESTMEANS = c("mm1|param|TRUE|1", "mm2|param|TRUE|1"),
TIpredNames = NULL) {
ctModel(type = "ct", silent = TRUE,
manifestNames = c("Y1", "Y2"), latentNames = c("eta1", "eta2"),
LAMBDA = diag(2), DRIFT = DRIFT, CINT = matrix(0, 2, 1),
DIFFUSION = matrix(c(0.8, 0, 0, 0.6), 2, 2), MANIFESTVAR = diag(0.5, 2),
MANIFESTMEANS = matrix(MANIFESTMEANS, 2, 1),
T0MEANS = matrix(0, 2, 1), T0VAR = diag(1, 2), TIpredNames = TIpredNames)
}The model fitted in every example is ctsem’s default for two latents with one indicator each: every drift, diffusion and measurement error parameter free and shared by all subjects, and individually varying measurement intercepts and initial states.
1. Subjects who differ in a parameter
Here the subjects differ in how quickly eta1 regulates:
the drift DRIFT[1,1] varies between subjects around -0.6,
with a standard deviation of about 0.25.
gen1 <- generator(DRIFT = matrix(c("drift11|-log1p_exp(-param)|TRUE|0.6", 0.25,
0, -0.4), 2, 2))
d1 <- ctGenerate(gen1, n = 150, Tpoints = 30, dtmean = 1, logdtsd = 0.5,
backend = "julia", values = c(drift11 = -0.6, mm1 = 0, mm2 = 0))
truedrift <- attr(d1, "subjectpars")$drift11
c(mean = mean(truedrift), sd = sd(truedrift))
#> mean sd
#> -0.6345464 0.2660944
d1 <- data.frame(d1)The default model shares one drift among all subjects.
fit1 <- ctFit(d1, model, backend = "julia", cores = 2)
st1 <- ctScoreTest(fit1, tests = "heterogeneity")
st1
#> Score tests at the fitted model (laplace route, 150 subjects)
#>
#> Unmodelled heterogeneity (LM: the variance with its covariances, against an
#> equal mixture of chi-square on df - 1 and df; z: the variance alone, one sided;
#> p: the stronger of the two; sd: the one-step estimate, raw scale):
#> parameter LM df z p p_adj sd sd_natural
#> drift_eta1 19.40 5 2.580 0.0055 0.0292 0.19100 0.18400
#> drift_eta1_eta2 16.30 5 1.160 0.0065 0.0292 0.16300 0.16300
#> diff_eta2 14.20 5 -0.104 0.0175 0.0525 0.00187 0.00217
#> diff_eta1 9.59 5 1.790 0.0420 0.0918 0.10400 0.15600
#> mvarY2 12.00 5 1.600 0.0510 0.0918 0.05810 0.05690
#> diff_eta2_eta1 5.00 5 0.776 0.3330 0.5000 0.09630 0.09630
#> mvarY1 2.61 5 0.656 0.4120 0.5300 0.05140 0.04990
#> drift_eta2_eta1 2.81 5 -0.157 0.7920 0.8900 NA NA
#> drift_eta2 2.48 5 -1.410 0.9020 0.9020 NA NA
#> Covariances with: T0m_eta1, T0m_eta2, mm_Y1, mm_Y2
#>
#> A new dimension of variation: largest eigenvalue 6.49, p = 0.0565; direction drift_eta1_eta2 1, drift_eta1 1, drift_eta2_eta1 0.54, diff_eta2_eta1 0.39, diff_eta1 0.38Each row tests whether that parameter varies over subjects, as
ctFit() would fit it if it were made individually varying:
its variance together with its covariances with what already varies,
here each subject’s initial state and measurement intercepts (the line
under the table names them). A variance cannot be negative, so that part
is one sided; LM is referred to an equal mixture of
chi-squares on df - 1 and df. The covariances
matter most for an intercept: one that varies independently of where
each subject starts makes subjects drift apart from a common start,
whereas the same intercept correlated with the initial state is a random
intercept. z is the variance alone, and p the
stronger of the two tests.
The drift of eta1 is flagged, on its variance alone
(z). Other rows answer too, mostly through their
covariances (LM large where z is small): the
cross effect of eta2 on eta1, and the
diffusion of eta2. That is expected rather than further
findings. A drift that varies between subjects, left out of the model,
changes how each subject’s data relate to their starting point and to
their intercepts, and the other parameters’ scores carry some of that.
The test says that the subjects differ, and roughly where; it cannot
always say which of several related parameters is the source. The way to
find out is to relax the strongest candidate and test again.
The line under the table tests every direction at once: whether the
subjects vary along some combination of these parameters, which no
single row would isolate when the variation is spread over several of
them, its covariances with what already varies held at zero. Here one
parameter carries the variation, and its own row is the sharper test. On
a model whose population covariance has reduced rank
(poprank) the rows test variation outside the shared
dimensions, and the line is not given: in simulation the rows found a
missing dimension far more often.
model1b <- model
model1b$pars$indvarying[model1b$pars$param %in% "drift_eta1"] <- TRUE
fit1b <- ctFit(d1, model1b, backend = "julia", cores = 2)
c(loglik_without = fit1$estimate$loglik, loglik_with = fit1b$estimate$loglik)
#> loglik_without loglik_with
#> -11177.43 -11166.31
ctScoreTest(fit1b, tests = "heterogeneity")
#> Score tests at the fitted model (laplace route, 150 subjects)
#>
#> Unmodelled heterogeneity (LM: the variance with its covariances, against an
#> equal mixture of chi-square on df - 1 and df; z: the variance alone, one sided;
#> p: the stronger of the two; sd: the one-step estimate, raw scale):
#> parameter LM df z p p_adj sd sd_natural
#> diff_eta2 14.60 6 -0.127 0.0295 0.236 NA NA
#> mvarY2 11.90 6 1.610 0.0950 0.259 0.0581 0.0569
#> drift_eta1_eta2 11.70 6 0.528 0.0970 0.259 0.1010 0.1010
#> diff_eta1 7.85 6 1.280 0.1660 0.331 0.0495 0.0757
#> diff_eta2_eta1 6.03 6 0.921 0.3020 0.484 0.1250 0.1250
#> mvarY1 5.85 6 0.385 0.5880 0.784 NA NA
#> drift_eta2_eta1 1.52 6 -0.527 0.9020 0.961 NA NA
#> drift_eta2 2.40 6 -1.410 0.9610 0.961 NA NA
#> Covariances with: T0m_eta1, T0m_eta2, drift_eta1, mm_Y1, mm_Y2
#>
#> A new dimension of variation: largest eigenvalue 2.67, p = 0.8; direction diff_eta2_eta1 1, drift_eta1_eta2 0.61, mvarY2 -0.39, mvarY1 0.34, drift_eta2_eta1 0.3With the drift free to vary the log likelihood rises by about 11, and
nothing else stands out once the tests are adjusted for their number
(p_adj): the other rows’ signals were the drift’s. (A drift
that varies between subjects enters the filter nonlinearly, so
ctFit() chose the Laplace route for the refit; the tests
work on either.)
2. A covariate the model does not contain
Now each subject has an age (standardised), and age
matters twice: older subjects have a higher intercept on
Y1, and in older subjects eta1 pushes
eta2 harder (the cross effect DRIFT[2,1] rises
by 0.3 per standard deviation of age). A second covariate,
w, is unrelated to anything.
gen2 <- generator(DRIFT = matrix(c(-0.6, "drift21||||age=0.3", 0, -0.4), 2, 2),
MANIFESTMEANS = c("mm1|param|TRUE|1|age=0.6", "mm2|param|TRUE|1"),
TIpredNames = c("age", "w"))
d2 <- data.frame(ctGenerate(gen2, n = 150, Tpoints = 30, dtmean = 1,
logdtsd = 0.5, backend = "julia", values = c(drift21 = 0.25, mm1 = 0, mm2 = 0)))
covariates <- unique(d2[, c("id", "age", "w")])
head(covariates)
#> id age w
#> 1 1 -1.3292073 0.4373329
#> 31 2 -0.8696768 1.4133750
#> 61 3 -0.8854495 0.1094298
#> 91 4 1.6697247 1.0385711
#> 121 5 2.0697549 -0.8508909
#> 151 6 0.9137555 0.6517414The model is fitted without either covariate, and the covariates are
handed to ctScoreTest() afterwards, one row per
subject.
fit2 <- ctFit(d2[, c("id", "time", "Y1", "Y2")], model, backend = "julia", cores = 2)
st2 <- ctScoreTest(fit2, covariates = covariates, tests = "covariates")
st2
#> Score tests at the fitted model (laplace route, 150 subjects)
#>
#> Covariate age (linear): joint LM = 128, df = 17, p = 5e-04
#> parameter estimate se estimate_natural p p_adj class
#> mm_Y1 0.0725 0.00833 0.7250 <5e-04 0.00425 <NA>
#> drift_eta2_eta1 0.3550 0.06990 0.3550 <5e-04 0.00425 <NA>
#> mm_Y2 0.0133 0.00773 0.1330 0.0925 0.39300 <NA>
#> drift_eta1 0.1420 0.09120 0.1350 0.1320 0.39300 A
#> popsd_T0m_eta1 0.0634 0.04260 NA 0.1520 0.39300 <NA>
#> mvarY1 0.0792 0.05570 0.0765 0.1550 0.39300 A
#> diff_eta1 -0.0928 0.06870 -0.1340 0.1740 0.39300 A
#> T0m_eta2 0.0101 0.00749 0.1010 0.1850 0.39300 <NA>
#> popsd_mm_Y1 0.0271 0.02090 NA 0.2080 0.39400 <NA>
#> diff_eta2_eta1 0.0474 0.04890 0.0474 0.3380 0.56900 <NA>
#>
#> Covariate w (linear): joint LM = 18.2, df = 17, p = 0.624
#> parameter estimate se estimate_natural p p_adj class
#> T0m_eta1 0.0137 0.00861 0.1370 0.118 0.904 <NA>
#> T0m_eta2 -0.0119 0.00851 -0.1190 0.164 0.904 <NA>
#> diff_eta1 0.1190 0.10100 0.1720 0.238 0.904 A
#> drift_eta1 -0.1770 0.16500 -0.1690 0.276 0.904 A
#> mvarY1 -0.0731 0.06630 -0.0706 0.278 0.904 A
#> popsd_mm_Y1 -0.0190 0.02340 NA 0.426 0.904 <NA>
#> drift_eta2_eta1 -0.0420 0.05740 -0.0420 0.460 0.904 <NA>
#> popsd_T0m_eta1 0.0364 0.04840 NA 0.477 0.904 <NA>
#> mvarY2 0.0122 0.01770 0.0114 0.492 0.904 <NA>
#> drift_eta1_eta2 0.0220 0.04890 0.0220 0.664 0.904 <NA>For each covariate, the joint test asks whether it matters anywhere;
then each parameter’s row gives the effect a refit with that covariate
would be expected to estimate, per unit of the covariate.
age is found, on the intercept of Y1 and on
the cross effect, at about the sizes it was generated with;
w is not. Two things make that attribution work. The
effects are estimated jointly, so a covariate that moves one parameter
is not read off another that happens to be correlated with it. And they
are estimated with the fitted model’s parameters held where the fit left
them: the model without age has already absorbed some of
its effect – the intercepts’ standard deviation has grown to take in the
age differences – and an estimate that lets those parameters move too
extrapolates the model’s curvature across that whole change.
The tests also cover the random effects’ standard deviations (the
popsd_ rows): whether subjects differ more at some ages
than others. A covariate can also be tested as a step at an unknown
threshold, form = "threshold", which scans the cuts and
accounts for having chosen the best one: an effect that switches on
above some age, say.
The refit puts age into the model as a time independent
predictor, with effects on every parameter:
model2b <- ctModel(type = "ct", silent = TRUE,
manifestNames = c("Y1", "Y2"), latentNames = c("eta1", "eta2"),
LAMBDA = diag(2), TIpredNames = "age")
fit2b <- ctFit(d2, model2b, backend = "julia", cores = 2)
effects <- summary(fit2b)$tipreds
effects[grepl("mm_Y1|drift_eta2_eta1", rownames(effects)), ]
#> mean sd 2.5% 50% 97.5% mode z
#> tip_age_drift_eta2_eta1 0.320 0.037 0.249 0.318 0.396 0.319 8.562
#> tip_age_mm_Y1 0.712 0.088 0.541 0.707 0.893 0.715 8.0883. Dynamics that depend on the state
Last, eta2 moderates how eta1 regulates:
when eta2 is high, eta1 returns to baseline
more slowly. The drift of eta1 is
-0.6 + 0.3 * tanh(eta2), so it ranges between -0.9 and
-0.3, and near eta2 = 0 it rises by about 0.3 per unit of
eta2.
gen3 <- generator(DRIFT = matrix(c("-0.6 + 0.3 * tanh(eta2)", 0.25, 0, -0.4), 2, 2))
d3 <- data.frame(ctGenerate(gen3, n = 150, Tpoints = 30, dtmean = 1,
logdtsd = 0.5, backend = "julia", values = c(mm1 = 0, mm2 = 0)))The fitted model is linear: each drift coefficient is a constant.
fit3 <- ctFit(d3, model, backend = "julia", cores = 2)
st3 <- ctScoreTest(fit3, tests = "statedependence")
st3
#> Score tests at the fitted model (laplace route, 150 subjects)
#>
#> State dependence of the drift:
#> equation LM df p
#> eta1 27.300 3 <5e-04
#> eta2 0.529 3 0.8990
#> all 29.900 6 <5e-04
#>
#> term equation LM p p_adj estimate se
#> eta1*eta2 eta1 25.7000 <5e-04 2.39e-06 0.19400 0.0383
#> eta1*eta1 eta1 9.0300 2.66e-03 7.97e-03 0.06970 0.0232
#> eta2*eta2 eta1 1.1100 2.91e-01 5.82e-01 0.03550 0.0336
#> eta1*eta2 eta2 0.2750 6.00e-01 8.44e-01 -0.01420 0.0271
#> eta2*eta2 eta2 0.1450 7.03e-01 8.44e-01 -0.00663 0.0174
#> eta1*eta1 eta2 0.0242 8.76e-01 8.76e-01 NA NA
#> reads
#> DRIFT[eta1,eta1] varies with eta2, or DRIFT[eta1,eta2] with eta1
#> DRIFT[eta1,eta1] varies with eta1
#> DRIFT[eta1,eta2] varies with eta2
#> DRIFT[eta2,eta1] varies with eta2, or DRIFT[eta2,eta2] with eta1
#> DRIFT[eta2,eta2] varies with eta2
#> DRIFT[eta2,eta1] varies with eta1Each term is one drift coefficient varying linearly with one state.
The term eta1*eta2 in the equation of eta1
stands out: either the drift of eta1 changes with
eta2 – which is what generated the data – or the effect of
eta2 on eta1 changes with eta1.
At this order the two are the same term, so the table names both, and
which reading to fit is a substantive choice. The estimate is the change
in the drift coefficient per unit of the state. Writing the dependence
into the model:
model3b <- ctModel(type = "ct", silent = TRUE,
manifestNames = c("Y1", "Y2"), latentNames = c("eta1", "eta2"),
LAMBDA = diag(2),
DRIFT = matrix(c("dr11 + dstate * eta2", "dr21", "dr12", "dr22"), 2, 2),
PARS = c("dr11|-log1p_exp(-param)", "dstate"))
fit3b <- ctFit(d3, model3b, backend = "julia", cores = 2)
c(loglik_without = fit3$estimate$loglik, loglik_with = fit3b$estimate$loglik)
#> loglik_without loglik_with
#> -11175.02 -11161.14
summary(fit3b)$popmeans[c("dr11", "dstate"), ]
#> mean sd 2.5% 50% 97.5% mode
#> dr11 -0.588 0.052 -0.692 -0.586 -0.494 -0.587
#> dstate 0.175 0.034 0.110 0.174 0.242 0.176The refit’s dstate is a little smaller than the one-step
estimate: the generating tanh flattens away from zero, and
a term linear in eta2 averages over its whole range. The
first screen also flagged a second, weaker term, eta1*eta1
in the equation of eta2. That one was an echo – the misfit
in eta1’s dynamics reaches eta2 through the
cross effect – and testing the refitted model shows nothing left:
ctScoreTest(fit3b, tests = "statedependence")
#> Score tests at the fitted model (laplace route, 150 subjects)
#>
#> State dependence of the drift:
#> equation LM df p
#> eta1 1.90 3 0.609
#> eta2 4.96 3 0.184
#> all 6.07 6 0.440
#>
#> term equation LM p p_adj estimate se
#> eta1*eta1 eta2 4.57e+00 0.0325 0.195 -0.0519 0.0243
#> eta1*eta2 eta2 2.70e+00 0.1000 0.301 -0.0438 0.0267
#> eta1*eta1 eta1 1.08e+00 0.2990 0.399 0.0249 0.0240
#> eta2*eta2 eta2 9.77e-01 0.3230 0.399 -0.0176 0.0178
#> eta2*eta2 eta1 9.40e-01 0.3320 0.399 -0.0331 0.0341
#> eta1*eta2 eta1 1.10e-10 1.0000 1.000 NA NA
#> reads
#> DRIFT[eta2,eta1] varies with eta1
#> DRIFT[eta2,eta1] varies with eta2, or DRIFT[eta2,eta2] with eta1
#> DRIFT[eta1,eta1] varies with eta1
#> DRIFT[eta2,eta2] varies with eta2
#> DRIFT[eta1,eta2] varies with eta2
#> DRIFT[eta1,eta1] varies with eta2, or DRIFT[eta1,eta2] with eta1The term already in the model now has no score left at all, and no other term stands out.
The test works because the filter evaluates each candidate term at the state it predicts from earlier observations. A weight that has not seen the observation being scored keeps that observation’s contribution to the score at mean zero when the model is right. Using smoothed or updated state estimates as covariates instead – regressing something on the estimated states after the fit – does not have that property, and rejects correct models: in simulation, nearly always.
The polynomial terms assume the dependence is smooth enough to show
in a linear term. tests = "instability" asks without
assuming a form. It takes each observation’s score – how much that
observation pulls each parameter – and orders the observations by time,
and by each latent’s state as predicted from earlier observations. Under
the model the running sum of those scores wanders like a Brownian bridge
whatever the order; a parameter that changes over the study, or differs
between low and high levels of a state, bends it.
ctScoreTest(fit3, tests = "instability")
#> Score tests at the fitted model (laplace route, 150 subjects)
#>
#> Stability across the data (Nyblom), by ordering:
#> ordering L df p
#> time 1.72 11 0.2040
#> state eta1 1.84 11 0.2860
#> state eta2 4.28 11 <5e-04
#>
#> ordering parameter L p p_adj
#> state eta2 diff_eta1 1.850 <5e-04 0.00550
#> state eta2 mvarY2 1.120 0.0015 0.00825
#> state eta2 drift_eta1 0.977 0.0045 0.01650
#> time diff_eta2_eta1 0.782 0.0045 0.04950
#> state eta2 mvarY1 0.786 0.0065 0.01790
#> state eta2 diff_eta2 0.852 0.0115 0.02530
#> time drift_eta2_eta1 0.329 0.0720 0.28000
#> time mm_Y1 0.104 0.0765 0.28000
#> state eta1 drift_eta2_eta1 0.394 0.0825 0.46500
#> state eta1 diff_eta1 0.369 0.0845 0.46500Ordered by eta2’s predicted state, the drift of
eta1 stands out, which is the moderation that generated the
data, found without being told its form. Ordered by time nothing
changes. The test needs each observation’s score, a pass of the filter
per eight parameters, so it is not run unless asked for.
Everything at once
By default ctScoreTest() runs every family: the three
above (covariates when given) and a fourth, the cells the model fixes at
zero – here the measurement error covariance and the cross loadings –
each tested as though freed. On the last dataset:
ctScoreTest(fit3)
#> Score tests at the fitted model (laplace route, 150 subjects)
#>
#> Unmodelled heterogeneity (LM: the variance with its covariances, against an
#> equal mixture of chi-square on df - 1 and df; z: the variance alone, one sided;
#> p: the stronger of the two; sd: the one-step estimate, raw scale):
#> parameter LM df z p p_adj sd sd_natural
#> diff_eta2 10.90 5 2.300 0.0185 0.159 0.1010 0.1070
#> drift_eta1_eta2 5.62 5 1.770 0.0535 0.159 0.1830 0.1830
#> diff_eta1 6.47 5 1.720 0.0615 0.159 0.0598 0.0921
#> drift_eta2 11.10 5 0.393 0.0705 0.159 0.0846 0.0517
#> mvarY1 9.85 5 0.858 0.1020 0.181 0.0528 0.0502
#> mvarY2 8.36 5 1.420 0.1200 0.181 0.0527 0.0525
#> diff_eta2_eta1 6.42 5 0.674 0.3540 0.456 0.0943 0.0943
#> drift_eta2_eta1 5.53 5 -0.519 0.4550 0.512 NA NA
#> drift_eta1 3.69 5 -0.209 0.7340 0.734 NA NA
#> Covariances with: T0m_eta1, T0m_eta2, mm_Y1, mm_Y2
#>
#> A new dimension of variation: largest eigenvalue 6.3, p = 0.097; direction drift_eta2 1, diff_eta2_eta1 -0.87, diff_eta2 0.85, drift_eta1 0.82, mvarY1 0.63
#>
#> State dependence of the drift:
#> equation LM df p
#> eta1 27.300 3 <5e-04
#> eta2 0.529 3 0.9210
#> all 29.900 6 <5e-04
#>
#> term equation LM p p_adj estimate se
#> eta1*eta2 eta1 25.7000 <5e-04 2.39e-06 0.19400 0.0383
#> eta1*eta1 eta1 9.0300 2.66e-03 7.97e-03 0.06970 0.0232
#> eta2*eta2 eta1 1.1100 2.91e-01 5.82e-01 0.03550 0.0336
#> eta1*eta2 eta2 0.2750 6.00e-01 8.44e-01 -0.01420 0.0271
#> eta2*eta2 eta2 0.1450 7.03e-01 8.44e-01 -0.00663 0.0174
#> eta1*eta1 eta2 0.0242 8.76e-01 8.76e-01 NA NA
#> reads
#> DRIFT[eta1,eta1] varies with eta2, or DRIFT[eta1,eta2] with eta1
#> DRIFT[eta1,eta1] varies with eta1
#> DRIFT[eta1,eta2] varies with eta2
#> DRIFT[eta2,eta1] varies with eta2, or DRIFT[eta2,eta2] with eta1
#> DRIFT[eta2,eta2] varies with eta2
#> DRIFT[eta2,eta1] varies with eta1
#>
#> State dependence of the innovation sds:
#> equation LM df p
#> eta1 25.20 2 <5e-04
#> eta2 5.13 2 0.0875
#> all 31.30 4 <5e-04
#>
#> term LM p p_adj estimate se
#> sd(eta1) x exp(v * eta2) 22.400 <5e-04 8.95e-06 0.1710 0.0362
#> sd(eta1) x exp(v * eta1) 7.490 6.22e-03 1.24e-02 0.0945 0.0345
#> sd(eta2) x exp(v * eta2) 5.030 2.49e-02 3.32e-02 -0.1090 0.0486
#> sd(eta2) x exp(v * eta1) 0.577 4.48e-01 4.48e-01 -0.0302 0.0397
#>
#> Fixed cells, freed: joint LM = 2.77, df = 1, p = 0.1
#> cell LM p p_adj estimate se
#> MANIFESTVAR[Y2,Y1] 2.77 0.096 0.096 0.087 0.0523
#>
#> Parameters depending on a subject's level (a random intercept):
#> level LM df p
#> mm_Y1 14.30 9 0.205
#> mm_Y2 9.75 9 0.468
#> all 24.90 18 0.358
#>
#> term LM p p_adj estimate se
#> diff_eta2 ~ mm_Y1 7.87 0.00502 0.075 -0.0413 0.0147
#> mvarY1 ~ mm_Y1 6.54 0.01050 0.075 -0.0280 0.0109
#> drift_eta2 ~ mm_Y1 6.24 0.01250 0.075 -0.0385 0.0154
#> mvarY2 ~ mm_Y1 4.30 0.03810 0.171 -0.0218 0.0105
#> drift_eta1 ~ mm_Y2 2.80 0.09450 0.294 0.0384 0.0230
#> diff_eta2_eta1 ~ mm_Y1 2.55 0.11000 0.294 NA NA
#> diff_eta1 ~ mm_Y1 2.49 0.11400 0.294 NA NA
#> drift_eta1_eta2 ~ mm_Y1 1.60 0.20700 0.465 NA NA
#> diff_eta1 ~ mm_Y2 1.41 0.23400 0.469 NA NA
#> diff_eta2_eta1 ~ mm_Y2 1.26 0.26200 0.472 NA NAThe state dependence stands out. The heterogeneity tests stay quiet here, but they need not: a drift that changes with the state can also look, subject by subject, like a drift that differs between subjects, and in simulations a missing nonlinearity did move them. A badly misspecified model can light up several tests at once, so read them as a map: relax what stands out most, refit, and test again.
Do the model’s predictions look like the data?
The usual check of a dynamic model compares the data with data the
fitted model generates: the autocorrelations and cross-correlations (ACF
and CCF), how much subjects differ from each other, and so on.
ctFitAdequacy() makes those comparisons tests, in families
aimed at different departures.
Most of the families work on the one-step-ahead prediction errors, which under the model are independent and standard normal however the observations are spaced, so they need no generated data. Each variable’s error is taken given the others observed at the same time, so a result names a variable.
-
residuals: the errors’ correlations within subjects, for every pair of a subject’s observations, binned by the time between them. This is the residual ACF and CCF in continuous time, and a slow process the model lacks shows as correlation over the gaps it spans. -
latent: the same for each latent process: how much each observation moves the predicted state, standardised, correlated with the same subject’s moves at other times. A model with many indicators per process can leave each item’s errors nearly uncorrelated while the processes themselves are misspecified, a cross effect missing, say; one statistic per process rather than per item does not dilute that. -
volatility: the same for the squared errors, for variance that clusters in time. -
pairs: whether two variables’ errors at the same time correlate as the model says, for a wrong measurement error covariance or loading. -
periodic: a cycle the model lacks, such as a daily rhythm: the periodogram of each variable’s errors and each process’s updates at a period. It depends only on the time between observations, so it finds a rhythm whatever its phase in each subject. Name the periods that matter (periods = 24with time in hours), or by default it scans. -
tvcovariates, giventvcovariates: whether time varying variables the model does not use predict its errors, at the same and the previous observation. -
distribution: skewness, kurtosis and outlying occasions. -
fit: whether some subjects are fitted much worse than the model allows. -
dependence: whether an error or its size depends, in any form, on what came before it: the predicted states, the previous errors, the time since the last observation. -
effects: on a model whose random effects are integrated byintoverpop = "laplace", whether their normal distribution fits. Each subject’s posterior probability that its effect lies below a cut, at cuts along each axis, is averaged over subjects; under the model that average is the normal’s, and a skewed or bimodal distribution of the subjects moves it. -
scores, when asked for: whether each observation’s score is uncorrelated with the scores of the same subject’s other observations, which holds under the model whatever is being looked for. That makes it a check for dynamic misspecification of any kind, but a weak one, spread over many statistics; the targeted tests above find what they cover far more often.
The error statistics account for the parameters having been estimated
from the same data. Each subject’s contribution is projected on its
scores, which removes what estimation did to the statistic, as the score
tests do. Three more families compare statistics of the data with
generated data: the spread over subjects of their own means, standard
deviations and lag-one correlations; whether change depends on level
nonlinearly; and, given covariates, whether subjects’ statistics depend
on them. With many observations every misfit is significant, so the size
of what remains is reported too, as root mean square residual
correlations, of the items and of the latent processes, with intervals
and a test that each is at most close (0.05 by
default).
On the second example’s fit, which lacks age, the checks
fail, through the subjects’ spread and their relation to age:
ctFitAdequacy(fit2, covariates = covariates)
#> Adequacy: misfit detected by 2 of 12 families at alpha = 0.05
#> (estimation removed by: derivative; 100 generated datasets)
#>
#> family statistics worst worst_z p p_adj
#> residuals 16 Y2~Y1 gap 4.87-10.20 1.610 8.06e-01 1.00e+00
#> latent 16 eta2~eta2 gap >10.2 -1.360 9.10e-01 1.00e+00
#> volatility 8 Y1^2 gap 2.54- 4.87 0.990 9.35e-01 1.00e+00
#> pairs 1 Y1:Y2 0.369 7.01e-01 1.00e+00
#> periodic 60 Y2 period 4.05 2.440 5.46e-01 1.00e+00
#> distribution 8 distance spread 1.530 3.94e-01 1.00e+00
#> fit 2 spread of unit fit 2.580 3.10e-02 3.10e-01
#> effects 20 P(u4 < 1.5) -1.730 6.56e-01 1.00e+00
#> dependence 4 dcor(Y2, past) 1.110 6.50e-01 1.00e+00
#> subjects 8 sd(lag1_Y1_Y2) 5.990 2.95e-07 3.25e-06
#> nonlinear 10 cor(dY1^2,Y2) 2.760 7.57e-02 6.81e-01
#> covariates 16 cor(age,lag1_Y1_Y2) 7.860 8.64e-11 1.04e-09
#>
#> Size of what remains (root mean square, bias-corrected; p_close tests that it is at least close):
#> measure estimate lower upper close
#> residual correlation, all gaps 0 0 0.00424 0.05
#> residual correlation, shortest gaps 0 0 0.00525 0.05
#> latent residual correlation, all gaps 0 0 0.00217 0.05
#> latent residual correlation, shortest gaps 0 0 0.00311 0.05
#> within-row correlation 0 0 0.00375 0.05
#> p_close
#> 0
#> 0
#> 0
#> 0
#> 0On the linear fit to the state-dependent data:
ctFitAdequacy(fit3)
#> Adequacy: no family detects misfit at alpha = 0.05
#> (estimation removed by: derivative; 100 generated datasets)
#>
#> family statistics worst worst_z p p_adj
#> residuals 16 Y1~Y2 gap 2.58- 4.91 1.59 0.710 1.000
#> latent 16 eta1~eta2 gap <=2.58 -1.41 0.800 1.000
#> volatility 8 Y2^2 gap 2.58- 4.91 1.91 0.326 1.000
#> pairs 1 Y1:Y2 1.27 0.109 1.000
#> periodic 60 Y1 period 2.35 2.28 0.816 1.000
#> distribution 8 skew(Y2) -2.13 0.831 1.000
#> fit 2 spread of unit fit 2.06 0.064 0.704
#> effects 20 P(u1 < 1.5) -1.48 0.857 1.000
#> dependence 4 dcor(Y1^2, past) 1.46 0.427 1.000
#> subjects 8 sd(lag1_Y2_Y1) 2.13 0.996 1.000
#> nonlinear 10 cor(dY2^2,Y2) -2.25 0.100 1.000
#>
#> Size of what remains (root mean square, bias-corrected; p_close tests that it is at least close):
#> measure estimate lower upper close
#> residual correlation, all gaps 0.00000 0 0.00890 0.05
#> residual correlation, shortest gaps 0.00000 0 0.00726 0.05
#> latent residual correlation, all gaps 0.00000 0 0.00759 0.05
#> latent residual correlation, shortest gaps 0.00000 0 0.00888 0.05
#> within-row correlation 0.00457 0 0.01400 0.05
#> p_close
#> 0.00e+00
#> 0.00e+00
#> 0.00e+00
#> 0.00e+00
#> 1.07e-171Every family passes, though this model is missing the state dependence the score test found at p = 5e-6. A moderated drift, averaged over the data, still decays much like a single exponential, so the errors it leaves are nearly uncorrelated: the root mean square residual correlation is indistinguishable from zero. The checks are worth having for what lies outside every score test’s reach: a slow process the model lacks, variance that clusters, errors with heavy tails, a process of the wrong order. In simulation the residual correlations over time gaps found a missing slow process (a time constant ten times the fast one’s) in 65 of 100 datasets, where correlations at a few lags of the observations found it in 5. But passing the checks is necessary rather than sufficient, which is why the next section screens with the score tests and stops on the checks.
Automating the search
ctModelImprove() puts the tests into a loop. It screens
the fit, refits with the most strongly indicated relaxations one at a
time (warm started from the current estimates), and keeps the best if
the refit’s likelihood confirms it. It repeats until nothing more is
indicated, then checks the model with ctFitAdequacy(). If a
check fails, it tries structural changes the score tests cannot reach: a
slow process driving a latent (ctModelSlowComponent()),
tried first for a latent whose updates stay correlated over long gaps; a
daily or other cycle at the period the periodic family
points to (ctModelPeriodicComponent()); a process of its
own for an item whose errors stay correlated with its own
(ctModelUniqueProcess()); making a process second order
(ctModelHigherOrder()); or models you supply. Finally it
tries dropping each change it kept, in case a later one made it
redundant.
Here the data contain two of the departures above at once:
age acts on the intercept of Y1 and on the
cross effect, eta2 moderates how eta1
regulates, and w is unrelated.
gen4 <- generator(DRIFT = matrix(c("-0.6 + 0.3 * tanh(eta2)", "drift21||||age=0.3",
0, -0.4), 2, 2),
MANIFESTMEANS = c("mm1|param|TRUE|1|age=0.6", "mm2|param|TRUE|1"),
TIpredNames = c("age", "w"))
d4 <- data.frame(ctGenerate(gen4, n = 150, Tpoints = 30, dtmean = 1,
logdtsd = 0.5, backend = "julia", values = c(drift21 = 0.25, mm1 = 0, mm2 = 0)))
fit4 <- ctFit(d4[, c("id", "time", "Y1", "Y2")], model, backend = "julia", cores = 2)
imp <- ctModelImprove(fit4, covariates = unique(d4[, c("id", "age", "w")]))
imp
#> Model improvement: stopped because adequate
#> Log likelihood -11322.58 -> -11146.26 (23 -> 26 parameters)
#>
#> round change screen_p dloglik dpar
#> 1 age as a predictor of mm_Y1, drift_eta2_eta1 1.00e-03 162.0 2
#> 2 eta1*eta2 in the drift of eta1 1.78e-06 13.2 1
#> prune drop age as a predictor of mm_Y1, drift_eta2_eta1 NA -159.0 -2
#> prune drop eta1*eta2 in the drift of eta1 NA NA -1
#> done note
#> TRUE LR p = 2.91e-71 against 0.0167
#> TRUE LR p = 2.71e-07 against 0.025
#> FALSE LR p = 1.42e-69 against 0.05
#> FALSE kept without a refit: Wald 26.4 against 7.68
#>
#> Adequacy: no family detects misfit at alpha = 0.05
#> (estimation removed by: plugin; 100 generated datasets)
#>
#> family statistics worst worst_z p p_adj
#> residuals 16 Y2~Y1 gap >10.1 -1.980 0.614 1
#> latent 16 eta2~eta1 gap >10.1 -1.780 0.671 1
#> volatility 8 Y2^2 gap <=2.56 -1.470 0.775 1
#> pairs 1 Y1:Y2 -1.220 0.173 1
#> periodic 56 Y1 period 5.36 2.590 0.299 1
#> distribution 9 distance 0.892 0.945 1
#> fit 2 spread of unit fit -0.560 0.697 1
#> effects 20 P(u1 < 1.5) -1.410 0.874 1
#> dependence 4 dcor(Y2, past) 1.080 0.687 1
#> subjects 8 sd(lag1_Y2_Y2) 2.260 0.876 1
#> nonlinear 10 cor(dY2,Y1*Y1) -0.984 0.425 1
#> covariates 8 cor(w,lag1_Y2_Y1) 1.320 0.894 1
#>
#> Size of what remains (root mean square, bias-corrected; p_close tests that it is at least close):
#> measure estimate lower upper close
#> residual correlation, all gaps 0.00000 0 0.00521 0.05
#> residual correlation, shortest gaps 0.00000 0 0.01260 0.05
#> latent residual correlation, all gaps 0.00000 0 0.00562 0.05
#> latent residual correlation, shortest gaps 0.00389 0 0.01390 0.05
#> within-row correlation 0.00488 0 0.01600 0.05
#> p_close
#> 0.00e+00
#> 2.94e-212
#> 0.00e+00
#> 2.37e-166
#> 1.34e-97Each round shows what was tried, the score test that proposed it, and
what the refit’s likelihood said. The first round kept age,
with effects on the cross effect, the intercept of Y1 and
the initial level of eta1; it also tried a random effect on
the cross effect, which the score tests had flagged because subjects of
different ages differ in it – a covariate that explains the variation
beats a random effect that only describes it. The second round kept the
moderation, and turned down the dimension test’s suggestion that three
dynamic parameters vary together, whose refit gained 8 log likelihood
units for 18 parameters. Then nothing more was indicated, every adequacy
check passed, and trying each kept change out again showed both were
still needed. The effects found are near those that generated the data
(0.3 on the cross effect, 0.6 on the intercept of Y1). The
effect on the initial level of eta1 was not generated: it
is small, and only just significant – a search that decides at
alpha = 0.05 keeps such things about one time in twenty,
and alpha = 0.01 is the stricter choice.
The final fit is an ordinary fit:
s <- summary(imp$fit)
s$tipreds
#> mean sd 2.5% 50% 97.5% mode z
#> tip_age_drift_eta2_eta1 0.310 0.023 0.267 0.311 0.355 0.312 13.333
#> tip_age_mm_Y1 0.611 0.074 0.462 0.613 0.751 0.611 8.308
s$popmeans[grepl("^sdep", rownames(s$popmeans)), ]
#> mean sd 2.5% 50% 97.5% mode
#> sdep_eta1_eta1_eta2 0.12 0.024 0.073 0.119 0.171 0.121The model was chosen by the data it is fitted to, so its standard
errors and tests are too optimistic about the structure the search
added. Treat that structure as a hypothesis, and confirm it on other
data. Within one dataset, holdout = 0.3 sets aside 30 per
cent of the subjects: the search runs on the rest, and each change it
keeps is then tested on the held-out subjects alone. With a large
dataset, close = 0.05 also stops the search once the
residual correlations that remain are not significantly larger than
that, where every small misfit would otherwise be significant.
How the tests work, briefly
- The scores are those of the objective the model was fitted with, so they integrate over random effects exactly as the fit did. A covariate’s effect adds to a parameter’s raw value, so its score is the covariate times each subject’s score in that parameter – any number of covariates costs a few matrix products after one set of per-subject scores and Hessians.
- A random effect’s variance has, at zero, the score , and its covariance with an effect already there : the information matrix test in those parameters, which is the score test for neglected heterogeneity. Every subject already has one such effect, their initial state, so the covariances are tested even in a model with no random effects.
- State dependence is tested on the fitted model extended with the drift terms at zero, which reproduces the fit exactly there.
- The fitted model’s curvature projects the fitted parameters out of each test and sizes the one-step estimates; each test’s variance is the spread of the subjects’ contributions, and its p-value comes from resampling subjects. That keeps a test valid when the model is wrong in some other way – the case a screen is for – where a variance from the model’s own curvature would make unrelated tests reject too.
The tests need subjects as independent units, so a model with levels
above the subject can be tested for state dependence only. See
?ctScoreTest for the details.