Skip to contents

A fitted model can be wrong in ways its own estimates do not show. Subjects may differ in a parameter the model holds fixed; a parameter may depend on a covariate that was never put in the model; the dynamics may depend on where the process is. ctScoreTest() asks of one fitted model, without refitting it, whether any of these relaxations would fit better, and if so where:

  • heterogeneity: does a parameter differ between subjects, so that it should be individually varying?
  • covariates: does a parameter depend on a subject-level covariate the model does not contain – including the size of a random effect, not only its mean?
  • state dependence: does a drift coefficient change with the level of a latent state, so that the dynamics are nonlinear?
  • volatility: does the size of a process’s random shocks change with the level of a state – negative affect more variable when it is high, say?

Each is a score (Lagrange multiplier) test: the slope of the log likelihood, at the fitted model, in the direction of the relaxation. It needs a few evaluations of the fitted model’s gradient rather than a new fit per candidate, so any number of candidates can be screened after one fit, and each test also says roughly how large the relaxation would be. The tests are screens: they point at what to relax, and the refit says how much it matters.

The three examples below each generate data with one thing the fitted model lacks, fit the model without it, ask ctScoreTest(), and then refit with what it found. The sections after them add a fourth family of tests (cells the model fixes at zero), the posterior predictive checks as tests (ctFitAdequacy()), and a search that combines the two and does the refitting itself (ctModelImprove()).

The process

Two latent processes, each measured by one indicator, observed 30 times at irregular intervals for 150 subjects. eta1 regulates towards its baseline at rate 0.6, eta2 at 0.4, and eta1 pushes eta2 up (a cross effect of 0.25). Each subject has their own measurement intercepts. The generating model is written once, and each example changes one cell of it.

generator <- function(DRIFT, MANIFESTMEANS = c("mm1|param|TRUE|1", "mm2|param|TRUE|1"),
  TIpredNames = NULL) {
  ctModel(type = "ct", silent = TRUE,
    manifestNames = c("Y1", "Y2"), latentNames = c("eta1", "eta2"),
    LAMBDA = diag(2), DRIFT = DRIFT, CINT = matrix(0, 2, 1),
    DIFFUSION = matrix(c(0.8, 0, 0, 0.6), 2, 2), MANIFESTVAR = diag(0.5, 2),
    MANIFESTMEANS = matrix(MANIFESTMEANS, 2, 1),
    T0MEANS = matrix(0, 2, 1), T0VAR = diag(1, 2), TIpredNames = TIpredNames)
}

The model fitted in every example is ctsem’s default for two latents with one indicator each: every drift, diffusion and measurement error parameter free and shared by all subjects, and individually varying measurement intercepts and initial states.

model <- ctModel(type = "ct", silent = TRUE,
  manifestNames = c("Y1", "Y2"), latentNames = c("eta1", "eta2"),
  LAMBDA = diag(2))

1. Subjects who differ in a parameter

Here the subjects differ in how quickly eta1 regulates: the drift DRIFT[1,1] varies between subjects around -0.6, with a standard deviation of about 0.25.

gen1 <- generator(DRIFT = matrix(c("drift11|-log1p_exp(-param)|TRUE|0.6", 0.25,
  0, -0.4), 2, 2))
d1 <- ctGenerate(gen1, n = 150, Tpoints = 30, dtmean = 1, logdtsd = 0.5,
  backend = "julia", values = c(drift11 = -0.6, mm1 = 0, mm2 = 0))
truedrift <- attr(d1, "subjectpars")$drift11
c(mean = mean(truedrift), sd = sd(truedrift))
#>       mean         sd 
#> -0.6345464  0.2660944
d1 <- data.frame(d1)

The default model shares one drift among all subjects.

fit1 <- ctFit(d1, model, backend = "julia", cores = 2)
st1 <- ctScoreTest(fit1, tests = "heterogeneity")
st1
#> Score tests at the fitted model (laplace route, 150 subjects)
#> 
#> Unmodelled heterogeneity (LM: the variance with its covariances, against an
#> equal mixture of chi-square on df - 1 and df; z: the variance alone, one sided;
#> p: the stronger of the two; sd: the one-step estimate, raw scale):
#>        parameter    LM df      z      p  p_adj      sd sd_natural
#>       drift_eta1 19.40  5  2.580 0.0055 0.0292 0.19100    0.18400
#>  drift_eta1_eta2 16.30  5  1.160 0.0065 0.0292 0.16300    0.16300
#>        diff_eta2 14.20  5 -0.104 0.0175 0.0525 0.00187    0.00217
#>        diff_eta1  9.59  5  1.790 0.0420 0.0918 0.10400    0.15600
#>           mvarY2 12.00  5  1.600 0.0510 0.0918 0.05810    0.05690
#>   diff_eta2_eta1  5.00  5  0.776 0.3330 0.5000 0.09630    0.09630
#>           mvarY1  2.61  5  0.656 0.4120 0.5300 0.05140    0.04990
#>  drift_eta2_eta1  2.81  5 -0.157 0.7920 0.8900      NA         NA
#>       drift_eta2  2.48  5 -1.410 0.9020 0.9020      NA         NA
#> Covariances with: T0m_eta1, T0m_eta2, mm_Y1, mm_Y2
#> 
#> A new dimension of variation: largest eigenvalue 6.49, p = 0.0565; direction drift_eta1_eta2 1, drift_eta1 1, drift_eta2_eta1 0.54, diff_eta2_eta1 0.39, diff_eta1 0.38

Each row tests whether that parameter varies over subjects, as ctFit() would fit it if it were made individually varying: its variance together with its covariances with what already varies, here each subject’s initial state and measurement intercepts (the line under the table names them). A variance cannot be negative, so that part is one sided; LM is referred to an equal mixture of chi-squares on df - 1 and df. The covariances matter most for an intercept: one that varies independently of where each subject starts makes subjects drift apart from a common start, whereas the same intercept correlated with the initial state is a random intercept. z is the variance alone, and p the stronger of the two tests.

The drift of eta1 is flagged, on its variance alone (z). Other rows answer too, mostly through their covariances (LM large where z is small): the cross effect of eta2 on eta1, and the diffusion of eta2. That is expected rather than further findings. A drift that varies between subjects, left out of the model, changes how each subject’s data relate to their starting point and to their intercepts, and the other parameters’ scores carry some of that. The test says that the subjects differ, and roughly where; it cannot always say which of several related parameters is the source. The way to find out is to relax the strongest candidate and test again.

The line under the table tests every direction at once: whether the subjects vary along some combination of these parameters, which no single row would isolate when the variation is spread over several of them, its covariances with what already varies held at zero. Here one parameter carries the variation, and its own row is the sharper test. On a model whose population covariance has reduced rank (poprank) the rows test variation outside the shared dimensions, and the line is not given: in simulation the rows found a missing dimension far more often.

model1b <- model
model1b$pars$indvarying[model1b$pars$param %in% "drift_eta1"] <- TRUE
fit1b <- ctFit(d1, model1b, backend = "julia", cores = 2)
c(loglik_without = fit1$estimate$loglik, loglik_with = fit1b$estimate$loglik)
#> loglik_without    loglik_with 
#>      -11177.43      -11166.31
ctScoreTest(fit1b, tests = "heterogeneity")
#> Score tests at the fitted model (laplace route, 150 subjects)
#> 
#> Unmodelled heterogeneity (LM: the variance with its covariances, against an
#> equal mixture of chi-square on df - 1 and df; z: the variance alone, one sided;
#> p: the stronger of the two; sd: the one-step estimate, raw scale):
#>        parameter    LM df      z      p p_adj     sd sd_natural
#>        diff_eta2 14.60  6 -0.127 0.0295 0.236     NA         NA
#>           mvarY2 11.90  6  1.610 0.0950 0.259 0.0581     0.0569
#>  drift_eta1_eta2 11.70  6  0.528 0.0970 0.259 0.1010     0.1010
#>        diff_eta1  7.85  6  1.280 0.1660 0.331 0.0495     0.0757
#>   diff_eta2_eta1  6.03  6  0.921 0.3020 0.484 0.1250     0.1250
#>           mvarY1  5.85  6  0.385 0.5880 0.784     NA         NA
#>  drift_eta2_eta1  1.52  6 -0.527 0.9020 0.961     NA         NA
#>       drift_eta2  2.40  6 -1.410 0.9610 0.961     NA         NA
#> Covariances with: T0m_eta1, T0m_eta2, drift_eta1, mm_Y1, mm_Y2
#> 
#> A new dimension of variation: largest eigenvalue 2.67, p = 0.8; direction diff_eta2_eta1 1, drift_eta1_eta2 0.61, mvarY2 -0.39, mvarY1 0.34, drift_eta2_eta1 0.3

With the drift free to vary the log likelihood rises by about 11, and nothing else stands out once the tests are adjusted for their number (p_adj): the other rows’ signals were the drift’s. (A drift that varies between subjects enters the filter nonlinearly, so ctFit() chose the Laplace route for the refit; the tests work on either.)

2. A covariate the model does not contain

Now each subject has an age (standardised), and age matters twice: older subjects have a higher intercept on Y1, and in older subjects eta1 pushes eta2 harder (the cross effect DRIFT[2,1] rises by 0.3 per standard deviation of age). A second covariate, w, is unrelated to anything.

gen2 <- generator(DRIFT = matrix(c(-0.6, "drift21||||age=0.3", 0, -0.4), 2, 2),
  MANIFESTMEANS = c("mm1|param|TRUE|1|age=0.6", "mm2|param|TRUE|1"),
  TIpredNames = c("age", "w"))
d2 <- data.frame(ctGenerate(gen2, n = 150, Tpoints = 30, dtmean = 1,
  logdtsd = 0.5, backend = "julia", values = c(drift21 = 0.25, mm1 = 0, mm2 = 0)))
covariates <- unique(d2[, c("id", "age", "w")])
head(covariates)
#>     id        age          w
#> 1    1 -1.3292073  0.4373329
#> 31   2 -0.8696768  1.4133750
#> 61   3 -0.8854495  0.1094298
#> 91   4  1.6697247  1.0385711
#> 121  5  2.0697549 -0.8508909
#> 151  6  0.9137555  0.6517414

The model is fitted without either covariate, and the covariates are handed to ctScoreTest() afterwards, one row per subject.

fit2 <- ctFit(d2[, c("id", "time", "Y1", "Y2")], model, backend = "julia", cores = 2)
st2 <- ctScoreTest(fit2, covariates = covariates, tests = "covariates")
st2
#> Score tests at the fitted model (laplace route, 150 subjects)
#> 
#> Covariate age (linear): joint LM = 128, df = 17, p = 5e-04
#>        parameter estimate      se estimate_natural      p   p_adj class
#>            mm_Y1   0.0725 0.00833           0.7250 <5e-04 0.00425  <NA>
#>  drift_eta2_eta1   0.3550 0.06990           0.3550 <5e-04 0.00425  <NA>
#>            mm_Y2   0.0133 0.00773           0.1330 0.0925 0.39300  <NA>
#>       drift_eta1   0.1420 0.09120           0.1350 0.1320 0.39300     A
#>   popsd_T0m_eta1   0.0634 0.04260               NA 0.1520 0.39300  <NA>
#>           mvarY1   0.0792 0.05570           0.0765 0.1550 0.39300     A
#>        diff_eta1  -0.0928 0.06870          -0.1340 0.1740 0.39300     A
#>         T0m_eta2   0.0101 0.00749           0.1010 0.1850 0.39300  <NA>
#>      popsd_mm_Y1   0.0271 0.02090               NA 0.2080 0.39400  <NA>
#>   diff_eta2_eta1   0.0474 0.04890           0.0474 0.3380 0.56900  <NA>
#> 
#> Covariate w (linear): joint LM = 18.2, df = 17, p = 0.624
#>        parameter estimate      se estimate_natural     p p_adj class
#>         T0m_eta1   0.0137 0.00861           0.1370 0.118 0.904  <NA>
#>         T0m_eta2  -0.0119 0.00851          -0.1190 0.164 0.904  <NA>
#>        diff_eta1   0.1190 0.10100           0.1720 0.238 0.904     A
#>       drift_eta1  -0.1770 0.16500          -0.1690 0.276 0.904     A
#>           mvarY1  -0.0731 0.06630          -0.0706 0.278 0.904     A
#>      popsd_mm_Y1  -0.0190 0.02340               NA 0.426 0.904  <NA>
#>  drift_eta2_eta1  -0.0420 0.05740          -0.0420 0.460 0.904  <NA>
#>   popsd_T0m_eta1   0.0364 0.04840               NA 0.477 0.904  <NA>
#>           mvarY2   0.0122 0.01770           0.0114 0.492 0.904  <NA>
#>  drift_eta1_eta2   0.0220 0.04890           0.0220 0.664 0.904  <NA>

For each covariate, the joint test asks whether it matters anywhere; then each parameter’s row gives the effect a refit with that covariate would be expected to estimate, per unit of the covariate. age is found, on the intercept of Y1 and on the cross effect, at about the sizes it was generated with; w is not. Two things make that attribution work. The effects are estimated jointly, so a covariate that moves one parameter is not read off another that happens to be correlated with it. And they are estimated with the fitted model’s parameters held where the fit left them: the model without age has already absorbed some of its effect – the intercepts’ standard deviation has grown to take in the age differences – and an estimate that lets those parameters move too extrapolates the model’s curvature across that whole change.

The tests also cover the random effects’ standard deviations (the popsd_ rows): whether subjects differ more at some ages than others. A covariate can also be tested as a step at an unknown threshold, form = "threshold", which scans the cuts and accounts for having chosen the best one: an effect that switches on above some age, say.

The refit puts age into the model as a time independent predictor, with effects on every parameter:

model2b <- ctModel(type = "ct", silent = TRUE,
  manifestNames = c("Y1", "Y2"), latentNames = c("eta1", "eta2"),
  LAMBDA = diag(2), TIpredNames = "age")
fit2b <- ctFit(d2, model2b, backend = "julia", cores = 2)
effects <- summary(fit2b)$tipreds
effects[grepl("mm_Y1|drift_eta2_eta1", rownames(effects)), ]
#>                          mean    sd  2.5%   50% 97.5%  mode     z
#> tip_age_drift_eta2_eta1 0.320 0.037 0.249 0.318 0.396 0.319 8.562
#> tip_age_mm_Y1           0.712 0.088 0.541 0.707 0.893 0.715 8.088

3. Dynamics that depend on the state

Last, eta2 moderates how eta1 regulates: when eta2 is high, eta1 returns to baseline more slowly. The drift of eta1 is -0.6 + 0.3 * tanh(eta2), so it ranges between -0.9 and -0.3, and near eta2 = 0 it rises by about 0.3 per unit of eta2.

gen3 <- generator(DRIFT = matrix(c("-0.6 + 0.3 * tanh(eta2)", 0.25, 0, -0.4), 2, 2))
d3 <- data.frame(ctGenerate(gen3, n = 150, Tpoints = 30, dtmean = 1,
  logdtsd = 0.5, backend = "julia", values = c(mm1 = 0, mm2 = 0)))

The fitted model is linear: each drift coefficient is a constant.

fit3 <- ctFit(d3, model, backend = "julia", cores = 2)
st3 <- ctScoreTest(fit3, tests = "statedependence")
st3
#> Score tests at the fitted model (laplace route, 150 subjects)
#> 
#> State dependence of the drift:
#>  equation     LM df      p
#>      eta1 27.300  3 <5e-04
#>      eta2  0.529  3 0.8990
#>       all 29.900  6 <5e-04
#> 
#>       term equation      LM        p    p_adj estimate     se
#>  eta1*eta2     eta1 25.7000   <5e-04 2.39e-06  0.19400 0.0383
#>  eta1*eta1     eta1  9.0300 2.66e-03 7.97e-03  0.06970 0.0232
#>  eta2*eta2     eta1  1.1100 2.91e-01 5.82e-01  0.03550 0.0336
#>  eta1*eta2     eta2  0.2750 6.00e-01 8.44e-01 -0.01420 0.0271
#>  eta2*eta2     eta2  0.1450 7.03e-01 8.44e-01 -0.00663 0.0174
#>  eta1*eta1     eta2  0.0242 8.76e-01 8.76e-01       NA     NA
#>                                                             reads
#>  DRIFT[eta1,eta1] varies with eta2, or DRIFT[eta1,eta2] with eta1
#>                                 DRIFT[eta1,eta1] varies with eta1
#>                                 DRIFT[eta1,eta2] varies with eta2
#>  DRIFT[eta2,eta1] varies with eta2, or DRIFT[eta2,eta2] with eta1
#>                                 DRIFT[eta2,eta2] varies with eta2
#>                                 DRIFT[eta2,eta1] varies with eta1

Each term is one drift coefficient varying linearly with one state. The term eta1*eta2 in the equation of eta1 stands out: either the drift of eta1 changes with eta2 – which is what generated the data – or the effect of eta2 on eta1 changes with eta1. At this order the two are the same term, so the table names both, and which reading to fit is a substantive choice. The estimate is the change in the drift coefficient per unit of the state. Writing the dependence into the model:

model3b <- ctModel(type = "ct", silent = TRUE,
  manifestNames = c("Y1", "Y2"), latentNames = c("eta1", "eta2"),
  LAMBDA = diag(2),
  DRIFT = matrix(c("dr11 + dstate * eta2", "dr21", "dr12", "dr22"), 2, 2),
  PARS = c("dr11|-log1p_exp(-param)", "dstate"))
fit3b <- ctFit(d3, model3b, backend = "julia", cores = 2)
c(loglik_without = fit3$estimate$loglik, loglik_with = fit3b$estimate$loglik)
#> loglik_without    loglik_with 
#>      -11175.02      -11161.14
summary(fit3b)$popmeans[c("dr11", "dstate"), ]
#>          mean    sd   2.5%    50%  97.5%   mode
#> dr11   -0.588 0.052 -0.692 -0.586 -0.494 -0.587
#> dstate  0.175 0.034  0.110  0.174  0.242  0.176

The refit’s dstate is a little smaller than the one-step estimate: the generating tanh flattens away from zero, and a term linear in eta2 averages over its whole range. The first screen also flagged a second, weaker term, eta1*eta1 in the equation of eta2. That one was an echo – the misfit in eta1’s dynamics reaches eta2 through the cross effect – and testing the refitted model shows nothing left:

ctScoreTest(fit3b, tests = "statedependence")
#> Score tests at the fitted model (laplace route, 150 subjects)
#> 
#> State dependence of the drift:
#>  equation   LM df     p
#>      eta1 1.90  3 0.609
#>      eta2 4.96  3 0.184
#>       all 6.07  6 0.440
#> 
#>       term equation       LM      p p_adj estimate     se
#>  eta1*eta1     eta2 4.57e+00 0.0325 0.195  -0.0519 0.0243
#>  eta1*eta2     eta2 2.70e+00 0.1000 0.301  -0.0438 0.0267
#>  eta1*eta1     eta1 1.08e+00 0.2990 0.399   0.0249 0.0240
#>  eta2*eta2     eta2 9.77e-01 0.3230 0.399  -0.0176 0.0178
#>  eta2*eta2     eta1 9.40e-01 0.3320 0.399  -0.0331 0.0341
#>  eta1*eta2     eta1 1.10e-10 1.0000 1.000       NA     NA
#>                                                             reads
#>                                 DRIFT[eta2,eta1] varies with eta1
#>  DRIFT[eta2,eta1] varies with eta2, or DRIFT[eta2,eta2] with eta1
#>                                 DRIFT[eta1,eta1] varies with eta1
#>                                 DRIFT[eta2,eta2] varies with eta2
#>                                 DRIFT[eta1,eta2] varies with eta2
#>  DRIFT[eta1,eta1] varies with eta2, or DRIFT[eta1,eta2] with eta1

The term already in the model now has no score left at all, and no other term stands out.

The test works because the filter evaluates each candidate term at the state it predicts from earlier observations. A weight that has not seen the observation being scored keeps that observation’s contribution to the score at mean zero when the model is right. Using smoothed or updated state estimates as covariates instead – regressing something on the estimated states after the fit – does not have that property, and rejects correct models: in simulation, nearly always.

The polynomial terms assume the dependence is smooth enough to show in a linear term. tests = "instability" asks without assuming a form. It takes each observation’s score – how much that observation pulls each parameter – and orders the observations by time, and by each latent’s state as predicted from earlier observations. Under the model the running sum of those scores wanders like a Brownian bridge whatever the order; a parameter that changes over the study, or differs between low and high levels of a state, bends it.

ctScoreTest(fit3, tests = "instability")
#> Score tests at the fitted model (laplace route, 150 subjects)
#> 
#> Stability across the data (Nyblom), by ordering:
#>    ordering    L df      p
#>        time 1.72 11 0.2040
#>  state eta1 1.84 11 0.2860
#>  state eta2 4.28 11 <5e-04
#> 
#>    ordering       parameter     L      p   p_adj
#>  state eta2       diff_eta1 1.850 <5e-04 0.00550
#>  state eta2          mvarY2 1.120 0.0015 0.00825
#>  state eta2      drift_eta1 0.977 0.0045 0.01650
#>        time  diff_eta2_eta1 0.782 0.0045 0.04950
#>  state eta2          mvarY1 0.786 0.0065 0.01790
#>  state eta2       diff_eta2 0.852 0.0115 0.02530
#>        time drift_eta2_eta1 0.329 0.0720 0.28000
#>        time           mm_Y1 0.104 0.0765 0.28000
#>  state eta1 drift_eta2_eta1 0.394 0.0825 0.46500
#>  state eta1       diff_eta1 0.369 0.0845 0.46500

Ordered by eta2’s predicted state, the drift of eta1 stands out, which is the moderation that generated the data, found without being told its form. Ordered by time nothing changes. The test needs each observation’s score, a pass of the filter per eight parameters, so it is not run unless asked for.

Everything at once

By default ctScoreTest() runs every family: the three above (covariates when given) and a fourth, the cells the model fixes at zero – here the measurement error covariance and the cross loadings – each tested as though freed. On the last dataset:

ctScoreTest(fit3)
#> Score tests at the fitted model (laplace route, 150 subjects)
#> 
#> Unmodelled heterogeneity (LM: the variance with its covariances, against an
#> equal mixture of chi-square on df - 1 and df; z: the variance alone, one sided;
#> p: the stronger of the two; sd: the one-step estimate, raw scale):
#>        parameter    LM df      z      p p_adj     sd sd_natural
#>        diff_eta2 10.90  5  2.300 0.0185 0.159 0.1010     0.1070
#>  drift_eta1_eta2  5.62  5  1.770 0.0535 0.159 0.1830     0.1830
#>        diff_eta1  6.47  5  1.720 0.0615 0.159 0.0598     0.0921
#>       drift_eta2 11.10  5  0.393 0.0705 0.159 0.0846     0.0517
#>           mvarY1  9.85  5  0.858 0.1020 0.181 0.0528     0.0502
#>           mvarY2  8.36  5  1.420 0.1200 0.181 0.0527     0.0525
#>   diff_eta2_eta1  6.42  5  0.674 0.3540 0.456 0.0943     0.0943
#>  drift_eta2_eta1  5.53  5 -0.519 0.4550 0.512     NA         NA
#>       drift_eta1  3.69  5 -0.209 0.7340 0.734     NA         NA
#> Covariances with: T0m_eta1, T0m_eta2, mm_Y1, mm_Y2
#> 
#> A new dimension of variation: largest eigenvalue 6.3, p = 0.097; direction drift_eta2 1, diff_eta2_eta1 -0.87, diff_eta2 0.85, drift_eta1 0.82, mvarY1 0.63
#> 
#> State dependence of the drift:
#>  equation     LM df      p
#>      eta1 27.300  3 <5e-04
#>      eta2  0.529  3 0.9210
#>       all 29.900  6 <5e-04
#> 
#>       term equation      LM        p    p_adj estimate     se
#>  eta1*eta2     eta1 25.7000   <5e-04 2.39e-06  0.19400 0.0383
#>  eta1*eta1     eta1  9.0300 2.66e-03 7.97e-03  0.06970 0.0232
#>  eta2*eta2     eta1  1.1100 2.91e-01 5.82e-01  0.03550 0.0336
#>  eta1*eta2     eta2  0.2750 6.00e-01 8.44e-01 -0.01420 0.0271
#>  eta2*eta2     eta2  0.1450 7.03e-01 8.44e-01 -0.00663 0.0174
#>  eta1*eta1     eta2  0.0242 8.76e-01 8.76e-01       NA     NA
#>                                                             reads
#>  DRIFT[eta1,eta1] varies with eta2, or DRIFT[eta1,eta2] with eta1
#>                                 DRIFT[eta1,eta1] varies with eta1
#>                                 DRIFT[eta1,eta2] varies with eta2
#>  DRIFT[eta2,eta1] varies with eta2, or DRIFT[eta2,eta2] with eta1
#>                                 DRIFT[eta2,eta2] varies with eta2
#>                                 DRIFT[eta2,eta1] varies with eta1
#> 
#> State dependence of the innovation sds:
#>  equation    LM df      p
#>      eta1 25.20  2 <5e-04
#>      eta2  5.13  2 0.0875
#>       all 31.30  4 <5e-04
#> 
#>                      term     LM        p    p_adj estimate     se
#>  sd(eta1) x exp(v * eta2) 22.400   <5e-04 8.95e-06   0.1710 0.0362
#>  sd(eta1) x exp(v * eta1)  7.490 6.22e-03 1.24e-02   0.0945 0.0345
#>  sd(eta2) x exp(v * eta2)  5.030 2.49e-02 3.32e-02  -0.1090 0.0486
#>  sd(eta2) x exp(v * eta1)  0.577 4.48e-01 4.48e-01  -0.0302 0.0397
#> 
#> Fixed cells, freed: joint LM = 2.77, df = 1, p = 0.1
#>                cell   LM     p p_adj estimate     se
#>  MANIFESTVAR[Y2,Y1] 2.77 0.096 0.096    0.087 0.0523
#> 
#> Parameters depending on a subject's level (a random intercept):
#>  level    LM df     p
#>  mm_Y1 14.30  9 0.205
#>  mm_Y2  9.75  9 0.468
#>    all 24.90 18 0.358
#> 
#>                     term   LM       p p_adj estimate     se
#>        diff_eta2 ~ mm_Y1 7.87 0.00502 0.075  -0.0413 0.0147
#>           mvarY1 ~ mm_Y1 6.54 0.01050 0.075  -0.0280 0.0109
#>       drift_eta2 ~ mm_Y1 6.24 0.01250 0.075  -0.0385 0.0154
#>           mvarY2 ~ mm_Y1 4.30 0.03810 0.171  -0.0218 0.0105
#>       drift_eta1 ~ mm_Y2 2.80 0.09450 0.294   0.0384 0.0230
#>   diff_eta2_eta1 ~ mm_Y1 2.55 0.11000 0.294       NA     NA
#>        diff_eta1 ~ mm_Y1 2.49 0.11400 0.294       NA     NA
#>  drift_eta1_eta2 ~ mm_Y1 1.60 0.20700 0.465       NA     NA
#>        diff_eta1 ~ mm_Y2 1.41 0.23400 0.469       NA     NA
#>   diff_eta2_eta1 ~ mm_Y2 1.26 0.26200 0.472       NA     NA

The state dependence stands out. The heterogeneity tests stay quiet here, but they need not: a drift that changes with the state can also look, subject by subject, like a drift that differs between subjects, and in simulations a missing nonlinearity did move them. A badly misspecified model can light up several tests at once, so read them as a map: relax what stands out most, refit, and test again.

Do the model’s predictions look like the data?

The usual check of a dynamic model compares the data with data the fitted model generates: the autocorrelations and cross-correlations (ACF and CCF), how much subjects differ from each other, and so on. ctFitAdequacy() makes those comparisons tests, in families aimed at different departures.

Most of the families work on the one-step-ahead prediction errors, which under the model are independent and standard normal however the observations are spaced, so they need no generated data. Each variable’s error is taken given the others observed at the same time, so a result names a variable.

  • residuals: the errors’ correlations within subjects, for every pair of a subject’s observations, binned by the time between them. This is the residual ACF and CCF in continuous time, and a slow process the model lacks shows as correlation over the gaps it spans.
  • latent: the same for each latent process: how much each observation moves the predicted state, standardised, correlated with the same subject’s moves at other times. A model with many indicators per process can leave each item’s errors nearly uncorrelated while the processes themselves are misspecified, a cross effect missing, say; one statistic per process rather than per item does not dilute that.
  • volatility: the same for the squared errors, for variance that clusters in time.
  • pairs: whether two variables’ errors at the same time correlate as the model says, for a wrong measurement error covariance or loading.
  • periodic: a cycle the model lacks, such as a daily rhythm: the periodogram of each variable’s errors and each process’s updates at a period. It depends only on the time between observations, so it finds a rhythm whatever its phase in each subject. Name the periods that matter (periods = 24 with time in hours), or by default it scans.
  • tvcovariates, given tvcovariates: whether time varying variables the model does not use predict its errors, at the same and the previous observation.
  • distribution: skewness, kurtosis and outlying occasions.
  • fit: whether some subjects are fitted much worse than the model allows.
  • dependence: whether an error or its size depends, in any form, on what came before it: the predicted states, the previous errors, the time since the last observation.
  • effects: on a model whose random effects are integrated by intoverpop = "laplace", whether their normal distribution fits. Each subject’s posterior probability that its effect lies below a cut, at cuts along each axis, is averaged over subjects; under the model that average is the normal’s, and a skewed or bimodal distribution of the subjects moves it.
  • scores, when asked for: whether each observation’s score is uncorrelated with the scores of the same subject’s other observations, which holds under the model whatever is being looked for. That makes it a check for dynamic misspecification of any kind, but a weak one, spread over many statistics; the targeted tests above find what they cover far more often.

The error statistics account for the parameters having been estimated from the same data. Each subject’s contribution is projected on its scores, which removes what estimation did to the statistic, as the score tests do. Three more families compare statistics of the data with generated data: the spread over subjects of their own means, standard deviations and lag-one correlations; whether change depends on level nonlinearly; and, given covariates, whether subjects’ statistics depend on them. With many observations every misfit is significant, so the size of what remains is reported too, as root mean square residual correlations, of the items and of the latent processes, with intervals and a test that each is at most close (0.05 by default).

On the second example’s fit, which lacks age, the checks fail, through the subjects’ spread and their relation to age:

ctFitAdequacy(fit2, covariates = covariates)
#> Adequacy: misfit detected by 2 of 12 families at alpha = 0.05
#> (estimation removed by: derivative; 100 generated datasets)
#> 
#>        family statistics                worst worst_z        p    p_adj
#>     residuals         16 Y2~Y1 gap 4.87-10.20   1.610 8.06e-01 1.00e+00
#>        latent         16  eta2~eta2 gap >10.2  -1.360 9.10e-01 1.00e+00
#>    volatility          8  Y1^2 gap 2.54- 4.87   0.990 9.35e-01 1.00e+00
#>         pairs          1                Y1:Y2   0.369 7.01e-01 1.00e+00
#>      periodic         60       Y2 period 4.05   2.440 5.46e-01 1.00e+00
#>  distribution          8      distance spread   1.530 3.94e-01 1.00e+00
#>           fit          2   spread of unit fit   2.580 3.10e-02 3.10e-01
#>       effects         20          P(u4 < 1.5)  -1.730 6.56e-01 1.00e+00
#>    dependence          4       dcor(Y2, past)   1.110 6.50e-01 1.00e+00
#>      subjects          8       sd(lag1_Y1_Y2)   5.990 2.95e-07 3.25e-06
#>     nonlinear         10        cor(dY1^2,Y2)   2.760 7.57e-02 6.81e-01
#>    covariates         16  cor(age,lag1_Y1_Y2)   7.860 8.64e-11 1.04e-09
#> 
#> Size of what remains (root mean square, bias-corrected; p_close tests that it is at least close):
#>                                     measure estimate lower   upper close
#>              residual correlation, all gaps        0     0 0.00424  0.05
#>         residual correlation, shortest gaps        0     0 0.00525  0.05
#>       latent residual correlation, all gaps        0     0 0.00217  0.05
#>  latent residual correlation, shortest gaps        0     0 0.00311  0.05
#>                      within-row correlation        0     0 0.00375  0.05
#>  p_close
#>        0
#>        0
#>        0
#>        0
#>        0

On the linear fit to the state-dependent data:

ctFitAdequacy(fit3)
#> Adequacy: no family detects misfit at alpha = 0.05
#> (estimation removed by: derivative; 100 generated datasets)
#> 
#>        family statistics                worst worst_z     p p_adj
#>     residuals         16 Y1~Y2 gap 2.58- 4.91    1.59 0.710 1.000
#>        latent         16 eta1~eta2 gap <=2.58   -1.41 0.800 1.000
#>    volatility          8  Y2^2 gap 2.58- 4.91    1.91 0.326 1.000
#>         pairs          1                Y1:Y2    1.27 0.109 1.000
#>      periodic         60       Y1 period 2.35    2.28 0.816 1.000
#>  distribution          8             skew(Y2)   -2.13 0.831 1.000
#>           fit          2   spread of unit fit    2.06 0.064 0.704
#>       effects         20          P(u1 < 1.5)   -1.48 0.857 1.000
#>    dependence          4     dcor(Y1^2, past)    1.46 0.427 1.000
#>      subjects          8       sd(lag1_Y2_Y1)    2.13 0.996 1.000
#>     nonlinear         10        cor(dY2^2,Y2)   -2.25 0.100 1.000
#> 
#> Size of what remains (root mean square, bias-corrected; p_close tests that it is at least close):
#>                                     measure estimate lower   upper close
#>              residual correlation, all gaps  0.00000     0 0.00890  0.05
#>         residual correlation, shortest gaps  0.00000     0 0.00726  0.05
#>       latent residual correlation, all gaps  0.00000     0 0.00759  0.05
#>  latent residual correlation, shortest gaps  0.00000     0 0.00888  0.05
#>                      within-row correlation  0.00457     0 0.01400  0.05
#>    p_close
#>   0.00e+00
#>   0.00e+00
#>   0.00e+00
#>   0.00e+00
#>  1.07e-171

Every family passes, though this model is missing the state dependence the score test found at p = 5e-6. A moderated drift, averaged over the data, still decays much like a single exponential, so the errors it leaves are nearly uncorrelated: the root mean square residual correlation is indistinguishable from zero. The checks are worth having for what lies outside every score test’s reach: a slow process the model lacks, variance that clusters, errors with heavy tails, a process of the wrong order. In simulation the residual correlations over time gaps found a missing slow process (a time constant ten times the fast one’s) in 65 of 100 datasets, where correlations at a few lags of the observations found it in 5. But passing the checks is necessary rather than sufficient, which is why the next section screens with the score tests and stops on the checks.

ctModelImprove() puts the tests into a loop. It screens the fit, refits with the most strongly indicated relaxations one at a time (warm started from the current estimates), and keeps the best if the refit’s likelihood confirms it. It repeats until nothing more is indicated, then checks the model with ctFitAdequacy(). If a check fails, it tries structural changes the score tests cannot reach: a slow process driving a latent (ctModelSlowComponent()), tried first for a latent whose updates stay correlated over long gaps; a daily or other cycle at the period the periodic family points to (ctModelPeriodicComponent()); a process of its own for an item whose errors stay correlated with its own (ctModelUniqueProcess()); making a process second order (ctModelHigherOrder()); or models you supply. Finally it tries dropping each change it kept, in case a later one made it redundant.

Here the data contain two of the departures above at once: age acts on the intercept of Y1 and on the cross effect, eta2 moderates how eta1 regulates, and w is unrelated.

gen4 <- generator(DRIFT = matrix(c("-0.6 + 0.3 * tanh(eta2)", "drift21||||age=0.3",
  0, -0.4), 2, 2),
  MANIFESTMEANS = c("mm1|param|TRUE|1|age=0.6", "mm2|param|TRUE|1"),
  TIpredNames = c("age", "w"))
d4 <- data.frame(ctGenerate(gen4, n = 150, Tpoints = 30, dtmean = 1,
  logdtsd = 0.5, backend = "julia", values = c(drift21 = 0.25, mm1 = 0, mm2 = 0)))
fit4 <- ctFit(d4[, c("id", "time", "Y1", "Y2")], model, backend = "julia", cores = 2)
imp <- ctModelImprove(fit4, covariates = unique(d4[, c("id", "age", "w")]))
imp
#> Model improvement: stopped because adequate
#> Log likelihood -11322.58 -> -11146.26 (23 -> 26 parameters)
#> 
#>  round                                            change screen_p dloglik dpar
#>      1      age as a predictor of mm_Y1, drift_eta2_eta1 1.00e-03   162.0    2
#>      2                    eta1*eta2 in the drift of eta1 1.78e-06    13.2    1
#>  prune drop age as a predictor of mm_Y1, drift_eta2_eta1       NA  -159.0   -2
#>  prune               drop eta1*eta2 in the drift of eta1       NA      NA   -1
#>   done                                         note
#>   TRUE               LR p = 2.91e-71 against 0.0167
#>   TRUE                LR p = 2.71e-07 against 0.025
#>  FALSE                 LR p = 1.42e-69 against 0.05
#>  FALSE kept without a refit: Wald 26.4 against 7.68
#> 
#> Adequacy: no family detects misfit at alpha = 0.05
#> (estimation removed by: plugin; 100 generated datasets)
#> 
#>        family statistics               worst worst_z     p p_adj
#>     residuals         16     Y2~Y1 gap >10.1  -1.980 0.614     1
#>        latent         16 eta2~eta1 gap >10.1  -1.780 0.671     1
#>    volatility          8     Y2^2 gap <=2.56  -1.470 0.775     1
#>         pairs          1               Y1:Y2  -1.220 0.173     1
#>      periodic         56      Y1 period 5.36   2.590 0.299     1
#>  distribution          9            distance   0.892 0.945     1
#>           fit          2  spread of unit fit  -0.560 0.697     1
#>       effects         20         P(u1 < 1.5)  -1.410 0.874     1
#>    dependence          4      dcor(Y2, past)   1.080 0.687     1
#>      subjects          8      sd(lag1_Y2_Y2)   2.260 0.876     1
#>     nonlinear         10      cor(dY2,Y1*Y1)  -0.984 0.425     1
#>    covariates          8   cor(w,lag1_Y2_Y1)   1.320 0.894     1
#> 
#> Size of what remains (root mean square, bias-corrected; p_close tests that it is at least close):
#>                                     measure estimate lower   upper close
#>              residual correlation, all gaps  0.00000     0 0.00521  0.05
#>         residual correlation, shortest gaps  0.00000     0 0.01260  0.05
#>       latent residual correlation, all gaps  0.00000     0 0.00562  0.05
#>  latent residual correlation, shortest gaps  0.00389     0 0.01390  0.05
#>                      within-row correlation  0.00488     0 0.01600  0.05
#>    p_close
#>   0.00e+00
#>  2.94e-212
#>   0.00e+00
#>  2.37e-166
#>   1.34e-97

Each round shows what was tried, the score test that proposed it, and what the refit’s likelihood said. The first round kept age, with effects on the cross effect, the intercept of Y1 and the initial level of eta1; it also tried a random effect on the cross effect, which the score tests had flagged because subjects of different ages differ in it – a covariate that explains the variation beats a random effect that only describes it. The second round kept the moderation, and turned down the dimension test’s suggestion that three dynamic parameters vary together, whose refit gained 8 log likelihood units for 18 parameters. Then nothing more was indicated, every adequacy check passed, and trying each kept change out again showed both were still needed. The effects found are near those that generated the data (0.3 on the cross effect, 0.6 on the intercept of Y1). The effect on the initial level of eta1 was not generated: it is small, and only just significant – a search that decides at alpha = 0.05 keeps such things about one time in twenty, and alpha = 0.01 is the stricter choice.

The final fit is an ordinary fit:

s <- summary(imp$fit)
s$tipreds
#>                          mean    sd  2.5%   50% 97.5%  mode      z
#> tip_age_drift_eta2_eta1 0.310 0.023 0.267 0.311 0.355 0.312 13.333
#> tip_age_mm_Y1           0.611 0.074 0.462 0.613 0.751 0.611  8.308
s$popmeans[grepl("^sdep", rownames(s$popmeans)), ]
#>                     mean    sd  2.5%   50% 97.5%  mode
#> sdep_eta1_eta1_eta2 0.12 0.024 0.073 0.119 0.171 0.121

The model was chosen by the data it is fitted to, so its standard errors and tests are too optimistic about the structure the search added. Treat that structure as a hypothesis, and confirm it on other data. Within one dataset, holdout = 0.3 sets aside 30 per cent of the subjects: the search runs on the rest, and each change it keeps is then tested on the held-out subjects alone. With a large dataset, close = 0.05 also stops the search once the residual correlations that remain are not significantly larger than that, where every small misfit would otherwise be significant.

How the tests work, briefly

  • The scores are those of the objective the model was fitted with, so they integrate over random effects exactly as the fit did. A covariate’s effect adds to a parameter’s raw value, so its score is the covariate times each subject’s score in that parameter – any number of covariates costs a few matrix products after one set of per-subject scores and Hessians.
  • A random effect’s variance has, at zero, the score 12∑i(sij2+Hi,jj)\frac{1}{2}\sum_i (s_{ij}^2 + H_{i,jj}), and its covariance with an effect already there ∑i(sijsir+Hi,jr)\sum_i (s_{ij} s_{ir} + H_{i,jr}): the information matrix test in those parameters, which is the score test for neglected heterogeneity. Every subject already has one such effect, their initial state, so the covariances are tested even in a model with no random effects.
  • State dependence is tested on the fitted model extended with the drift terms at zero, which reproduces the fit exactly there.
  • The fitted model’s curvature projects the fitted parameters out of each test and sizes the one-step estimates; each test’s variance is the spread of the subjects’ contributions, and its p-value comes from resampling subjects. That keeps a test valid when the model is wrong in some other way – the case a screen is for – where a variance from the model’s own curvature would make unrelated tests reject too.

The tests need subjects as independent units, so a model with levels above the subject can be tested for state dependence only. See ?ctScoreTest for the details.