Tests a fitted model in families, each aimed at one way a continuous time model can fall short, and gives each family a p-value. Most families use the one-step-ahead prediction errors, which under the model are independent and standard normal whatever the spacing of the observations; each variable's error is taken given the others observed at the same time, so a result names a variable.
Usage
ctFitAdequacy(
fit,
covariates = NULL,
families = c("residuals", "latent", "volatility", "pairs", "periodic", "distribution",
"fit", "dependence", "effects", "tvcovariates", "subjects", "nonlinear",
"covariates"),
gaps = NULL,
periods = NULL,
tvcovariates = NULL,
close = 0.05,
alpha = 0.05,
projection = c("auto", "derivative", "plugin", "opg", "none"),
parameters = NULL,
nsamples = 100,
lags = 0:3,
nboot = 1999L,
cores = getOption("mc.cores", 2L)
)Arguments
- fit
A fit from
ctFit, either backend. The projection and the scores it needs requirebackend = 'julia'.- covariates
Optional subject-level covariates, as for
ctScoreTest, for thecovariatesfamily.- families
Which families to run; see Details.
- gaps
Break points, in the data's time units, for the residual correlations' gap bins. By default the eighth, quarter and half quantiles of the time between every pair of a unit's rows.
- periods
Periods, in the data's time units, for the
periodicfamily: 24 for a daily cycle with time in hours. By default a scan.- tvcovariates
Optional time varying covariates for the
tvcovariatesfamily: a data frame with the model's subject id and time columns and one or more numeric columns, matched to the fit's rows by id and time.- close
The root mean square residual correlation regarded as a close fit.
- alpha
Level for
adequateandclose_fit.- projection
How the error-based statistics account for estimation:
'derivative'differences the statistics along every parameter (two passes of the filter per parameter) and projects with the fit's Hessian;'plugin', on a Laplace fit, takes those differences from plug-in passes, which filter each subject at effects solved once from the whole record rather than row by row – several times cheaper, and approximate (on a pilot-shaped model it recovered the distribution family's misfit, z 5.3 against 5.6 exact and 1.8 uncorrected). Where random effects reach the dynamics or the variances, either projects only the unit-level families (distribution, fit, effects, scores) and leaves the error-based ones unprojected: there their projected variance was unreliable, while unprojected they held their level on data generated from the model;'opg'uses the outer product of the scores, valid when units outnumber parameters many times;'none'ignores estimation, which is conservative, and for the distribution family can hide a strong misfit.'auto'takes the first that is affordable:'derivative'when the fit's Hessian is stored and the passes take undergetOption('ctsem.adequacy.budget', 120)seconds, then'plugin'on the same terms,'opg'with at least ten units a parameter, else'none'.- parameters
For the
scoresfamily, the raw parameters whose row scores are used: by default up to 16 of the dynamic and measurement parameters, those in DRIFT, DIFFUSION and CINT first.- nsamples
Number of generated datasets, for the families that need them. Reused from
fitwhen it already carries at least as many.- lags
Observation lags for the
laggedfamily.- nboot
Bootstrap draws for each family's calibration.
- cores
Cores for the engine and for generating.
Value
An object of class ctFitAdequacy: families (one row
per family: the number of statistics, the worst, its z, the family's
p-value and the Holm-adjusted one), statistics (every statistic:
what it is about – variable, with (the other variable of
a pair, or what the variable is correlated with), gap (the gap
bin of a pair of rows, or a lag), period and covariate,
each NA where it does not apply – and its observed value, z and
two-sided p), size (the root mean square residual correlations,
their intervals and close-fit tests), errors (the rows the
error-based families are computed from, in long form: Subject,
Time, Element – 'error', each variable's
standardised one-step-ahead error given the row's other observed
values; 'update', each latent's standardised update;
'etaprior', its predicted mean – Row, the variable, and
value; NULL when no such family ran), adequate,
close_fit, informative (with tvcovariates, whether
they predict the errors at alpha; NA without), projection
(the method used), and fit
(with any generated data, for reuse). plot() draws the families,
or one family's statistics.
Details
- residuals
Correlations of the errors within units, for every pair of rows, by time gap (
gaps): the residual autocorrelation and cross-correlation in continuous time. Slow structure the model lacks shows as correlation over the gaps it spans.- latent
The same for each latent process: the update each row makes to the predicted state, standardised, correlated with the same subject's updates at other times. One per process rather than one per item, so a misfit in the dynamics is not diluted by item-specific structure.
- volatility
The same for the squared errors, per variable: variance that clusters in time.
- pairs
Each pair of variables' errors in the same row against the correlation the model implies: a wrong measurement error covariance or loading.
- periodic
A cycle the model lacks, such as a daily rhythm: each variable's errors, and each latent's updates, correlated with the same unit's at other times weighted by the cosine of the time between them over a period – the periodogram of the errors at that period. It depends only on the time between rows, so it finds a cycle whatever its phase in each unit and needs no clock. At the named
periods, or by default scanning the periods a typical record resolves, from half its length to twice the typical interval between rows. The estimate is the errors' autocorrelation at the period: a cycle of amplitude a, in error standard deviations, gives a^2 / 2.- distribution
Skewness and kurtosis of each variable's errors, and each row's squared whitened distance against its chi-square mean and variance, together with Anderson-Darling tests against the normal and chi-square.
- fit
How unevenly units are fitted: the spread of each unit's summed distance against the chi-square's, and the worst unit.
- dependence
The distance correlation of each variable's error, and of its square, with what came before – the predicted latent states, the previous errors, the time interval – by permutation: a conditional mean or variance the model gets wrong, in any form.
- effects
On a single-level Laplace fit, whether the random effects' normal distribution is adequate: each subject's posterior probability, smoothed, that its standardised effect lies below a cut, at cuts along each axis, against the normal's. Under the model the posterior averages to the prior, so the differences have mean zero; a skewed or bimodal distribution, or a subgroup, moves them. These integrate the gradient function (Verbeke & Molenberghs, 2013; Drikvandi, Verbeke & Molenberghs, 2017), which is too heavy tailed to test directly.
- tvcovariates
With
tvcovariates, whether time varying variables the model does not use carry information about what it predicts: each variable's error, and each latent's update, against each covariate at the same row and at the unit's previous row, the covariate standardised. The estimate is the regression of the standardised error on the covariate. The previous row's value is a test of prediction (Granger's sense); the same row's is an association, which a covariate the outcome itself moves would show as well. Its null is larger than the other families': the errors have mean zero given what the model conditions on, the outcomes' own past, and need not be uncorrelated with anything else. A correct model of the outcomes fails it when a covariate is another measure of the same states, since that measure's previous value sharpens the estimate of the state and so predicts the next error. So it says whether a model using these variables would predict better, not that the model of the outcomes is wrong, and it is not counted inadequate.- scores
Not run unless asked for. White's (1987) dynamic information matrix test: each data row's score, the derivative of its one-step log likelihood, for the dynamic and measurement parameters (
parameters), correlated with the scores of the same subject's other rows, by time gap. Under the model a subject's row scores are uncorrelated whatever the departure, so this is a check for dynamic misspecification of any kind; being that broad, over many statistics, it is the least powerful of the families, and the targeted tests ofctScoreTest(stability, state dependence) find what they cover far more often. It costs a dual pass of the filter per eight parameters, and on the Laplace route one per truncated record, which is hours on large data.- subjects, nonlinear, covariates, lagged
Statistics of the data against datasets generated from the fit: how much subjects differ in mean, standard deviation and lag-one correlations; whether change depends on level nonlinearly; with
covariates, whether subjects' statistics depend on them; and the data's own lagged correlations (lags). These use the data twice and are conservative; the lagged correlations especially, as the fit reproduces them, so they are not run unless asked for.
The error-based statistics are moment tests: each unit's contribution is
projected on its scores, which removes what estimating the parameters did
to the statistic (projection), and the variance is the errors' own:
they are independent over rows under the model, so a sum of products over
pairs of rows has the sum of the products' squares as its variance, which
holds with heavy tails or a changing variance; estimating the parameters
takes a known part of it away. That is the variance of the residual,
latent, volatility, within-row, periodic and covariate families. A periodic
statistic is in each unit a quadratic form of rank two in its errors, the
periodogram, skewed to the right however many units there are; its z is
read through a chi-square of the skewness that form has under the model,
worked out exactly from the unit's times. The distribution and fit
families, whose statistics are sums of high powers, use the spread of the
units' contributions and a studentised bootstrap over units. Within a family the
largest standardised statistic is calibrated with the statistics'
correlation over units, and the sum of their squares, where the variances
are the model's, by its terms between units, whose mean is zero under the
model (Chen & Qin, 2010), and otherwise by the bootstrap; the two are
combined, with any exact p-values the family has, by the Cauchy
combination; families are combined by Holm. adequate is TRUE
when no family's adjusted p-value is below alpha. The
tvcovariates family tests a larger information set than the model's
and is held out of both: its p-value is its own, and informative
reports it.
On a Laplace fit (intoverpop = 'laplace') whose random effects reach
the dynamics or the variances, the errors come from the model's own
predictive distribution: each row predicted from the subject's earlier rows
with the random effects integrated against their posterior given those rows
by importance sampling (getOption('ctsem.sequential.particles', 512)
quasi-random draws a subject), the own variances integrated in the filter
exactly. The draws are fixed by getOption('ctsem.sequential.seed'),
so a call repeats; another seed, or more draws, shows how much of a result
is the sampling's. The approximation the fit itself uses there – the posterior's
mode on each truncated record, its uncertainty added by the delta method –
is exact only for effects that enter linearly; elsewhere it put a 1.5 per
cent bias into the predictive variance, and correct models were rejected
through it. Where that predictive cannot be had – random effects at more
than one level – the row distances are left out and the latent family's
p-values are approximate, and the call says so.
A binary, ordinal, count or censored item's error is not normal, so its place is taken by its randomised probability integral transform through the normal quantile: the item's predictive distribution function at what was observed, given the subject's earlier rows and the items the filter had already applied in the row (non-Gaussian items one at a time, in item order), with a fixed uniform between its values just below and at the observation where that has probability mass. Under the model these are standard normal and independent, so every family reads them as it reads Gaussian errors; on the Laplace route they come from the exact predictive. The latent family's updates on such rows rest on the filter's Gaussian projection of each item, so its p-values are approximate there, and the call says so. Within a row, a non-Gaussian item's transform and a Gaussian item's error have no model-implied correlation the pairs family could test against, and are not paired.
With many observations every misfit is significant, so the sizes of the
residual, latent and within-row correlations the model leaves are reported
too, as
root mean squares, each squared correlation less its sampling variance (as
for the SRMR, Maydeu-Olivares 2017): the mean square is unbiased, its root,
truncated at zero, bias-corrected. Each has a 95 per cent interval and
p_close, the p-value of the test that it is at least close.
close_fit is TRUE when every family that fails is one of those
three and that test rejects at alpha – the size is shown to be
below close, its one-sided upper bound under it. A size whose
interval reaches close is not a close fit: not showing that a misfit
is large does not show that it is small. A close fit is about these
averages over many correlations; one large discrepancy among them, or a
biased estimate of a quantity of interest, can sit under a small average.
Detection is not localization: a failure in one variable's equation reaches
the others' errors through the fitted covariance. ctScoreTest
is the more sensitive check for the departures it covers, and
ctModelImprove uses both.
References
Newey, W. K. (1985). Maximum likelihood specification testing and conditional moment tests. Econometrica, 53, 1047-1070.
White, H. (1987). Specification testing in dynamic models. In T. Bewley (Ed.), Advances in Econometrics: Fifth World Congress. Cambridge University Press.
Verbeke, G., & Molenberghs, G. (2013). The gradient function as an exploratory goodness-of-fit assessment of the random-effects distribution in mixed models. Biostatistics, 14, 477-490.
Drikvandi, R., Verbeke, G., & Molenberghs, G. (2017). Diagnosing misspecification of the random-effects distribution in mixed models. Biometrics, 73, 63-71.
Chang, J., Yao, Q., & Zhou, W. (2017). Testing for high-dimensional white noise using maximum cross-correlations. Biometrika, 104, 111-127.
Chen, S. X., & Qin, Y.-L. (2010). A two-sample test for high-dimensional data with applications to gene-set testing. The Annals of Statistics, 38, 808-835.
Liu, Y., & Xie, J. (2020). Cauchy combination test. Journal of the American Statistical Association, 115, 393-402.
Maydeu-Olivares, A. (2017). Assessing the size of model misfit in structural equation models. Psychometrika, 82, 533-558.
See also
ctPostPredPlots and ctFitCheckCov to
see the data against the fit.