Skip to contents

Tests a fitted model in families, each aimed at one way a continuous time model can fall short, and gives each family a p-value. Most families use the one-step-ahead prediction errors, which under the model are independent and standard normal whatever the spacing of the observations; each variable's error is taken given the others observed at the same time, so a result names a variable.

Usage

ctFitAdequacy(
  fit,
  covariates = NULL,
  families = c("residuals", "latent", "volatility", "pairs", "periodic", "distribution",
    "fit", "dependence", "effects", "tvcovariates", "subjects", "nonlinear",
    "covariates"),
  gaps = NULL,
  periods = NULL,
  tvcovariates = NULL,
  close = 0.05,
  alpha = 0.05,
  projection = c("auto", "derivative", "plugin", "opg", "none"),
  parameters = NULL,
  nsamples = 100,
  lags = 0:3,
  nboot = 1999L,
  cores = getOption("mc.cores", 2L)
)

Arguments

fit

A fit from ctFit, either backend. The projection and the scores it needs require backend = 'julia'.

covariates

Optional subject-level covariates, as for ctScoreTest, for the covariates family.

families

Which families to run; see Details.

gaps

Break points, in the data's time units, for the residual correlations' gap bins. By default the eighth, quarter and half quantiles of the time between every pair of a unit's rows.

periods

Periods, in the data's time units, for the periodic family: 24 for a daily cycle with time in hours. By default a scan.

tvcovariates

Optional time varying covariates for the tvcovariates family: a data frame with the model's subject id and time columns and one or more numeric columns, matched to the fit's rows by id and time.

close

The root mean square residual correlation regarded as a close fit.

alpha

Level for adequate and close_fit.

projection

How the error-based statistics account for estimation: 'derivative' differences the statistics along every parameter (two passes of the filter per parameter) and projects with the fit's Hessian; 'plugin', on a Laplace fit, takes those differences from plug-in passes, which filter each subject at effects solved once from the whole record rather than row by row – several times cheaper, and approximate (on a pilot-shaped model it recovered the distribution family's misfit, z 5.3 against 5.6 exact and 1.8 uncorrected). Where random effects reach the dynamics or the variances, either projects only the unit-level families (distribution, fit, effects, scores) and leaves the error-based ones unprojected: there their projected variance was unreliable, while unprojected they held their level on data generated from the model; 'opg' uses the outer product of the scores, valid when units outnumber parameters many times; 'none' ignores estimation, which is conservative, and for the distribution family can hide a strong misfit. 'auto' takes the first that is affordable: 'derivative' when the fit's Hessian is stored and the passes take under getOption('ctsem.adequacy.budget', 120) seconds, then 'plugin' on the same terms, 'opg' with at least ten units a parameter, else 'none'.

parameters

For the scores family, the raw parameters whose row scores are used: by default up to 16 of the dynamic and measurement parameters, those in DRIFT, DIFFUSION and CINT first.

nsamples

Number of generated datasets, for the families that need them. Reused from fit when it already carries at least as many.

lags

Observation lags for the lagged family.

nboot

Bootstrap draws for each family's calibration.

cores

Cores for the engine and for generating.

Value

An object of class ctFitAdequacy: families (one row per family: the number of statistics, the worst, its z, the family's p-value and the Holm-adjusted one), statistics (every statistic: what it is about – variable, with (the other variable of a pair, or what the variable is correlated with), gap (the gap bin of a pair of rows, or a lag), period and covariate, each NA where it does not apply – and its observed value, z and two-sided p), size (the root mean square residual correlations, their intervals and close-fit tests), errors (the rows the error-based families are computed from, in long form: Subject, Time, Element – 'error', each variable's standardised one-step-ahead error given the row's other observed values; 'update', each latent's standardised update; 'etaprior', its predicted mean – Row, the variable, and value; NULL when no such family ran), adequate, close_fit, informative (with tvcovariates, whether they predict the errors at alpha; NA without), projection (the method used), and fit (with any generated data, for reuse). plot() draws the families, or one family's statistics.

Details

residuals

Correlations of the errors within units, for every pair of rows, by time gap (gaps): the residual autocorrelation and cross-correlation in continuous time. Slow structure the model lacks shows as correlation over the gaps it spans.

latent

The same for each latent process: the update each row makes to the predicted state, standardised, correlated with the same subject's updates at other times. One per process rather than one per item, so a misfit in the dynamics is not diluted by item-specific structure.

volatility

The same for the squared errors, per variable: variance that clusters in time.

pairs

Each pair of variables' errors in the same row against the correlation the model implies: a wrong measurement error covariance or loading.

periodic

A cycle the model lacks, such as a daily rhythm: each variable's errors, and each latent's updates, correlated with the same unit's at other times weighted by the cosine of the time between them over a period – the periodogram of the errors at that period. It depends only on the time between rows, so it finds a cycle whatever its phase in each unit and needs no clock. At the named periods, or by default scanning the periods a typical record resolves, from half its length to twice the typical interval between rows. The estimate is the errors' autocorrelation at the period: a cycle of amplitude a, in error standard deviations, gives a^2 / 2.

distribution

Skewness and kurtosis of each variable's errors, and each row's squared whitened distance against its chi-square mean and variance, together with Anderson-Darling tests against the normal and chi-square.

fit

How unevenly units are fitted: the spread of each unit's summed distance against the chi-square's, and the worst unit.

dependence

The distance correlation of each variable's error, and of its square, with what came before – the predicted latent states, the previous errors, the time interval – by permutation: a conditional mean or variance the model gets wrong, in any form.

effects

On a single-level Laplace fit, whether the random effects' normal distribution is adequate: each subject's posterior probability, smoothed, that its standardised effect lies below a cut, at cuts along each axis, against the normal's. Under the model the posterior averages to the prior, so the differences have mean zero; a skewed or bimodal distribution, or a subgroup, moves them. These integrate the gradient function (Verbeke & Molenberghs, 2013; Drikvandi, Verbeke & Molenberghs, 2017), which is too heavy tailed to test directly.

tvcovariates

With tvcovariates, whether time varying variables the model does not use carry information about what it predicts: each variable's error, and each latent's update, against each covariate at the same row and at the unit's previous row, the covariate standardised. The estimate is the regression of the standardised error on the covariate. The previous row's value is a test of prediction (Granger's sense); the same row's is an association, which a covariate the outcome itself moves would show as well. Its null is larger than the other families': the errors have mean zero given what the model conditions on, the outcomes' own past, and need not be uncorrelated with anything else. A correct model of the outcomes fails it when a covariate is another measure of the same states, since that measure's previous value sharpens the estimate of the state and so predicts the next error. So it says whether a model using these variables would predict better, not that the model of the outcomes is wrong, and it is not counted in adequate.

scores

Not run unless asked for. White's (1987) dynamic information matrix test: each data row's score, the derivative of its one-step log likelihood, for the dynamic and measurement parameters (parameters), correlated with the scores of the same subject's other rows, by time gap. Under the model a subject's row scores are uncorrelated whatever the departure, so this is a check for dynamic misspecification of any kind; being that broad, over many statistics, it is the least powerful of the families, and the targeted tests of ctScoreTest (stability, state dependence) find what they cover far more often. It costs a dual pass of the filter per eight parameters, and on the Laplace route one per truncated record, which is hours on large data.

subjects, nonlinear, covariates, lagged

Statistics of the data against datasets generated from the fit: how much subjects differ in mean, standard deviation and lag-one correlations; whether change depends on level nonlinearly; with covariates, whether subjects' statistics depend on them; and the data's own lagged correlations (lags). These use the data twice and are conservative; the lagged correlations especially, as the fit reproduces them, so they are not run unless asked for.

The error-based statistics are moment tests: each unit's contribution is projected on its scores, which removes what estimating the parameters did to the statistic (projection), and the variance is the errors' own: they are independent over rows under the model, so a sum of products over pairs of rows has the sum of the products' squares as its variance, which holds with heavy tails or a changing variance; estimating the parameters takes a known part of it away. That is the variance of the residual, latent, volatility, within-row, periodic and covariate families. A periodic statistic is in each unit a quadratic form of rank two in its errors, the periodogram, skewed to the right however many units there are; its z is read through a chi-square of the skewness that form has under the model, worked out exactly from the unit's times. The distribution and fit families, whose statistics are sums of high powers, use the spread of the units' contributions and a studentised bootstrap over units. Within a family the largest standardised statistic is calibrated with the statistics' correlation over units, and the sum of their squares, where the variances are the model's, by its terms between units, whose mean is zero under the model (Chen & Qin, 2010), and otherwise by the bootstrap; the two are combined, with any exact p-values the family has, by the Cauchy combination; families are combined by Holm. adequate is TRUE when no family's adjusted p-value is below alpha. The tvcovariates family tests a larger information set than the model's and is held out of both: its p-value is its own, and informative reports it.

On a Laplace fit (intoverpop = 'laplace') whose random effects reach the dynamics or the variances, the errors come from the model's own predictive distribution: each row predicted from the subject's earlier rows with the random effects integrated against their posterior given those rows by importance sampling (getOption('ctsem.sequential.particles', 512) quasi-random draws a subject), the own variances integrated in the filter exactly. The draws are fixed by getOption('ctsem.sequential.seed'), so a call repeats; another seed, or more draws, shows how much of a result is the sampling's. The approximation the fit itself uses there – the posterior's mode on each truncated record, its uncertainty added by the delta method – is exact only for effects that enter linearly; elsewhere it put a 1.5 per cent bias into the predictive variance, and correct models were rejected through it. Where that predictive cannot be had – random effects at more than one level – the row distances are left out and the latent family's p-values are approximate, and the call says so.

A binary, ordinal, count or censored item's error is not normal, so its place is taken by its randomised probability integral transform through the normal quantile: the item's predictive distribution function at what was observed, given the subject's earlier rows and the items the filter had already applied in the row (non-Gaussian items one at a time, in item order), with a fixed uniform between its values just below and at the observation where that has probability mass. Under the model these are standard normal and independent, so every family reads them as it reads Gaussian errors; on the Laplace route they come from the exact predictive. The latent family's updates on such rows rest on the filter's Gaussian projection of each item, so its p-values are approximate there, and the call says so. Within a row, a non-Gaussian item's transform and a Gaussian item's error have no model-implied correlation the pairs family could test against, and are not paired.

With many observations every misfit is significant, so the sizes of the residual, latent and within-row correlations the model leaves are reported too, as root mean squares, each squared correlation less its sampling variance (as for the SRMR, Maydeu-Olivares 2017): the mean square is unbiased, its root, truncated at zero, bias-corrected. Each has a 95 per cent interval and p_close, the p-value of the test that it is at least close. close_fit is TRUE when every family that fails is one of those three and that test rejects at alpha – the size is shown to be below close, its one-sided upper bound under it. A size whose interval reaches close is not a close fit: not showing that a misfit is large does not show that it is small. A close fit is about these averages over many correlations; one large discrepancy among them, or a biased estimate of a quantity of interest, can sit under a small average.

Detection is not localization: a failure in one variable's equation reaches the others' errors through the fitted covariance. ctScoreTest is the more sensitive check for the departures it covers, and ctModelImprove uses both.

References

Newey, W. K. (1985). Maximum likelihood specification testing and conditional moment tests. Econometrica, 53, 1047-1070.

White, H. (1987). Specification testing in dynamic models. In T. Bewley (Ed.), Advances in Econometrics: Fifth World Congress. Cambridge University Press.

Verbeke, G., & Molenberghs, G. (2013). The gradient function as an exploratory goodness-of-fit assessment of the random-effects distribution in mixed models. Biostatistics, 14, 477-490.

Drikvandi, R., Verbeke, G., & Molenberghs, G. (2017). Diagnosing misspecification of the random-effects distribution in mixed models. Biometrics, 73, 63-71.

Chang, J., Yao, Q., & Zhou, W. (2017). Testing for high-dimensional white noise using maximum cross-correlations. Biometrika, 104, 111-127.

Chen, S. X., & Qin, Y.-L. (2010). A two-sample test for high-dimensional data with applications to gene-set testing. The Annals of Statistics, 38, 808-835.

Liu, Y., & Xie, J. (2020). Cauchy combination test. Journal of the American Statistical Association, 115, 393-402.

Maydeu-Olivares, A. (2017). Assessing the size of model misfit in structural equation models. Psychometrika, 82, 533-558.

See also

ctPostPredPlots and ctFitCheckCov to see the data against the fit.