Effect Sizes of Model Misfit in Structural Equation Models

Structural equation modeling (SEM) is a general set of methods that can be applied to fit multivariate data arising from discrete or continuous outcomes. In this paper, we focus on models for continuous outcomes. Furthermore, for ease of exposition we focus on models with unconstrained means, that is, on covariance structure models. Given the multivariate nature of covariance structure models, assessing the fit of these models (i.e., the discrepancy between the data generating mechanism and the fitted model) is also a multivariate problem.

Residual covariances (i.e., the difference between the sample covariances and the covariances expected under the fitted model) provide a natural estimate of the fit of covariance structure models: the larger (in absolute value) the residual covariance, the worse the fit. However, the magnitude of a covariance is difficult to interpret and, as a result, residual covariances are ill suited as effect sizes of the misfit of a model. To overcome this problem, we can divide the residual covariances by the sample standard deviations of the variables involved leading to the standardized residual covariance. However, standardized residual covariances need not be in an interval from (−1, 1). From this point of view, residual correlations may be preferable to standardized residual covariances.

Population standardized residual covariances (or alternatively, residual correlations) provide standardized effect sizes of the (multivariate) misfit of a structural equations model. Confidence intervals (CIs) for these population effect sizes can be used to gauge the precision with which they are estimated. Alternatively, z statistics may be performed to test the hypothesis that these population parameters are zero (or equal to some alternative, arbitrary, small value). These z statistics are simply the ratio of the estimated statistic, say a residual correlation, to its standard error. Given that so many z tests are examined simultaneously, one should control the error rate for multiple testing. The most convenient way to do so is by using a Bonferroni adjustment of the significance level using the total number of nonredundant covariances or correlations. Of course, a more precise estimate of the overall magnitude of model misfit can be obtained by combining the standardized residual covariances (or residual correlations) into an overall statistic, such as the standardized root mean squared residual (SRMR) or the correlation root mean square residual (CRMR). These overall effect sizes of the misfit can be interpreted (roughly) as the model’s average standardized residual covariance and residual correlation, respectively. Of course, in applications we wish to estimate the population SRMR or CRMR and construct a confidence interval for it (Maydeu-Olivares, 2017).

Yet, it is only meaningful to examine the overall size of model misfit when there is little variability of the standardized residual covariances (or residual correlations) around the sample SRMR (or CRMR). Thus, it is important to inspect the largest values (in absolute value) of the statistically significant standardized residual covariances (or residual correlations). Although it has been repeatedly advised to examine the full matrix of residual covariances or correlations (see for instance, McDonald & Ho, 2002) – or at least the most extreme values within the matrix- this examination is seldom performed in applications. Also, SEM programs typically provide information on the statistical significance (i.e., z statistics) of (unstandardized) residual covariances or they provide standardized residual covariances, but not both. Yet, both should to be examined. If a residual covariance is statistically significant, we wish to examine its standardized counterpart, so that we can judge qualitatively the size of the misfit. Conversely, if a residual correlation suggests that the misfit is unacceptably large, we wish to examine its confidence interval (particularly if sample size is small) to ensure that the magnitude of the observed residual cannot be attributed to chance.

In addition to provide an effect size of the model misfit, standardized residual covariances and residual correlations may be used to assess the source of misfit in poorly fitting models. To be useful to this aim, it is important to organize the observed variables in consonance with the fitted model. For instance, if the substantive model under consideration is a two-factor model with indicators x₁–x₄ and y₁–y₄, respectively, the covariance matrix should match this model-induced ordering of the indicators. No pattern between these residuals should be apparent; rather, the residuals should be well scattered. However, unlike score tests (known as modification indices in the SEM literature, Sörbom, 1989), standardized residual covariances do not immediately suggest how to modify the model to obtain a better fit. As a result, standardized residual covariances have not received as much attention as modification indices and little is known about their small sample behavior, as well as the behavior of their standard errors. In this article, we aim at filling this gap by performing a small simulation study to investigate the small sample behavior of standardized residual covariances (and of residual correlations) and their standard errors. In particular, we examine coverage rates for standardized residual covariances and residual correlations for different degrees of model misspecification and non-normality. In addition, we compare the empirical Type I errors and power rates of standardized residual covariances and modification indices.

The remainder of this article is organized as follows: First, we review statistical theory regarding confidence intervals for standardized residual covariances and residual correlations both under normality assumptions and under the asymptotically distribution free assumptions set forth by Browne (1982, 1984). Next, we report the results of our small simulation study. Then, we present an applied example to illustrate the use of standardized residual covariances versus modification indices. We conclude with a general discussion and some recommendations for applied users.

Statistical Theory for Standardized Residual Covariances

Let p denote the number of observed variables being modeled, N denote sample size, and let σ denote a t = p(p + 1)/2 vector of population covariances. We consider a covariance structure σ₀ = σ(θ) where θ is a q ≤ t vector of parameters to be estimated from the data. The two best known procedures for estimating the model parameters involve minimizing the discrepancy functions

Under the null hypothesis, for any member of the class of estimators (1), as well as for the ML estimator (for details see Browne & Arminger, 1995), the covariance matrix of the estimated parameters is

In Equation 3 and NΓ is the asymptotic covariance matrix of the sample covariances, s, that is,

Γ can be computed under normality assumptions (Γ_NT) or under ADF assumptions (Γ_ADF; e.g., non-normality). For asymptotically optimal (AO) estimators, that is, for estimators with minimum variance within this class of estimators, (3) simplifies to

Under normality assumptions, the AO estimators are the (NT, normal theory) generalized least squares (GLS) estimator and the ML estimator. Under ADF assumptions, the AO estimator is the WLS estimator.

We now turn to the residual covariances (and variances) , where , with typical element . For any for the estimators considered in this paper, their asymptotic covariance matrix is

Using Γ_NT or Γ_ADF in Equation 6, one obtains standard errors for the residual covariances under normality or robust to non-normality using the square root of the diagonal elements of Equation 6. For AO estimators, it simplifies to

Note that if ML estimation is used with robust standard errors, Equation 6 should be used, not Equation 7, as the ML estimator is not asymptotically optimal under ADF assumptions.

Standardized Residual Covariances and Residual Correlations

Residual covariances cannot be substantively interpreted as their magnitude depends on the units of the variables. To overcome this problem, we can divide the residual covariances by their sample standard deviations leading to the standardized residual or in matrix form , where and s_ii denotes a p vector of sample variances. Instead of using standardized residual covariances, we could use the t − p residual correlations . In matrix form they are written as , where .

Asymptotic standard errors for the standardized residuals can be obtained (Ogasawara, 2001) as the square root of the diagonal elements of

(8) where , and acov(e_u) is given by Equation 6. For AO estimators only, we may use Equation 7 instead of Equation 6. Similarly, asymptotic standard errors for the residual correlations can be obtained as the square root of the diagonal elements of

(9) where (Maydeu-Olivares, 2017).

Confidence Intervals and Wald (z) Statistics

Letting σ denote the population covariances, under parameter drift assumptions, the sample standardized residual covariances and residual correlations are asymptotically normally distributed with mean equal to their population counterparts

(10) and covariance matrices given by Equations 8 and 9. As a result, in large samples, the (100 − α)% confidence interval for Ɛ (i.e., Ɛ_s or Ɛ_r) can be obtained using

(11) where denotes the estimate of the effect size (e.g., the sample standardized residual covariance) and ASE denotes its asymptotic standard error. Similarly, we can test whether a population standardized residual covariance or residual correlation equals some cut-off criteria c (usual zero) using

(12)

Note that the z statistics for standardized residual covariances must equal the z statistics for unstandardized residual covariances. This is because and . Therefore, . Similarly, in well-fitting models, the variances estimated under the model will be close to the sample variances, or In such cases , and the residual correlations will be very similar to the standardized residual covariances and their z statistics will be very similar as well.

A Monte-Carlo Comparison of Standardized Residual Covariances, Correlation Residuals, and Modification Indices

To investigate the coverage rates of standardized residual covariances and residual correlations, we performed a small simulation study with 16 conditions. The conditions were obtained by crossing

• (a)
  sample size (small, N = 100; large, N = 1,000),
• (b)
  model size (small, p = 10 observed variables; large, p = 30),
• (c)
  model misspecification (correctly specified, and misspecified),
• (d)
  distribution of the data (normal, non-normal).

For each condition 1,000 replications were obtained.

More specifically, we generated multivariate normal data with mean zero and a covariance structure conforming to an independent clusters two-factor model with factor loadings equal to 0.7 and uniqueness equal to 0.51 for all variables. The data was then discretized into seven categories coded 0–6 where the thresholds were selected so that the observed data had the desired skewness and kurtosis. A one-factor model was fitted in all cases using ML estimation using the Lavaan package in R (Rosseel, 2012). Standard errors were computed under normality assumptions, and also under ADF assumptions (see Satorra & Bentler, 1994).

The misspecified model was obtained by setting the correlation between the factors to ρ = 0.8 when generating the data. The correctly specified model was obtained by setting the correlation to ρ = 1. The threshold values used for all variables in the normal and non-normal conditions are provided in Table 1. These thresholds yield a population skewness of 0 and (excess) kurtosis of 0 for the normal condition, and of −2 and 3, respectively, for the non-normal conditions. Table 1 also contains the population Root Mean Square Error of Approximation (RMSEA; Browne & Cudeck, 1993), population Standardized Root Mean squared Residual (SRMR), and Correlation Root Mean squared Residual (CRMR). The computation of these population quantities is described in Maydeu-Olivares (2017). The computation of the population skewness and kurtosis for discretized normal variables is described in Maydeu-Olivares, Coffman, and Hartmann (2007).

Table 1 Population thresholds, skewness, kurtosis, RMSEA, SRMR, and CRMR

p	ρ	Item thresholds	Kur.	Skew.	RMSEA	SRMR	CRMR
Notes. RMSEA = root mean square error of approximation; SRMR = standardized root mean squared residual; CRMR = correlation root mean squared residual. For correctly specified models (ρ = 1) the same thresholds were used, leading to the same values of kurtosis and skewness. In this case, however, the population RMSEA, SRMR, and CRMR are all zero.
10	0.8	−1.64, −1.08, −0.52, 0.52, 1.08, 1.64	0	0	0.075	0.041	0.046
10	0.8	−2.33, −1.88, −1.55, −1.17, −0.84, −0.55	3.2	−2.0	0.063	0.039	0.043
30	0.8	−1.64, −1.08, −0.52, 0.52, 1.08, 1.64	0	0	0.054	0.044	0.046
30	0.8	−2.33, −1.88, −1.55, −1.17, −.84, −0.55	3.2	−2.0	0.046	0.042	0.043

Table 1 Population thresholds, skewness, kurtosis, RMSEA, SRMR, and CRMR

View as image HTML

We see in Table 1 that the population RMSEA values for misspecified models range from 0.046 to 0.075; population SRMR values range from 0.039 to 0.042, and population CRMR values range from 0.043 to 0.046. Note that these population values are smaller for non-normal data than for normal data.

We provide in Table 2 the empirical Type I error rates (for correctly specified models) and empirical power rates (for misspecified models) of z statistics (12) for testing whether the population standardized residual covariances and residual correlations equal zero (both under normality and ADF assumptions). We also provide in this table the empirical Type I errors and power rates for modification indices for comparison.

Table 2 Empirical rejection rates at α = 0.05 for modification indices, standardized residual covariances, and residual correlations

					Modification indices						Standardized residual covariances						Residual correlations
					NT			ADF			NT			ADF			NT			ADF
p	N	ρ	Kur.	Skew.	Min	Max	Med	Min	Max	Med	Min	Max	Med	Min	Max	Med	Min	Max	Med	Min	Max	Med
Notes. p = Number of observed variables; N = sample size; ρ = population inter-factor correlation; Kur. = excess kurtosis; Skew. = skewness; NT = normality assumptions; ADF = asymptotically distribution free assumptions; med = median.
10	100	1	0	0	0.04	0.07	0.05	0.04	0.07	0.05	0.04	0.07	0.05	0.04	0.08	0.06	0.04	0.07	0.05	0.04	0.08	0.06
10	100	1	3.2	−2	0.13	0.18	0.15	0.04	0.08	0.06	0.12	0.17	0.14	0.04	0.08	0.06	0.12	0.17	0.14	0.04	0.08	0.06
10	1000	1	0	0	0.04	0.07	0.05	0.04	0.07	0.05	0.04	0.07	0.05	0.04	0.07	0.05	0.04	0.07	0.05	0.04	0.07	0.05
10	1000	1	3.2	−2	0.13	0.17	0.15	0.03	0.06	0.05	0.13	0.17	0.15	0.04	0.07	0.05	0.13	0.17	0.15	0.04	0.07	0.05
30	100	1	0	0	0.03	0.08	0.05	0.03	0.08	0.05	0.03	0.07	0.05	0.03	0.08	0.06	0.03	0.07	0.05	0.03	0.08	0.06
30	100	1	3.2	−2	0.12	0.18	0.15	0.03	0.08	0.06	0.11	0.17	0.14	0.04	0.09	0.06	0.11	0.17	0.14	0.04	0.09	0.06
30	1000	1	0	0	0.03	0.07	0.05	0.03	0.07	0.05	0.03	0.07	0.05	0.03	0.07	0.05	0.03	0.07	0.05	0.03	0.07	0.05
30	1000	1	3.2	−2	0.12	0.18	0.15	0.03	0.07	0.05	0.12	0.18	0.14	0.03	0.07	0.05	0.12	0.18	0.14	0.03	0.07	0.05
10	100	0.8	0	0	0.10	0.19	0.13	0.10	0.19	0.13	0.11	0.15	0.13	0.12	0.17	0.14	0.11	0.15	0.13	0.12	0.17	0.14
10	100	0.8	3.2	−2	0.14	0.26	0.18	0.04	0.14	0.08	0.14	0.23	0.18	0.06	0.13	0.10	0.14	0.23	0.18	0.06	0.13	0.10
10	1000	0.8	0	0	0.67	0.89	0.72	0.66	0.88	0.71	0.68	0.88	0.73	0.68	0.88	0.72	0.68	0.88	0.73	0.68	0.88	0.72
10	1000	0.8	3.2	−2	0.50	0.70	0.55	0.31	0.52	0.35	0.51	0.69	0.56	0.34	0.47	0.40	0.51	0.69	0.56	0.34	0.47	0.40
30	100	0.8	0	0	0.10	0.17	0.13	0.10	0.17	0.13	0.10	0.15	0.12	0.10	0.17	0.13	0.10	0.15	0.12	0.10	0.17	0.13
30	100	0.8	3.2	−2	0.14	0.23	0.19	0.05	0.12	0.08	0.14	0.22	0.18	0.05	0.14	0.09	0.14	0.22	0.18	0.05	0.14	0.09
30	1000	0.8	0	0	0.66	0.79	0.72	0.66	0.79	0.72	0.66	0.79	0.72	0.66	0.78	0.72	0.66	0.79	0.72	0.66	0.78	0.72
30	1000	0.8	3.2	−2	0.48	0.61	0.55	0.29	0.42	0.36	0.49	0.60	0.54	0.31	0.40	0.36	0.49	0.60	0.54	0.31	0.40	0.36

Table 2 Empirical rejection rates at α = 0.05 for modification indices, standardized residual covariances, and residual correlations

View as image HTML

Now, the fitted model is a one-factor model, , where λ denotes the p vector of factor loadings and Θ denotes the p × p covariance matrix among the unique errors in the factor model. For this choice of fitted model, only modification indices for the off-diagonal elements of Θ are computed. For each model, there are t − q modification indices, t − q z statistics for residual correlations; we used only the t − q z statistics for residual covariances (we excluded the tests for residual variances). For each replication and choice of test statistic, we computed the minimum, maximum, and median p-value across the t − q tests of the population parameter being zero. Table 2 provides the average of these minimum, maximum, and medians across the 1,000 replications of each condition. For correctly specified models (ρ = 1), rejection rates should be as close as possible to the nominal rate (0.05); for incorrectly specified models (ρ = 0.8), rejection rates should be as large as possible.

We see in Table 2 that rejection rates for modification indices, standardized residual covariances, and residual correlations are very similar. When the statistics are computed assuming normality and the data is actually non-normal, all three statistics will suggest that the model fits more poorly than what it really does (p values are too large). In all other conditions, rejection rates are right on target across all conditions, even at the smallest sample size (N = 100) and largest model (p = 30) considered.

In examining rejection rates for the misspecified model, results obtained under normality assumptions when the data is non-normal should be ignored, as empirical Type I errors are incorrect. Inspecting the other conditions, we see that (a) power is very similar across the three statistics considered (modification indices, z statistics for residual covariances, and z statistics for residual correlations), (b) power logically increases for increasing sample size, (c) power does not increase for increasing model size, and (d) power does not increase when data is normally distributed and normality is assumed. Since there appears to be no gain in assuming normality when the data is combining the Type I and power results we conclude that it is preferable to use ADF assumptions.

Table 3 provides 95% coverage rates for the standardized residual covariances and residual correlations. Population values ranged from 0.041 to 0.051 (in absolute value) in both cases. Consistent with the results presented for the test statistics, we see Table 3 that the coverage rates are rather accurate, across all conditions of model size, normality, sample size, and model misspecification. The exception is confidence intervals computed under normality assumptions when the data is non-normal. In this case, confidence intervals are too narrow: median coverage rate is 0.86 for 95% intervals.

Table 3 95% Coverage rates for standardized residual covariances and residual correlations

					Standardized residual covariances						Residual correlations
					NT			ADF			NT			ADF
p	N	ρ	Kur.	Skew.	Min	Max	Med	Min	Max	Med	Min	Max	Med	Min	Max	Med
Notes. p = Number of observed variables; N = sample size; ρ = population inter-factor correlation; Kur. = excess kurtosis; Skew. = skewness; NT = normality assumptions; ADF = asymptotically distribution free assumptions.
10	100	1	0	0	0.93	0.96	0.95	0.92	0.96	0.94	0.93	0.96	0.95	0.92	0.96	0.94
10	100	1	3.2	−2	0.83	0.88	0.86	0.92	0.96	0.94	0.83	0.88	0.86	0.92	0.96	0.94
10	1,000	1	0	0	0.93	0.96	0.95	0.93	0.96	0.95	0.93	0.96	0.95	0.93	0.96	0.95
10	1,000	1	3.2	−2	0.83	0.87	0.85	0.93	0.96	0.95	0.83	0.87	0.85	0.93	0.96	0.95
30	100	1	0	0	0.93	0.97	0.95	0.92	0.97	0.95	0.93	0.97	0.95	0.92	0.97	0.95
30	100	1	3.2	−2	0.83	0.89	0.86	0.91	0.96	0.94	0.83	0.89	0.86	0.91	0.96	0.94
30	1,000	1	0	0	0.93	0.97	0.95	0.93	0.97	0.95	0.93	0.97	0.95	0.93	0.97	0.95
30	1,000	1	3.2	−2	0.82	0.88	0.86	0.93	0.97	0.95	0.82	0.88	0.86	0.93	0.97	0.95
10	100	0.8	0	0	0.93	0.96	0.95	0.93	0.96	0.94	0.93	0.96	0.95	0.93	0.96	0.94
10	100	0.8	3.2	−2	0.83	0.90	0.86	0.91	0.96	0.94	0.83	0.90	0.86	0.91	0.96	0.94
10	1,000	0.8	0	0	0.92	0.97	0.95	0.92	0.96	0.95	0.92	0.97	0.95	0.92	0.96	0.95
10	1,000	0.8	3.2	−2	0.83	0.89	0.86	0.93	0.96	0.95	0.83	0.89	0.86	0.93	0.96	0.95
30	100	0.8	0	0	0.93	0.97	0.95	0.92	0.97	0.94	0.93	0.97	0.95	0.92	0.97	0.94
30	100	0.8	3.2	−2	0.83	0.91	0.87	0.92	0.96	0.94	0.83	0.91	0.87	0.92	0.96	0.94
30	1,000	0.8	0	0	0.93	0.97	0.95	0.93	0.97	0.95	0.93	0.97	0.95	0.93	0.97	0.95
30	1,000	0.8	3.2	−2	0.82	0.90	0.86	0.93	0.97	0.95	0.82	0.90	0.86	0.93	0.97	0.95

Table 3 95% Coverage rates for standardized residual covariances and residual correlations

View as image HTML

An Example: Modeling the LOT

The Life Orientation Test (LOT; Scheier & Carver, 1985) consists of eight items designed to measure generalized outcome expectancies plus four filler items. The response scale for the items is graded, consisting of five points (0, …, 4). Four of the items are positively worded, while the remaining items are negatively worded.

The LOT was designed to measure a single construct. However, several factor analytic studies (e.g., Chang, D’Zurilla, & Maydeu-Olivares, 1994; Chang & McBride-Chang, 1996; Marshall, Wortman, Kusulas, Hervig, & Vickers, 1992; Robinson-Whelen, Kim, MacCallum, & Kiecolt-Glaser, 1997; Scheier & Carver, 1985) have revealed that a one-factor model does not fit well. Instead, a correlated two-factor model in which all positively worded items load on one factor and all negatively worded items load on another factor provides a substantially better fit to the data (but see Maydeu-Olivares & Coffman, 2006).

Here, we fitted a one-factor model by ML to data gathered from 389 respondents by Chang et al. (1994). The model does not fit well. Under normality assumptions the 90% confidence interval (CI) for the RMSEA is [.15, .19]; for the SRMR it is [.08, .12], and for the CRMR it is [.09, .12]. These data is quite are normal: Excess kurtosis ranges from −0.72 to 0.01; skewness ranges from −0.57 to 0.71. As a result CIs for the RMSEA, SRMR, and CRMR computed under ADF assumptions equal those computed under normality to three decimals. The CI for the RMSEA robust to non-normality was computed using the mean adjusted RMSEA as described in Brosseau-Liard, Savalei and Li (2012).

Next, we can examine the standardized residual covariances or residual correlations to gauge the extent of the misfit. In this example residual covariances and residual correlations (and their z statistics) are equal up to two decimals. As a result, only standardized residual covariances are reported and they are shown in Table 4. Residual variances in this example are zero and they have not been printed in this table. In addition, we have ordered the LOT items as (1, 4, 5, 11, 3, 8, 9, 12). The first four items are positively worded (they reflect optimism), and the last four are negatively worded (they reflect pessimism). We see in this table that residuals among the negatively worded items are small. Residuals are also small between positively and negatively worded items. Yet, they are large among the positively worded items. Thus, after suitably reordering the items according to a substantive theory with two factors (optimism and pessimism), the pattern we observe in this table is the typical pattern we would observe if a two-factor model is the data generating model but a one-factor model is fitted.

Table 4 Results for a one-factor model applied to the Life Orientation Test (LOT) data: standardized residual covariances

Item	1	4	5	11	3	8	9	12
Notes. We have boldfaced the standardized residual covariances that are statistically significant at the 5% level after applying a Bonferroni correction. Above the diagonal we have boldfaced statistics significant under asymptotically distribution free (ADF) assumptions; below the diagonal, statistics significant under normality (NT) assumptions.
1		0.31	0.25	0.13	−0.08	−0.05	−0.07	−0.05
4	0.31		0.29	0.19	−0.08	−0.04	−0.11	0.01
5	0.25	0.29		0.08	−0.03	−0.06	−0.07	−0.02
11	0.13	0.19	0.08		−0.08	−0.06	−0.02	0.05
3	−0.08	−0.08	−0.03	−0.08		0.01	0.03	0.04
8	−0.05	−0.04	−0.06	−0.06	0.01		0.05	−0.01
9	−0.07	−0.11	−0.07	−0.02	0.03	0.05		0.00
12	−0.05	0.01	−0.02	0.05	0.04	−0.01	0.00

Table 4 Results for a one-factor model applied to the Life Orientation Test (LOT) data: standardized residual covariances

View as image HTML

In inspecting standardized residual covariances it is important to take into account their sampling variability. One way to do so is to indicate, as we have done in Table 4, which of these effect sizes is statistically significant. More specifically, we have boldfaced in this table the standardized residual covariances that are statistically significant at the α = 0.05 level using a Bonferroni correction. That is, since there are t = 28 residual covariances, we used α = 0.05/28 = 0.0018. Of course, a more precise procedure, such as the Benjamini-Hochberg procedure (see Thissen, Steinberg, & Kuang, 2002), could have been used.

Alternatively, confidence intervals for the standardized residual covariances can be constructed. We display in Table 5 90% confidence intervals for these population parameters. Do modification indices provide a similar picture in this application? To address this question, we provide in Table 6 the modification indices for this model both under normality robust to non-normality assumptions. Comparing Tables 4 and 6 we see that modification indices and standardized residual covariances provide very much the same picture for these data, both under normality and under ADF assumptions. Under normality, both sets of statistics suggest that there is some misfit associated to Item 9, in addition to the block misfit due to the misspecification of the latent trait dimensionality. Standardized residual covariances immediately convey the magnitude of the misfit: it is not large. Furthermore, these statistics fail to be significant when the precision with which they are estimated is computed under ADF assumptions.

Table 5 Results for a one-factor model applied to the Life Orientation Test (LOT) data: 90% confidence intervals for standardized residual covariances

Item	1	4	5	11	3	8	9	12
Note. Above the diagonal, under ADF assumptions; below the diagonal, under normality assumptions.
1		0.19, 0.43	0.14; 0.37	0.00, 0.26	−0.18, 0.02	−0.11, 0.01	−0.14, 0.00	−0.14, 0.05
4	0.19, 0.44		0.16, 0.43	0.06, 0.31	−0.17, 0.01	0.11, 0.03	−0.17, −0.04	−0.08, 0.10
5	0.13, 0.38	0.17, 0.41		−0.05, 0.20	−0.14, 0.08	−0.12, 0.01	−0.15, 0.00	−0.12, 0.08
11	0.00, 0.26	0.06, 0.31	−0.04, 0.20		−0.19, 0.03	−0.12, −0.01	−0.10, 0.06	−0.05, 0.14
3	−0.18, 0.02	−0.17, 0.01	−0.12, 0.07	−0.18, 0.02		−0.05, 0.07	−0.04, 0.10	−0.06, 0.14
8	−0.10, 0.01	−0.09, 0.01	−0.11, −0.01	−0.12, 0.00	−0.04, 0.06		0.00, 0.10	−0.06, 0.04
9	−0.14, −0.01	−0.16, −0.05	−0.13, −0.01	−0.10, 0.05	−0.02, 0.09	0.02, 0.08		−0.06, 0.06
12	−0.14, 0.04	−0.07, 0.10	−0.10, 0.07	−0.05, 0.15	−0.04, 0.12	−0.05, 0.03	−0.05, 0.05

Table 5 Results for a one-factor model applied to the Life Orientation Test (LOT) data: 90% confidence intervals for standardized residual covariances

View as image HTML

Table 6 Results for a one-factor model applied to the Life Orientation Test (LOT) data: modification indices for covariances among error terms

Item	1	4	5	11	3	8	9	12
Notes. Statistics computed under asymptotically distribution free (ADF) assumptions above the diagonal, under normality assumptions (NT) below the diagonal. We have boldfaced the statistics that are significant at the 5% level after applying a Bonferroni correction.
1		52.38	32.94	6.97	4.49	4.40	7.83	1.78
4	66.06		52.10	17.17	5.06	3.82	20.49	0.11
5	41.54	65.71		2.76	0.63	7.38	8.53	0.25
11	8.80	21.66	3.49		3.85	7.00	0.78	1.64
3	5.67	6.38	0.80	4.86		0.24	2.74	1.71
8	5.55	4.82	9.31	8.83	0.30		29.75	0.24
9	9.88	25.84	10.76	0.98	3.46	37.52		0.05
12	2.24	0.14	0.32	2.07	2.16	0.31	0.06

Table 6 Results for a one-factor model applied to the Life Orientation Test (LOT) data: modification indices for covariances among error terms

View as image HTML

Discussion and Conclusions

Bivariate standardized effect sizes (standardized residual covariances and residual correlations) may be a helpful tool to detect the source of mispecified covariance structure models. In factor analysis applications, after suitably ordering the observed variables, we look for blocks of residuals as an indication of mispecified latent trait dimensionality, as well as for rows of residuals as an indication of mispecified secondary loadings. Factor analysis is often used in test construction. Within this context it is often of interest to remove poor fitting indicators. To this aim, we find helpful printing the average of the absolute value of the residual correlations (or standardized residual covariances) for each item (Maydeu-Olivares, 2015). It is also possible to use residuals to detect the source of misfit in path analysis models, but see Costner and Schoenberg (1978). Admittedly, when the assumptions underlying the use of modification indices are met (misfit caused by a few omitted paths), one should expect modification indices to be more helpful than residuals in suggesting ways to improve the fit of the model. Otherwise, as the example presented here illustrates, modification indices and residuals provide similar information.

Standardized residual covariances (or residual correlations) also provide effect sizes of the misfit of the model. This is important, as in models involving a large number of variables one often needs to settle for approximate models due to time constraints. Also, when large sample sizes are used, well-fitting models may be rejected due to high power. In both cases, these standardized effect sizes need to be examined to gauge qualitatively the extent of the misfit. The use of overall effect sizes of the model’s misfit of such as the SRMR should not be used in the presence of large standardized residual covariances. Of course, modification indices can also be used to gauge the effect size of the misfit (via the use of expected parameter change, see Saris, Satorra, & van der Veld, 2009). We have shown in this paper that modification indices and residuals show similar Type I and II errors. However, often we find the use of standardized residual covariances more convenient than modification indices and expected parameter changes to convey the magnitude of a model’s misfit.

In closing, we have demonstrated that, under the conditions investigated, standardized residual covariances and residual correlations display a similar behavior. However, the conditions investigated here involved scale invariant models. Further research involving models that are not scale invariant is needed in helping choose between both sets of statistics.