Longitudinal Measurement Invariance of the DSM-5 Anxiety and Depression Severity Measures
Abstract
Abstract: There is scarce evidence on the psychometric properties of the Severity Measures (SMs) of the DSM-5 longitudinally, especially among Spanish young adults. Thus, the main objective of the present study was to analyze the Longitudinal Measurement Invariance (LMI) of the SMs for Depression, Generalized Anxiety, Social Anxiety, Panic, Separation Anxiety, Agoraphobia, and Specific Phobia scales in four assessment waves at 6-month intervals. A total of 564 Spanish young adults (Mage = 21.5 years; 67.9% females) participated in the study. Confirmatory Factor Analyses (CFAs) were carried out at each wave. Once the adequacy of the factor structure of the models was confirmed, three levels of LMI were examined (i.e., configural, metric, and scalar). Finally, we examined internal consistency (i.e., Cronbach’s α and ω) across assessment waves and gender groups. CFAs analyses supported a two-factor solution for the specific phobia scale and a one-factor solution for the rest of the SMs in baseline models. Evidence of LMI at the configural, metric, and scalar levels was observed for all seven SMs. Reliability indices ranged from .73 to .96. These results suggest that the DSM-5 SMs are useful assessment tools for longitudinal and follow-up studies in Spanish young adults.
The categorical diagnostic classification system for mental health disorders has been questioned for more than 10 years due to a lack of empirical data supporting the fundamental assumption that psychopathology refers to discrete phenomena (Kotov et al., 2021). For example, studies have consistently found evidence of continuity between psychopathological symptoms and normality (Haslam et al., 2020). Further, clinical features, and not etiological assumptions, tend to define the criteria evaluation system (Brown & Barlow, 2005; Krueger et al., 2018). To this end, even though the categories make clinical decisions easier, they only do so in presence versus absence terms (Krueger et al., 2018), which is problematic given that numerous threshold problems have been identified. For example, high rates of Not Otherwise Specified diagnoses have been applied (Kotov et al., 2021). Moreover, the temporal stability of taxometric diagnoses is very low (Bromet et al., 2011), thereby leading to a loss of clinical information and reliability (MacCallum et al., 2002; Markon et al., 2011). Consequently, the DSM-5 Task Force outlined the need to consider the dimensional approach of psychopathology while revising the new edition of the DSM (Kraemer, 2007). Although the categorical system was retained within the latest edition (i.e., the DSM-5), a new section provides several dimensional assessment tools (American Psychiatric Association [APA], 2013).
DSM-5 Section III comprises two distinct groups of measures: (1) Self-Rated Cross-Cutting Symptom Measures, which assess symptoms across diagnostic categories (see Narrow et al., 2013 for further details); and (2) Severity Measures (SMs), which assess disorder-specific symptoms, such anxiety-specific problems (social anxiety, agoraphobia, specific phobia, separation anxiety, panic, generalized anxiety disorder), depression, dissociative symptoms, and two measures for problems related to stress (posttraumatic and acute stress symptoms). The SMs were designed as short self-reported measures to be administrated both at an initial intake interview and over time to track the severity of the individual’s disorder and response to treatment (APA, 2013). Considering their short design (i.e., up to 10 items), the DSM-5 SMs could be useful to cover different psychopathological conditions related to internalizing symptoms in a short evaluation period. This point is especially relevant in some at-risk groups, such as college students, not only due to their high prevalence of distress problems but also their associations with, for example, suicidal thoughts and behaviors (Auerbach et al., 2019; Bravo et al., 2018). Thus, providing short self-reported measures, based on the DSM-5, to cover different psychopathological conditions related to distress (i.e., anxiety-related problems and depression SMs) could be of great utility. For example, a short assessment could help to reduce time spent on assessing individuals (National Guidelines Alliance, 2016), thereby reducing the waiting lists for psychological assistance services, a common barrier identified in undergraduate students to participate in psychological treatments (Vidourek et al., 2014).
Previous studies conducted across various countries have provided reliability and validity evidence of the anxiety SMs scores among samples of non-clinical young adults (US, LeBeau et al., 2012; Germany, Knappe et al., 2013; Spain, Vidal-Arenas et al., 2021) non-clinical adults (Brazil, DeSousa et al., 2017; Germany, Knappe et al., 2013; The Netherlands, Möller & Bögels, 2016), and clinical populations (US, LeBeau et al., 2012, 2016); Germany, Beesdo-Baum et al., 2012, Knappe et al., 2013). These studies have found evidence to support the unidimensionality for most of the anxiety SMs (DeSousa et al., 2017; Vidal-Arenas et al., 2021; Yalın Sapmaz et al., 2017). One exception is the Specific Phobia domain, which incorporates an anxiety and avoidance subscale (Vidal-Arenas et al., 2021). All these prior studies provided reliable evidence of its scores (Cronbach’s α from .82 to .98) (LeBeau et al., 2012; Vidal-Arenas et al., 2021).
The SM for assessing depressive symptoms incorporated in the new section of the DSM-5 (APA, 2013) is an adaptation from the 9-item Patient Health Questionnaire (PHQ-9; Löwe et al., 2004). Some studies have found a two-factor solution in which a somatic factor and a cognitive-affective factor are differentiated (Beard et al., 2016; Chilcot et al., 2013; Guo et al., 2017; Keum et al., 2018; Petersen et al., 2015). This two-factor solution is invariant across genders (Petersen et al., 2015) and time (Guo et al., 2017). However, specific items that load onto each factor differ across these prior studies (Beard et al., 2016; Chilcot et al., 2013; Guo et al., 2017; Keum et al., 2018). In combination with the fact that both factors are highly correlated (from .85 to .97; Beard et al., 2016; González-Blanch et al., 2018; Keum et al., 2018), some researchers suggest that a one-factor solution could better depict the structure of the PHQ-9 (Boothroyd et al., 2019). To this end, there is evidence of a one-factor solution among clinical (González-Blanch et al., 2018) and general populations (Kocalevent et al., 2013). This unidimensional structure has also been shown to be invariant across gender and differing age groups among Chinese adolescents (Leung et al., 2020).
Although previous studies have provided some preliminary validity and reliability evidence of the DSM-5 SMs scores, limited research has explored their psychometric properties over time. This is an important gap, as the dimensional measures of the DSM-5 were mainly created to circumvent the problem of the categorical diagnosis structure to detect changes in psychopathology across time when patients are attending therapy (APA, 2013), among other purposes.
For these reasons, the present study aimed to examine the psychometric properties of Depression, Generalized Anxiety, Social Anxiety, Panic, Separation Anxiety, Agoraphobia, and Specific Phobia scales from the DSM-5 over time in a sample of undergraduate students from Spain. Specifically, we examined the Longitudinal Measurement Invariance (LMI) of each SM across four assessment waves, and we provide reliable evidence of its scores at each wave. We expected that the structure of the SMs would be invariant over time, showing evidence of reliability at each assessment wave.
Methods
Participants and Procedure
College students from a university located in the eastern region of Spain participated in an 18-month longitudinal-funded project (code. UJI-A2017-18, Universitat Jaume I) regarding personal mental health (i.e., internalizing and externalizing behaviors), personality traits, and contextual risk factors. Through online surveys, four waves of data were collected at 6-month intervals from early February 2018 to early December 2019, using the Qualtrics platform. Participants completed informed consents and received financial compensation at each assessment wave (i.e., €5, €10, €15, and €15 at Time 1 [T1], 2 [T2], 3 [T3], and 4 [T4], respectively). Given the aim of the present study, only data from participants that completed the DSM-5 SMs were analyzed (T1, n = 564; T2, n = 362; T3, n = 301; T4, n = 279). Negligible differences in magnitude were observed in terms of age between the original and final sample (d < .07). Participants were primarily females (T1 = 68.3%; T2 = 71.2%; T3 = 71.8%; T4 = 73.9%), with a mean age of 21.6 (SD = 3.65). Significant differences, but small in magnitude (ϕ = .12), between those who completed all waves compared with those who only completed the first wave were observed in terms of gender groups.
Instruments
Depression Severity Measure
Adapted from the Patient Health Questionnaire short version (Löwe et al., 2004), this measure assesses depressive symptoms on a 4-point response scale (0 = never, 3 = every day) based on 9 items (APA, 2013). We made some modifications to the Spanish version used (APA, 2014), such that: (1) we changed the time frame of assessment to report the symptoms experienced “in the last 6 months” instead of “the last 7 days”, and we also (2) adapted the statement for each scale to an online assessment format.
Anxiety Severity Measures
The Spanish version of generalized anxiety, social anxiety, panic, separation anxiety, agoraphobia, and specific phobia SMs were included in the present study (Vidal-Arenas et al., 2021). Each scale is composed of 10 items, which are rated on a 5-point scale from 0 = never to 4 = always.
Data Analysis
Before running Longitudinal Measurement Invariance (LMI) analyses, we examined the unidimensional structure for each measure at each wave using Confirmatory Factor Analyses (CFA), except for specific phobia, where we tested a two-factor structure. To evaluate overall model fit, we used the following criteria: Comparative Fit Index (CFI) > .90 (acceptable) > .95 (optimal), Tucker-Lewis Index (TLI) > .90 (acceptable) > .95 (optimal), Root Mean Square Error of Approximation (RMSEA) < .06 (Marsh et al., 2009). In addition, to examine the internal consistency of the SMs, we estimated Cronbach’s αs and ordinal ωs with 95% CIs (Dunn et al., 2014) at each assessment wave.
Once the adequacy of the factor structure of the models was confirmed, we next tested the LMI for each measure. In particular, three levels of measurement invariance were tested: (1) configural (test whether all items load on the proposed factor), (2) metric (test whether item-factor loadings are similar across time), and (3) scalar (test whether the unstandardized item thresholds are similar across time). To indicate a significant decrement in fit when testing for measurement invariance, we used model comparison criteria of ΔCFI/ΔTLI ≥ .01 (i.e., decrease indicates worse fit; Cheung & Rensvold, 2002) and ΔRMSEA ≥ .015 (i.e., increase indicates worse fit; Chen, 2007). For CFA and LMI, due to the non-normality observed in the data and the sample size (Li, 2016), a diagonally weighted least squares (WLSMV) model estimator was used.
Analyses were performed using Mplus 8.4, and SPSS v. 25. All data, analysis code, and research materials are available at DOI: 10.17605/OSF.IO/JZ4GE.
Results
Structure Validity Evidence and Reliability
Results from the CFAs of all SMs across the different waves of assessment are presented in Table 1. Overall, CFA analyses supported a two-factor solution for the specific phobia scale and a one-factor solution for the rest of the SMs at baseline models with acceptable to optimal fit indices (CFIs ≥ .927; TLIs ≥ .904; RMSEAs ≤ .177). The mean total score and internal consistency indexes for each DSM-5 severity measure at each time point (and across gender groups) are presented in Table 2. Cronbach’s αs and ordinal ωs ranged from .73 to .96. Following the cut-offs provided (APA, 2013), higher distress-related symptoms were observed (i.e., 64,4% and 52% reported medium to moderate depressive and generalized anxiety symptoms) compared with fears related problems (e.g., mild-severe 25.7% panic symptoms) at the intake (see Supplemental Table 5 in the OSF deposit). Normative data across each DSM scale are available in Supplemental Tables 1–4 at DOI: 10.17605/OSF.IO/JZ4GE.
Longitudinal Measurement Invariance
Once the structure of the SMs was established at each wave, further examination of LMI was performed (Table 1). Results showed good fit indexes for all the configurable models of the DSM-5 SMs (CFIs ≥ .931; TLIs ≥ .926; RMSEAs ≤ .088). When the constraints of the factor loading across waves were added, good fit indexes (CFIs ≥ .940; TLIs ≥ .938; RMSEAs ≤ .082) and an improvement of CFIs, TLIs, and RMSEAs compared with the previous model (i.e., configured) were found, which suggested metric invariance. The addition of constraints between the thresholds across the different assessment points of each scale also provided good fit indexes (CFIs ≥ .937; TLIs ≥ .938; RMSEAs ≤ .081) and negligible differences among CFI/TLI/RMSEA, suggesting scalar invariance.
Discussion
The Severity Measures (SMs) from the DSM-5 to assess symptoms of depressive, generalized anxiety, social anxiety, panic, separation anxiety, agoraphobia, and specific phobia symptoms were created, among other purposes, to help clinicians in detecting changes in psychopathology across time. However, limited research has studied the longitudinal measurement invariance of the scales, which is a necessary step before using the measures in follow-up assessments. Thus, the aims of the present study were: (1) to test the structure of the SMs across different assessment waves and provide reliable evidence of its scores, (2) to examine the Longitudinal Measurement Invariance (LMI) of the SMs by examining three invariance levels (i.e., configural, metric, and scalar).
When the structure of the SMs was tested independently at each wave, acceptable-to-optimal fit indexes were observed, supporting the hypothesized two-factor solution for the specific phobia SM and the one-factor solution for the rest of the anxiety SMs (DeSousa et al., 2017; Vidal-Arenas et al., 2021) and the depression SM (González-Blanch et al., 2018; Kocalevent et al., 2013; Leung et al., 2020). The only fit index that was over the standard cut-offs was the RMSEA (Marsh et al., 2009). However, this result could be expected due to the non-normality distribution observed of the item scores from each scale (skewness ≥ 1.5; kurtosis ≥ 3.0) and sample size (n ≥ 500) of the present study (Li, 2016) and it also similar to those found in previous studies about the structure of the DSM-5 SMs (DeSousa et al., 2017; Vidal-Arenas et al., 2021). For each SMs at each wave, adequate reliability coefficients were observed (> .73). Altogether, these results provide new evidence on the structure and reliability of the scores of the Spanish version of the DSM-5 SMs among Spanish undergraduates.
Once the structure of the scales was confirmed cross-sectionally, LMI was tested. The LMI analyses revealed that the factor solution tested for SMs (i.e., one-factor solution for depressive, generalized anxiety, social anxiety, panic, separation anxiety, and agoraphobia symptoms scales, and two-factor solution for the specific phobia symptoms scale) were invariant across time, such that all items loaded on the proposed factor (i.e., configural-invariance), item-factor loadings were similar across time (i.e., metric-invariance) and the unstandardized item thresholds were similar across time (i.e., scalar-invariance). Then, these findings support the utility of the DSM-5 anxiety and depression SMs for follow-up assessments.
The present study has some limitations that should be considered. The first limitation is that our participants were a small sample of undergraduates from Spain; therefore, future studies among other populations (e.g., clinical samples), and larger sample sizes are needed to investigate the replicability and generalizability of our findings. Moreover, further longitudinal studies should consider the inclusion of time-invariant and time-varying variables (e.g., SES), due to its effects in between and within-individual change in psychopathology (Wickrama et al., 2016), both psychological (e.g., personality), personal (e.g., clinical antecedents) and sociocontextual (e.g., ethnicity/race).
Despite these limitations, the present study provides evidence for the structure, reliability, and longitudinal measurement invariance of the Spanish DSM-5 anxiety and depression SMs. These results suggest that the DSM-5 SMs are useful tools to dimensionally assess the symptoms related to depression and anxiety DSM-5 disorder over time. The advantages of this are clear both at a clinical level, reducing the time spent on assessing individuals (National Guidelines Alliance, 2016) and facilitating improved monitoring of these symptoms across different phases of treatment; and at the research level, by decreasing participant fatigue in longitudinal studies, and helping the researcher cover a significant number of internalizing symptoms briefly.
References
. (2013). Diagnostic and statistical manual of mental disorders. APA.
. (2014). Diagnostic and statistical manual of mental disorders (5th ed.). Editorial Médica Panamérica.
(2019). Mental disorder comorbidity and suicidal thoughts and behaviors in the World Health Organization World Mental Health Surveys International College Student initiative. International Journal of Methods in Psychiatric Research, 28(2), Article
e1752 . https://doi.org/10.1002/mpr.1752(2016). Validation of the PHQ-9 in a psychiatric sample. Journal of Affective Disorders, 193, 267–273. https://doi.org/10.1016/J.JAD.2015.12.075
(2012). Psychometric properties of the dimensional anxiety scales for DSM-V in an unselected sample of German treatment seeking patients. Depression and Anxiety, 29(12), 1014–1024. https://doi.org/10.1002/da.21994
(2019). PHQ-9: One factor or two? Psychiatry Research, 271, 532–534. https://doi.org/10.1016/J.PSYCHRES.2018.12.048
(2018). College student mental health: An evaluation of the DSM-5 self-rated Level 1 cross-cutting symptom measure. Psychological Assessment, 30(10), 1382–1389. https://doi.org/10.1037/PAS0000628
(2011). Diagnostic shifts during the decade following first admission for psychosis. American Journal of Psychiatry, 168(11), 1186–1194. https://doi.org/10.1176/APPI.AJP.2011.11010048/ASSET/IMAGES/LARGE/AJP_168_11_1186_F002.JPEG
(2005). Dimensional versus categorical classification of mental disorders in the fifth edition of the diagnostic and statistical manual of mental disorders and beyond: Comment on the special section. Journal of Abnormal Psychology, 114(4), 551–556.
(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5
(2013). The factor structure of the PHQ-9 in palliative care. Journal of Psychosomatic Research, 75(1), 60–64. https://doi.org/10.1016/J.JPSYCHORES.2012.12.012
(2017). Psychometric properties of the dimensional anxiety scales for DSM-5 in a Brazilian community sample. International Journal of Methods in Psychiatric Research, 26(3), Article
e1531 . https://doi.org/10.1002/mpr.1531(2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412. https://doi.org/10.1111/bjop.12046
(2018). Factor structure and measurement invariance across various demographic groups and over time for the PHQ-9 in primary care patients in Spain. PLoS One, 13(2), Article
e0193356 . https://doi.org/10.1371/JOURNAL.PONE.0193356(2017). Factor structure and longitudinal measurement invariance of PHQ-9 for specialist mental health care patients with persistent major depressive disorder: Exploratory Structural Equation Modelling. Journal of Affective Disorders, 219, 1–8. https://doi.org/10.1016/j.jad.2017.05.020
(2020). Dimensions over categories: A meta-analysis of taxometric research. Psychological Medicine, 50(9), 1418–1432. https://doi.org/10.1017/S003329172000183X
(2018). Testing the factor structure and measurement invariance of the PHQ-9 across racially diverse US college Students. Psychological Assessment, 30(8), 1096–1106. https://doi.org/10.1037/PAS0000550
(2013). Dimensional anxiety scales for DSM-5: Sensitivity to clinical severity. European Psychiatry: The Journal of the Association of European Psychiatrists, 28(7), 448–456. https://doi.org/10.1016/j.eurpsy.2013.02.001
(2013). Standardization of the depression screener Patient Health Questionnaire (PHQ-9) in the general population. General Hospital Psychiatry, 35(5), 551–555. https://doi.org/10.1016/J.GENHOSPPSYCH.2013.04.006
(2021). The Hierarchical Taxonomy of Psychopathology (HiTOP): A quantitative nosology based on consensus of evidence. Annual Review of Clinical Psychology, 17(1), 83–108. https://doi.org/10.1146/annurev-clinpsy-081219-093304
(2007). DSM categories and dimensions in clinical and research contexts. International Journal of Methods in Psychiatric Research, 16(S1), S8–S15. https://doi.org/10.1002/mpr.211
(2018). Progress in achieving quantitative classification of psychopathology. World Psychiatry, 17(3), 282–293. https://doi.org/10.1002/wps.20566
(2012). A dimensional approach to measuring anxiety for DSM-5. International Journal of Methods in Psychiatric Research, 21(4), 258–272. https://doi.org/10.1002/mpr.1369
(2016). The DSM-5 Social Anxiety Disorder Severity Scale: Evidence of validity and reliability in a clinical sample. Psychiatry Research, 244, 94–96. https://doi.org/10.1016/J.PSYCHRES.2016.07.024
(2020). Measurement invariances of the PHQ-9 across gender and age groups in Chinese adolescents. Asia-Pacific Psychiatry, 12(3), Article
e12381 . https://doi.org/10.1111/APPY.12381(2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/S13428-015-0619-7
(2004). Measuring depression outcome with a brief self-report instrument: Sensitivity to change of the Patient Health Questionnaire (PHQ-9). Journal of Affective Disorders, 81(1), 61–66. https://doi.org/10.1016/S0165-0327(03)00198-8
(2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7(1), 19–40. https://doi.org/10.1037/1082-989X.7.1.19
(2011). The reliability and validity of discrete and continuous measures of psychopathology: A quantitative review. Psychological Bulletin, 137(5), 856–879. https://doi.org/10.1037/A0023678
(2009). In search of golden rules: Comment on hypothesistesting approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11(3), 320–341. https://doi.org/10.1207/S15328007SEM1103_2
(2016). The DSM-5 Dimensional Anxiety Scales in a Dutch non-clinical sample: Psychometric properties including the Adult Separation Anxiety Disorder Scale. International Journal of Methods in Psychiatric Research, 25(3), 232–239. https://doi.org/10.1002/mpr.1515
(2013). DSM-5 field trials in the United States and Canada, Part III: Development and reliability testing of a cross-cutting symptom assessment for DSM-5. The American Journal of Psychiatry, 170(1), 71–82. https://doi.org/10.1176/APPI.AJP.2012.12071000
. (2016). Mental Health Problems in People with Learning Disabilities: Prevention, Assessment and Management. National Institute for Health and Care Excellence (NICE). http://www.ncbi.nlm.nih.gov/pubmed/27683921
(2015). Factor structure and measurement invariance of the Patient Health Questionnaire – 9 for female and male primary care patients with major depression in Germany. Journal of Affective Disorders, 170, 138–142. https://doi.org/10.1016/J.JAD.2014.08.053
(2022). Five-factor model of personality & internalising psychopathology: New contributions for evaluation and aetiological models (Doctoral dissertation). Universitat Jaume I.
(2023). Longitudinal examination of DSM-5 anxiety and depression scales [Data, Materials]. https://OSF.IO/JZ4GE
(2021). Selfreported DSM-5 anxiety severity measures: Evidence of validity and reliability in Spanish youths. Psicothema, 33(2), 312–319. https://doi.org/10.7334/PSICOTHEMA2020.398
(2014). Students’ benefits and barriers to mental health help-seeking. Health Psychology and Behavioral Medicine, 2(1), 1009–1022. https://doi.org/10.1080/21642850.2014.963586
(2016). Higher-order growth curves and mixture modeling with Mplus: A practical guide. Routledge.
(2017). Validity and reliability of the Turkish version of DSM-5 Social Anxiety Disorder Severity Scale – Child Form. Noropsikiyatri Arsivi, 54(4), 354–357. https://doi.org/10.5152/npa.2017.17027
