From Bi-Dimensionality to Uni-Dimensionality in Self-Report Questionnaires
Applying the Random Intercept Factor Analysis Model to Six Psychological Tests
Abstract
Abstract. The common factor model – by far the most widely used model for factor analysis – assumes equal item intercepts across respondents. Due to idiosyncratic ways of understanding and answering items of a questionnaire, this assumption is often violated, leading to an underestimation of model fit. Maydeu-Olivares and Coffman (2006) suggested the introduction of a random intercept into the model to address this concern. The present study applies this method to six established instruments (measuring depression, procrastination, optimism, self-esteem, core self-evaluations, and self-regulation) with ambiguous factor structures, using data from representative general population samples. In testing and comparing three alternative factor models (one-factor model, two-factor model, and one-factor model with a random intercept) and analyzing differential correlational patterns with an external criterion, we empirically demonstrate the random intercept model’s merit, and clarify the factor structure for the above-mentioned questionnaires. In sum, we recommend the random intercept model for cases in which acquiescence is suspected to affect response behavior.
References
2000). Optimism and depression as predictors of physical and mental health functioning: The Normative Aging Study. Annals of Behavioral Medicine, 22, 127–130. https://doi.org/10.1007/BF02895776
(2015). On the factor structure of the Rosenberg (1965) General Self-Esteem Scale. Psychological Assessment, 27, 621. https://doi.org/10.1037/pas0000073
(2010). Much more than model fitting? Evidence for the heritability of method effect associated with positively worded items of the Life Orientation Test Revised. Structural Equation Modeling, 17, 642–653. https://doi.org/10.1080/10705511.2010.510064
(2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137–143. https://doi.org/10.1007/S11336-008-9100-1
(1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606. https://doi.org/10.1037/0033-2909.88.3.588
(2016). Procrastination, Distress and Life Satisfaction across the Age Range – A German Representative Community Study. PLoS One, 11, e0148054. https://doi.org/10.1371/journal.pone.0148054
(2000). Modeling acquiescence in measurement models for two balanced sets of items. Structural Equation Modeling, 7(4), 608–628.
(1980). Issues in the comparative measurement of political democracy. American Sociological Review, 45, 370–390.
(1989). Structural equations with latent variables. New York, NY: Wiley.
(2007). Psychometric properties of the CES-D scale among Colombian adults from the general population. Revista Colombiana de Psiquiatría, 36, 664–674.
(2019). Bevölkerung [Population]. Retrieved from https://www.destatis.de/DE/Themen/Gesellschaft-Umwelt/Bevoelkerung/_inhalt.html
. (1996). Messung des Selbstwertgefühls: Befunde zu Reliabilität, Validität und Stabilität der Rosenberg-Skala
([Measurement of self-esteem: Findings on reliability, validity, and stability of the Rosenberg Scale] . Diagnostica, 42, 284–292.2010). Standardization in psychological research. International Journal of Psychological Research, 3, 88–96. https://doi.org/10.21500/20112084.852
(2008). The German version of the Life-Orientation-Test (LOT-R) for dispositional optimism and pessimism. Zeitschrift für Gesundheitspsychologie, 16, 26–31. https://doi.org/10.1026/0943-8149.16.1.26
(2012). Psychometric properties and population-based norms of the Life Orientation Test Revised (LOT-R). British Journal of Health Psychology, 17, 432–445. https://doi.org/10.1111/j.2044-8287.2011.02046.x
(2003). Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: Do they matter? Personality and Individual Differences, 35, 1241–1254. https://doi.org/10.1016/S0191-8869(02)00331-8
(2012). Allgemeine Depressionsskala (ADS). Manual
([General Depression Scale (ADS). Manual] . Göttingen, Germany: Hogrefe.2011). Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16, 319–336. https://doi.org/10.1037/a0024917
(2017). The role of core self-evaluations in explaining depression and work engagement among managers. Current Psychology, 36, 516–529. https://doi.org/10.1007/s12144-016-9439-x
(1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74, 167. https://doi.org/10.1037/h0029780
(2003). The core self-evaluation scale: Development of a measure. Personnel Psychology, 56, 303–331. https://doi.org/10.1111/j.1744-6570.2003.tb00152.x
(1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151.
(2011). Race/ethnicity and the factor structure of the Center for Epidemiologic Studies Depression Scale: A meta-analysis. Cultural Diversity and Ethnic Minority Psychology, 17, 381. https://doi.org/10.1037/a0025434
(2019). Assessing Procrastination: Dimensionality and Measurement Invariance of the General Procrastination Scale–Screening (GPS-S) in a Representative Sample. European Journal of Psychological Assessment, 35, 633–640. https://doi.org/10.1027/1015-5759/a000441
(2012). Allgemeine Prokrastination: Entwicklung und Validierung einer deutschsprachigen Kurzskala der General Procrastination Scale (Lay, 1986)
([Development and validation of the German Short Scale of the General Procrastination Scale (Lay, 1986)] . Diagnostica, 58, 182–193. https://doi.org/10.1026/0012-1924/a0000601986). At last, my research article on procrastination. Journal of Research in Personality, 20, 474–495. https://doi.org/10.1016/0092-6566(86)90127-3
(2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavioral Research, 48, 936–949. https://doi.org/10.3758/s13428-015-0619-7
(1957). Objective tests as instruments of psychological theory. Psychological Reports, 3, 635–694. https://doi.org/10.2466/pr0.1957.3.3.635
(2005). Detecting and monitoring depression with a two-item questionnaire (PHQ-2). Journal of Psychosomatic Research, 58, 163–171. https://doi.org/10.1016/j.jpsychores.2004.09.006
(1986). Negative item bias in ratings scales for preadolescent children: A cognitive-developmental phenomenon. Developmental Psychology, 22, 37–49. https://doi.org/10.1037/0012-1649.22.1.37
(1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70, 810–819. https://doi.org/10.1037/0022-3514.70.4.810
(2006). Random intercept item factor analysis. Psychological Methods, 11, 344–362. https://doi.org/10.1037/1082-989X.11.4.344
(1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.
(2005). The role of item extremity in the emergence of keying-related factors: An exploration with the Life Orientation Test. Psychological Methods, 10, 120. https://doi.org/10.1037/1082-989X.10.1.120
(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543. https://doi.org/10.1007/BF02294825
(2016). Factorial Structure of the Perceived Stress Scale and implications for scoring. European Review of Applied Psychology, 66, 309–316. https://doi.org/10.1016/j.erap.2016.07.002
(2016). Method effects on an adaptation of the Rosenberg Self-Esteem Scale in Greek and the role of personality traits. Journal of Personality Assessment, 98, 178–188. https://doi.org/10.1080/00223891.2015.1089248
(2017). Method/group factors: Inconsequential but meaningful. A comment on Donnellan, Ackerman and Brecheen (2016). Journal of Personality Assessment, 99, 334–335.
(2016). Personality correlates and gender invariance of wording effects in the German version of the Rosenberg Self-Esteem Scale. Personality and Individual Differences, 97, 13–18. https://doi.org/10.1016/j.paid.2016.03.011
(2010). Comparison of the PHQ-9 and CES-D depression scales in systemic sclerosis: Internal consistency reliability, convergent validity and clinical correlates. Rheumatology, 49, 789–796. https://doi.org/10.1093/rheumatology/kep443
. (1997). The factor structure of the CES-D in two surveys of elderly Mexican Americans. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 52, 259–269. https://doi.org/10.1093/geronb/52B.5.S259
(2013). The limitations of model fit in comparing the bi-factor versus higher-order models of human cognitive ability structure. Intelligence, 41, 407–422. https://doi.org/10.1016/j.intell.2013.06.004
(2018). Recent methods for the study of measurement invariance with many groups: Alignment and random effects. Sociological Methods & Research, 47, 637–664. https://doi.org/10.1177/0049124117701488
(1978). Psychometric theory. New York, NY: McGraw-Hill.
(2005). What are we measuring? An evaluation of the CES-D across race/ethnicity and immigrant generation. Social Forces, 83, 1567–1601. https://doi.org/10.1353/sof.2005.0077
(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879. https://doi.org/10.1037/0021-9010.88.5.879
(2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569. https://doi.org/10.1146/annurev-psych-120710-100452
(2018). R: A language and environment for statistical computing [Computer software] Retrieved from https://www.R-project.org/. Vienna, Austria: R Foundation for Statistical Computing.
. (1977). The CES-D Scale: A Self-Report Depression Scale for research in the general population. Applied Psychological Measurement, 1, 385–401. https://doi.org/10.1177/014662167700100306
(2007). Method effects due to social desirability as a parsimonious explanation of the deviation from unidimensionality in LOT-R scores. Personality and Individual Differences, 42, 1597–1607. https://doi.org/10.1016/j.paid.2006.10.035
(2019). Latent variables should remain as such: Evidence from a Monte Carlo study. The Journal of General Psychology, 146, 417–442. https://doi.org/10.1080/00221309.2019.1596064
(2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129–140. https://doi.org/10.1080/00223891.2012.725437
(2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145–154. https://doi.org/10.1007/s11336-008-9102-z
(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354–373. https://doi.org/10.1037/a0029315
(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137. https://doi.org/10.1037/met0000045
(1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press.
(2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48, 1–36. Retrieved from http://www.jstatsoft.org/v48/i02/
(2008). Dimensionality and norms of the Rosenberg Self-esteem Scale in a German general population sample. European Journal of Psychological Assessment, 24, 190–197. https://doi.org/10.1027/1015-5759.24.3.190
(2014). Recovering substantive factor loadings in the presence of acquiescence bias: A comparison of three approaches. Multivariate Behavioral Research, 49, 407–424. https://doi.org/10.1080/00273171.2014.931800
(1994). Distinguishing optimism from neuroticism (and trait anxiety, selfmastery, and self-esteem): A re-evaluation of the Life Orientation Test. Journal of Personality and Social Psychology, 67, 1063–1078.
(in press).
(Modellbasierte Methoden der Reliabilitätsschätzung [Model-based methods of reliability estimation] In H. MoosbruggerA. KelavaEds., Testtheorie und Fragebogenkonstruktion[Test theory and scale construction] , Heidelberg, Germany: Springer.2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online (pp. 23–74). 8. https://doi.org/10.1.1.509.4258
(1999). Self-Regulation Scale. Retrieved from http://www.fu-berlin.de/gesund/skalen
(1995). Self-regulation and academic procrastination. The Journal of Social Psychology, 135, 607–619. https://doi.org/10.1080/00224545.1995.9712234
(1984). Academic procrastination: Frequency and cognitive-behavioral correlates. Journal of Counseling Psychology, 31, 503. https://doi.org/10.1037/0022-0167.31.4.503
(2010). Core self-evaluations in Germany: Validation of a German measure and its relationships with career success. Applied Psychology, 59, 674–700. https://doi.org/10.1111/j.1464-0597.2010.00422.x
(2003). The influence of social support and problematic support on optimism and depression in chronic illness: A prospective study evaluating self-esteem as a mediator. Health Psychology, 22, 123. https://doi.org/10.1037/0278-6133.22.2.123
(1935). The vectors of mind: Multiple-factor analysis for the isolation of primary traits. https://doi.org/10.1037/10018-000. Chicago, IL: University of Chicago Press
(1947). Multiple factor analysis. Chicago, IL: University of Chicago Press.
(2019). The Self-Regulation Scale (SRS): Psychometric properties and normative values of the German general population. Manuscript submitted for publication.
(2013). Reversed item bias: An integrative model. Psychological Methods, 18, 320–334. https://doi.org/10.1037/a0032121
(2008). Assessing response styles across modes of data collection. Journal of the Academy of Marketing Science, 36, 409–422. https://doi.org/10.1007/s11747-007-0077-6
(1981). Evaluating circumplexity in personality data. Multivariate Behavioral Research, 16, 263–289.
(1940). Factor analysis to 1940 (Psychometric Monograph No. 3). Chicago, IL: University of Chicago Press.
(2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34, 806–838. https://doi.org/10.1177/0011000006288127
(2015). The Core Self-evaluation Scale: Psychometric properties of the German version in a representative sample. Journal of Personality Assessment, 97, 310–318. https://doi.org/10.1080/00223891.2014.989367
(2015). Measuring depression with CES-D in Chinese patients with type 2 diabetes: The validity and its comparison to PHQ-9. BMC Psychiatry, 15, 198. https://doi.org/10.1186/s12888-015-0580-0
(2005). Cronbach’s α, Revelle’s β, and McDonald’s ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123–133. https://doi.org/10.1007/s11336-003-0974-7
(