Skip to main content
Multistudy Report

From Bi-Dimensionality to Uni-Dimensionality in Self-Report Questionnaires

Applying the Random Intercept Factor Analysis Model to Six Psychological Tests

Published Online:https://doi.org/10.1027/1015-5759/a000583

Abstract. The common factor model – by far the most widely used model for factor analysis – assumes equal item intercepts across respondents. Due to idiosyncratic ways of understanding and answering items of a questionnaire, this assumption is often violated, leading to an underestimation of model fit. Maydeu-Olivares and Coffman (2006) suggested the introduction of a random intercept into the model to address this concern. The present study applies this method to six established instruments (measuring depression, procrastination, optimism, self-esteem, core self-evaluations, and self-regulation) with ambiguous factor structures, using data from representative general population samples. In testing and comparing three alternative factor models (one-factor model, two-factor model, and one-factor model with a random intercept) and analyzing differential correlational patterns with an external criterion, we empirically demonstrate the random intercept model’s merit, and clarify the factor structure for the above-mentioned questionnaires. In sum, we recommend the random intercept model for cases in which acquiescence is suspected to affect response behavior.

References

  • Achat, H., Kawachi, I., Spiro, A., DeMolles, D. A., & Sparrow, D. (2000). Optimism and depression as predictors of physical and mental health functioning: The Normative Aging Study. Annals of Behavioral Medicine, 22, 127–130. https://doi.org/10.1007/BF02895776 First citation in articleCrossrefGoogle Scholar

  • Alessandri, G., Vecchione, M., Eisenberg, N., & Łaguna, M. (2015). On the factor structure of the Rosenberg (1965) General Self-Esteem Scale. Psychological Assessment, 27, 621. https://doi.org/10.1037/pas0000073 First citation in articleCrossrefGoogle Scholar

  • Alessandri, G., Vecchione, M., Fagnani, C., Bentler, P. M., Barbaranelli, C., Medda, E., … Caprara, G. V. (2010). Much more than model fitting? Evidence for the heritability of method effect associated with positively worded items of the Life Orientation Test Revised. Structural Equation Modeling, 17, 642–653. https://doi.org/10.1080/10705511.2010.510064 First citation in articleCrossrefGoogle Scholar

  • Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137–143. https://doi.org/10.1007/S11336-008-9100-1 First citation in articleCrossrefGoogle Scholar

  • Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606. https://doi.org/10.1037/0033-2909.88.3.588 First citation in articleCrossrefGoogle Scholar

  • Beutel, M. E., Klein, E. M., Aufenanger, S., Brähler, E., Dreier, M., Müller, K. W., … Wölfling, K. (2016). Procrastination, Distress and Life Satisfaction across the Age Range – A German Representative Community Study. PLoS One, 11, e0148054. https://doi.org/10.1371/journal.pone.0148054 First citation in articleCrossrefGoogle Scholar

  • Billiet, J. B., & McClendon, M. J. (2000). Modeling acquiescence in measurement models for two balanced sets of items. Structural Equation Modeling, 7(4), 608–628. First citation in articleCrossrefGoogle Scholar

  • Bollen, K. A. (1980). Issues in the comparative measurement of political democracy. American Sociological Review, 45, 370–390. First citation in articleCrossrefGoogle Scholar

  • Bollen, K. A. (1989). Structural equations with latent variables. New York, NY: Wiley. First citation in articleCrossrefGoogle Scholar

  • Campo-Arias, A., Díaz-Martínez, L. A., Rueda-Jaimes, G. E., Cadena-Afanador, L. D. P., & Hernández, N. L. (2007). Psychometric properties of the CES-D scale among Colombian adults from the general population. Revista Colombiana de Psiquiatría, 36, 664–674. First citation in articleGoogle Scholar

  • Federal Statistical Office of Germany. (2019). Bevölkerung [Population]. Retrieved from https://www.destatis.de/DE/Themen/Gesellschaft-Umwelt/Bevoelkerung/_inhalt.html First citation in articleGoogle Scholar

  • Ferring, D., & Filipp, S. H. (1996). Messung des Selbstwertgefühls: Befunde zu Reliabilität, Validität und Stabilität der Rosenberg-Skala [Measurement of self-esteem: Findings on reliability, validity, and stability of the Rosenberg Scale]. Diagnostica, 42, 284–292. First citation in articleGoogle Scholar

  • Fischer, R., & Milfont, T. L. (2010). Standardization in psychological research. International Journal of Psychological Research, 3, 88–96. https://doi.org/10.21500/20112084.852 First citation in articleCrossrefGoogle Scholar

  • Glaesmer, H., Hoyer, J., Klotsche, J., & Herzberg, P. Y. (2008). The German version of the Life-Orientation-Test (LOT-R) for dispositional optimism and pessimism. Zeitschrift für Gesundheitspsychologie, 16, 26–31. https://doi.org/10.1026/0943-8149.16.1.26 First citation in articleLinkGoogle Scholar

  • Glaesmer, H., Rief, W., Martin, A., Mewes, R., Brähler, E., Zenger, M., & Hinz, A. (2012). Psychometric properties and population-based norms of the Life Orientation Test Revised (LOT-R). British Journal of Health Psychology, 17, 432–445. https://doi.org/10.1111/j.2044-8287.2011.02046.x First citation in articleCrossrefGoogle Scholar

  • Greenberger, E., Chen, C., Dmitrieva, J., & Farruggia, S. P. (2003). Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: Do they matter? Personality and Individual Differences, 35, 1241–1254. https://doi.org/10.1016/S0191-8869(02)00331-8 First citation in articleCrossrefGoogle Scholar

  • Hautzinger, M., Bailer, M., Hofmeister, D., & Keller, F. (2012). Allgemeine Depressionsskala (ADS). Manual [General Depression Scale (ADS). Manual]. Göttingen, Germany: Hogrefe. First citation in articleGoogle Scholar

  • Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Bühner, M. (2011). Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16, 319–336. https://doi.org/10.1037/a0024917 First citation in articleCrossrefGoogle Scholar

  • Hentrich, S., Zimber, A., Sosnowsky-Waschek, N., Gregersen, S., & Petermann, F. (2017). The role of core self-evaluations in explaining depression and work engagement among managers. Current Psychology, 36, 516–529. https://doi.org/10.1007/s12144-016-9439-x First citation in articleCrossrefGoogle Scholar

  • Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74, 167. https://doi.org/10.1037/h0029780 First citation in articleCrossrefGoogle Scholar

  • Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2003). The core self-evaluation scale: Development of a measure. Personnel Psychology, 56, 303–331. https://doi.org/10.1111/j.1744-6570.2003.tb00152.x First citation in articleCrossrefGoogle Scholar

  • Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151. First citation in articleCrossrefGoogle Scholar

  • Kim, G., DeCoster, J., Huang, C. H., & Chiriboga, D. A. (2011). Race/ethnicity and the factor structure of the Center for Epidemiologic Studies Depression Scale: A meta-analysis. Cultural Diversity and Ethnic Minority Psychology, 17, 381. https://doi.org/10.1037/a0025434 First citation in articleCrossrefGoogle Scholar

  • Klein, E. M., Beutel, M. E., Müller, K. W., Wölfling, K., Brähler, E., & Zenger, M. (2019). Assessing Procrastination: Dimensionality and Measurement Invariance of the General Procrastination Scale–Screening (GPS-S) in a Representative Sample. European Journal of Psychological Assessment, 35, 633–640. https://doi.org/10.1027/1015-5759/a000441 First citation in articleLinkGoogle Scholar

  • Klingsieck, K. B., & Fries, S. (2012). Allgemeine Prokrastination: Entwicklung und Validierung einer deutschsprachigen Kurzskala der General Procrastination Scale (Lay, 1986) [Development and validation of the German Short Scale of the General Procrastination Scale (Lay, 1986)]. Diagnostica, 58, 182–193. https://doi.org/10.1026/0012-1924/a000060 First citation in articleLinkGoogle Scholar

  • Lay, C. H. (1986). At last, my research article on procrastination. Journal of Research in Personality, 20, 474–495. https://doi.org/10.1016/0092-6566(86)90127-3 First citation in articleCrossrefGoogle Scholar

  • Li, C.-H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavioral Research, 48, 936–949. https://doi.org/10.3758/s13428-015-0619-7 First citation in articleCrossrefGoogle Scholar

  • Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3, 635–694. https://doi.org/10.2466/pr0.1957.3.3.635 First citation in articleCrossrefGoogle Scholar

  • Löwe, B., Kroenke, K., & Gräfe, K. (2005). Detecting and monitoring depression with a two-item questionnaire (PHQ-2). Journal of Psychosomatic Research, 58, 163–171. https://doi.org/10.1016/j.jpsychores.2004.09.006 First citation in articleCrossrefGoogle Scholar

  • Marsh, H. W. (1986). Negative item bias in ratings scales for preadolescent children: A cognitive-developmental phenomenon. Developmental Psychology, 22, 37–49. https://doi.org/10.1037/0012-1649.22.1.37 First citation in articleCrossrefGoogle Scholar

  • Marsh, H. W. (1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70, 810–819. https://doi.org/10.1037/0022-3514.70.4.810 First citation in articleCrossrefGoogle Scholar

  • Maydeu-Olivares, A., & Coffman, D. L. (2006). Random intercept item factor analysis. Psychological Methods, 11, 344–362. https://doi.org/10.1037/1082-989X.11.4.344 First citation in articleCrossrefGoogle Scholar

  • McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum. First citation in articleGoogle Scholar

  • McPherson, J., & Mohr, P. (2005). The role of item extremity in the emergence of keying-related factors: An exploration with the Life Orientation Test. Psychological Methods, 10, 120. https://doi.org/10.1037/1082-989X.10.1.120 First citation in articleCrossrefGoogle Scholar

  • Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543. https://doi.org/10.1007/BF02294825 First citation in articleCrossrefGoogle Scholar

  • Michaelides, M. P., Christodoulou, A., Kkeli, N., Karekla, M., & Panayiotou, G. (2016). Factorial Structure of the Perceived Stress Scale and implications for scoring. European Review of Applied Psychology, 66, 309–316. https://doi.org/10.1016/j.erap.2016.07.002 First citation in articleCrossrefGoogle Scholar

  • Michaelides, M. P., Koutsogiorgi, C., & Panayiotou, G. (2016). Method effects on an adaptation of the Rosenberg Self-Esteem Scale in Greek and the role of personality traits. Journal of Personality Assessment, 98, 178–188. https://doi.org/10.1080/00223891.2015.1089248 First citation in articleCrossrefGoogle Scholar

  • Michaelides, M. P., Koutsogiorgi, C., & Panayiotou, G. (2017). Method/group factors: Inconsequential but meaningful. A comment on Donnellan, Ackerman and Brecheen (2016). Journal of Personality Assessment, 99, 334–335. First citation in articleCrossrefGoogle Scholar

  • Michaelides, M. P., Zenger, M., Koutsogiorgi, C., Brähler, E., Stöbel-Richter, Y., & Berth, H. (2016). Personality correlates and gender invariance of wording effects in the German version of the Rosenberg Self-Esteem Scale. Personality and Individual Differences, 97, 13–18. https://doi.org/10.1016/j.paid.2016.03.011 First citation in articleCrossrefGoogle Scholar

  • Milette, K., Hudson, M., Baron, M., Thombs, B. D. & Canadian Scleroderma Research Group. (2010). Comparison of the PHQ-9 and CES-D depression scales in systemic sclerosis: Internal consistency reliability, convergent validity and clinical correlates. Rheumatology, 49, 789–796. https://doi.org/10.1093/rheumatology/kep443 First citation in articleCrossrefGoogle Scholar

  • Miller, T. Q., Markides, K. S., & Black, S. A. (1997). The factor structure of the CES-D in two surveys of elderly Mexican Americans. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 52, 259–269. https://doi.org/10.1093/geronb/52B.5.S259 First citation in articleCrossrefGoogle Scholar

  • Murray, A. L., & Johnson, W. (2013). The limitations of model fit in comparing the bi-factor versus higher-order models of human cognitive ability structure. Intelligence, 41, 407–422. https://doi.org/10.1016/j.intell.2013.06.004 First citation in articleCrossrefGoogle Scholar

  • Muthén, B., & Asparouhov, T. (2018). Recent methods for the study of measurement invariance with many groups: Alignment and random effects. Sociological Methods & Research, 47, 637–664. https://doi.org/10.1177/0049124117701488 First citation in articleCrossrefGoogle Scholar

  • Nunnally, J. M. (1978). Psychometric theory. New York, NY: McGraw-Hill. First citation in articleGoogle Scholar

  • Perreira, K. M., Deeb-Sossa, N., Harris, K. M., & Bollen, K. (2005). What are we measuring? An evaluation of the CES-D across race/ethnicity and immigrant generation. Social Forces, 83, 1567–1601. https://doi.org/10.1353/sof.2005.0077 First citation in articleCrossrefGoogle Scholar

  • Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879. https://doi.org/10.1037/0021-9010.88.5.879 First citation in articleCrossrefGoogle Scholar

  • Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569. https://doi.org/10.1146/annurev-psych-120710-100452 First citation in articleCrossrefGoogle Scholar

  • R Core Team. (2018). R: A language and environment for statistical computing [Computer software] Retrieved from https://www.R-project.org/. Vienna, Austria: R Foundation for Statistical Computing. First citation in articleGoogle Scholar

  • Radloff, L. S. (1977). The CES-D Scale: A Self-Report Depression Scale for research in the general population. Applied Psychological Measurement, 1, 385–401. https://doi.org/10.1177/014662167700100306 First citation in articleCrossrefGoogle Scholar

  • Rauch, W. A., Schweizer, K., & Moosbrugger, H. (2007). Method effects due to social desirability as a parsimonious explanation of the deviation from unidimensionality in LOT-R scores. Personality and Individual Differences, 42, 1597–1607. https://doi.org/10.1016/j.paid.2006.10.035 First citation in articleCrossrefGoogle Scholar

  • Rdz-Navarro, K. (2019). Latent variables should remain as such: Evidence from a Monte Carlo study. The Journal of General Psychology, 146, 417–442. https://doi.org/10.1080/00221309.2019.1596064 First citation in articleCrossrefGoogle Scholar

  • Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129–140. https://doi.org/10.1080/00223891.2012.725437 First citation in articleCrossrefGoogle Scholar

  • Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145–154. https://doi.org/10.1007/s11336-008-9102-z First citation in articleCrossrefGoogle Scholar

  • Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354–373. https://doi.org/10.1037/a0029315 First citation in articleCrossrefGoogle Scholar

  • Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137. https://doi.org/10.1037/met0000045 First citation in articleCrossrefGoogle Scholar

  • Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press. First citation in articleCrossrefGoogle Scholar

  • Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48, 1–36. Retrieved from http://www.jstatsoft.org/v48/i02/ First citation in articleCrossrefGoogle Scholar

  • Roth, M., Decker, O., Herzberg, P. Y., & Brähler, E. (2008). Dimensionality and norms of the Rosenberg Self-esteem Scale in a German general population sample. European Journal of Psychological Assessment, 24, 190–197. https://doi.org/10.1027/1015-5759.24.3.190 First citation in articleLinkGoogle Scholar

  • Savalei, V., & Falk, C. F. (2014). Recovering substantive factor loadings in the presence of acquiescence bias: A comparison of three approaches. Multivariate Behavioral Research, 49, 407–424. https://doi.org/10.1080/00273171.2014.931800 First citation in articleCrossrefGoogle Scholar

  • Scheier, M. F., Carver, C. S., & Bridges, M. (1994). Distinguishing optimism from neuroticism (and trait anxiety, selfmastery, and self-esteem): A re-evaluation of the Life Orientation Test. Journal of Personality and Social Psychology, 67, 1063–1078. First citation in articleCrossrefGoogle Scholar

  • Schermelleh-Engel, K., & Gäde, J. C. (in press). Modellbasierte Methoden der Reliabilitätsschätzung [Model-based methods of reliability estimation] In H. MoosbruggerA. KelavaEds., Testtheorie und Fragebogenkonstruktion [Test theory and scale construction], Heidelberg, Germany: Springer. First citation in articleGoogle Scholar

  • Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online (pp. 23–74). 8. https://doi.org/10.1.1.509.4258 First citation in articleGoogle Scholar

  • Schwarzer, R., Diehl, M., & Schmitz, G. S. (1999). Self-Regulation Scale. Retrieved from http://www.fu-berlin.de/gesund/skalen First citation in articleGoogle Scholar

  • Senecal, C., Koestner, R., & Vallerand, R. J. (1995). Self-regulation and academic procrastination. The Journal of Social Psychology, 135, 607–619. https://doi.org/10.1080/00224545.1995.9712234 First citation in articleCrossrefGoogle Scholar

  • Solomon, L. J., & Rothblum, E. D. (1984). Academic procrastination: Frequency and cognitive-behavioral correlates. Journal of Counseling Psychology, 31, 503. https://doi.org/10.1037/0022-0167.31.4.503 First citation in articleCrossrefGoogle Scholar

  • Stumpp, T., Muck, P. M., Hülsheger, U. R., Judge, T. A., & Maier, G. W. (2010). Core self-evaluations in Germany: Validation of a German measure and its relationships with career success. Applied Psychology, 59, 674–700. https://doi.org/10.1111/j.1464-0597.2010.00422.x First citation in articleCrossrefGoogle Scholar

  • Symister, P., & Friend, R. (2003). The influence of social support and problematic support on optimism and depression in chronic illness: A prospective study evaluating self-esteem as a mediator. Health Psychology, 22, 123. https://doi.org/10.1037/0278-6133.22.2.123 First citation in articleCrossrefGoogle Scholar

  • Thurstone, L. L. (1935). The vectors of mind: Multiple-factor analysis for the isolation of primary traits. https://doi.org/10.1037/10018-000. Chicago, IL: University of Chicago Press First citation in articleCrossrefGoogle Scholar

  • Thurstone, L. L. (1947). Multiple factor analysis. Chicago, IL: University of Chicago Press. First citation in articleGoogle Scholar

  • Tibubos, A. N., Zenger, M., Beutel, M. E., Schmalbach, B., Hinz, A., & Brähler, E. (2019). The Self-Regulation Scale (SRS): Psychometric properties and normative values of the German general population. Manuscript submitted for publication. First citation in articleGoogle Scholar

  • Weijters, B., Baumgartner, H., & Schillewaert, N. (2013). Reversed item bias: An integrative model. Psychological Methods, 18, 320–334. https://doi.org/10.1037/a0032121 First citation in articleCrossrefGoogle Scholar

  • Weijters, B., Schillewaert, N., & Geuens, M. (2008). Assessing response styles across modes of data collection. Journal of the Academy of Marketing Science, 36, 409–422. https://doi.org/10.1007/s11747-007-0077-6 First citation in articleCrossrefGoogle Scholar

  • Wiggins, J. S., Steiger, J. H., & Gaelick, L. (1981). Evaluating circumplexity in personality data. Multivariate Behavioral Research, 16, 263–289. First citation in articleGoogle Scholar

  • Wolfle, D. (1940). Factor analysis to 1940 (Psychometric Monograph No. 3). Chicago, IL: University of Chicago Press. First citation in articleGoogle Scholar

  • Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34, 806–838. https://doi.org/10.1177/0011000006288127 First citation in articleCrossrefGoogle Scholar

  • Zenger, M., Körner, A., Maier, G. W., Hinz, A., Stöbel-Richter, Y., Brähler, E., & Hilbert, A. (2015). The Core Self-evaluation Scale: Psychometric properties of the German version in a representative sample. Journal of Personality Assessment, 97, 310–318. https://doi.org/10.1080/00223891.2014.989367 First citation in articleCrossrefGoogle Scholar

  • Zhang, Y., Ting, R. Z., Lam, M. H., Lam, S. P., Yeung, R. O., Nan, H., … Sartorius, N. (2015). Measuring depression with CES-D in Chinese patients with type 2 diabetes: The validity and its comparison to PHQ-9. BMC Psychiatry, 15, 198. https://doi.org/10.1186/s12888-015-0580-0 First citation in articleCrossrefGoogle Scholar

  • Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s α, Revelle’s β, and McDonald’s ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123–133. https://doi.org/10.1007/s11336-003-0974-7 First citation in articleCrossrefGoogle Scholar