Analyzing Observed Composite Differences Across Groups
Is Partial Measurement Invariance Enough?
Abstract
Although the use of structural equation modeling has increased during the last decades, the typical procedure to investigate mean differences across groups is still to create an observed composite score from several indicators and to compare the composite’s mean across the groups. Whereas the structural equation modeling literature has emphasized that a comparison of latent means presupposes equal factor loadings and indicator intercepts for most of the indicators (i.e., partial invariance), it is still unknown if partial invariance is sufficient when relying on observed composites. This Monte-Carlo study investigated whether one or two unequal factor loadings and indicator intercepts in a composite can lead to wrong conclusions regarding latent mean differences. Results show that unequal indicator intercepts substantially affect the composite mean difference and the probability of a significant composite difference. In contrast, unequal factor loadings demonstrate only small effects. It is concluded that analyses of composite differences are only warranted in conditions of full measurement invariance, and the author recommends the analyses of latent mean differences with structural equation modeling instead.
References
(1991). Multitrait-multimethod matrices in consumer research. Journal of Consumer Research, 17, 426–439.
(1998). Multi-group latent variable models for varying numbers of items and factors with cross-national and longitudinal applications. Marketing Letters, 9, 21–35.
(1989). Structural equations with latent variables. New York: Wiley.
(2006). The attack of the psychometricians. Psychometrika, 71, 425–440.
(2002). True scores, latent variables, and constructs: A comment on Schmidt and Hunter. Intelligence, 30, 505–514.
(2006). Confirmatory factor analysis for applied research. New York: Guilford.
(1993). The role of negative affectivity in understanding relations between self-reports of stressors and strains: A comment on the applied psychology literature. Journal of Applied Psychology, 78, 402–412.
(1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456–466.
(2000). Assessing extreme and acquiescence response sets in cross-cultural research using structural equation modeling. Journal of Cross-Cultural Psychology, 31, 187–212.
(1981). Bias in testing. American Psychologist, 36, 1067–1077.
(2009). The quality of the measurement of interest in the political issues. Paper presented at the QMSS2 meeting.
(2004). Test bias in a cognitive test: Differential item functioning in the CASI. Statistics in Medicine, 23, 241–256.
(2008). A cross-country and cross-time comparison of the human values measurements with the second round of the European Social Survey. Survey Research Methods, 2, 33–46.
(2008). Bringing values back in: A multiple group comparison with 20 countries using the European Social Survey. Public Opinion Quarterly, 72, 420–445.
(2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5, 155–174.
(2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241–261.
(1978). The many definitions of test bias. American Psychologist, 33, 671–679.
(2007). Cross-cultural organizational behavior. Annual Review of Psychology, 58, 479–514.
(1989). Structural equation modeling – essentials and advances. Baltimore, MA and London: The John Hopkins University Press.
(1995). Cognitive assessment in education in a multicultural society. European Journal of Psychological Assessment, 11, 158–169.
(1993). Differential item functioning. Hillsdale, NJ: Erlbaum.
(1985). Measurement in cross-cultural psychology. Journal of Cross-Cultural Psychology, 16, 131–152.
(1974). Racial and ethnic bias in psychological tests: Divergent implications of two definitions of test bias. American Psychologist, 29, 1–8.
(2005). The gender similarities hypothesis. American Psychologist, 60, 581–592.
(1980). Bias in mental testing. New York: Free Press.
(1971). Simultaneous factor analysis in several populations. Psychometrika, 36, 409–426.
(2002). Personality and leadership: A qualitative and quantitative review. Journal of Applied Psychology, 87, 765–780.
(2004). Intelligence and leadership: A quantitative review and test of theoretical propositions. Journal of Applied Psychology, 89, 542–552.
(1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165–172.
(2007). Scenario-based scales measuring cultural orientations of business owners. Journal of Evolutionary Economics, 17, 211–239.
(1997). Mean and covariance structures (MACS) analyses of cross-cultural data: Practical and theoretical issues. Multivariate Behavioral Research, 32, 53–76.
(2000). On the comparability of constructs in cross-cultural research. Journal of Cross-Cultural Psychology, 31, 213–219.
(2005). The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 90, 710–730.
(2006). Multidimensional self-concept structure for preadolescents with mild intellectual disabilities: A hybrid multigroup-MIMIC approach to factorial invariance and latent mean differences. Educational and Psychological Measurement, 66, 795–818.
(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543.
(2007). Mplus user’s guide. Los Angeles, CA: Muthén & Muthén.
(1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77–101.
(1983). Test item bias. Newbury Park, CA: Sage.
(2001). Monte Carlo experiments: Design and implementation. Structural Equation Modeling, 8, 287–312.
(2004). Applications of mean and covariance structure analysis: Integrating correlational and experimental approaches. Organizational Research Methods, 7, 27–65.
(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.
(1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, 114, 552–566.
(1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner?. Journal of Management, 20, 643–671.
(2007). Design, evaluation and analysis of questionnaires for survey research. New York: Wiley.
(2009). Testing structural equation models or detection of misspecifications?. Structural Equation Modeling, 16, 561–582.
(2003). A review of cross-cultural methodologies for organizational research: A best-practices approach. Organizational Research Methods, 6, 169–215.
(2005). Basic human values: Their content and structure across cultures. In , Valores e Comportamento nas Organizações
[Values and behavior in organizations] (pp. 21–55). Petrópolis, Brazil: Vozes.(2004). Acquiescent response bias as an aspect of cultural communication style. Journal of Cross-Cultural Psychology, 35, 50–61.
(1987). Method variance as an artifact in self-reported affect and perceptions at work: Myth or significant problem? Journal of Applied Psychology, 72, 438–443.
(2003). Measurement equivalence of 10 value types from the Schwartz value survey across 21 countries. Journal of Cross-Cultural Psychology, 34, 3–23.
(2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91, 1292–1306.
(1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78–90.
(2010). Estimation and comparison of latent means across cultures. In , Cross-cultural analysis: Methods and applications. (pp. 85–116). Oxford: Routledge.
(2009). Testing measurement invariance using multigroup CFA: Differences between educational groups in human values measurement. Quality and Quantity, 43, 599–616.
(2006). Overview of quantitative measurement methods: Equivalence, invariance, and differential item functioning in health applications. Medical Care, 44, 39–49.
(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–69.
