Change? What Change?
An Exploration of the Use of Mixed-Methods Research to Understand Longitudinal Measurement Variance
Abstract
A primary objective of panel studies is to analyze change. The same questionnaire is used to compare data recorded at various times. Panel designs assume that the meaning of the questions and the concept of interest are stable over time. Analyses of measurement invariance often show the contrary. A qualitative part supplementing a panel survey can help us understand this phenomenon. In this study, 261 first-year psychology students completed questionnaires about their study motivation on two occasions; we interviewed some students as well. The survey showed that study motivation is not invariant over time. The qualitative data converged with the quantitative outcomes and explained the lack of invariance by the students’ overall transition during the first study year. We conclude that mixing quantitative and qualitative research methods for panel studies helps us understand change in constructs over time. We can study change at the macrolevel and better understand such change at the microlevel.
References
. (2008). Standard definitions: final dispositions of case codes and outcome reports for surveys (5th ed.). Lenexa, KS: American Association for Public Opinion Research.
(2007). AMOS 16.0. Spring House, PA: AMOS Development Corporation.
(2010). Analysis in qualitative research. London, UK: Sage.
(2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6, 97–113.
(2007). Designing and conducting mixed methods research. London, UK: Sage.
(2009). The application of mixed methods design to trauma research. Journal of Traumatic Stress, 22, 612–621.
(1990). Masculinity, femininity, and sex role attitudes in early adolescence: Exploring gender intensification. Child Development, 61, 1905–1914.
(1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavorial Science, 12, 133–157.
(1984). The pictorial scale of perceived competence and social acceptance for young children. Child Development, 55, 1969–1982.
(2000). Motivating the academically unmotivated: A critical issue for the 21st century. Review of Educational Research, 70, 151–179.
(1989). Participant observation. A methodology for human studies. Newbury Park, CA: Sage.
(2005). Principles and practice of structural equation modeling (2nd ed.). New York, NY: The Guildford Press.
(2007). MaxQDA – professional software for qualitative data analysis. Berlin, Germany: Verbi.
(2009). Towards a flexible online mixed method design with a feedback loop. Quality and Quantity, 43, 585–597.
(2006). Mixing methods in a qualitative driven way. Qualitative Research, 6, 9–25.
(2009). Mixed method design: Principles and procedures. Walnut Creek, CA: Left Coast Press.
(2005). The psychometric paradox of practice effects due to retesting: Measurement invariance and stable ability estimates in the face of observed score change. Intelligence, 33, 535–549.
(1989). Perceived locus of causality and internalization: Examining reasons for acting in two domains. Journal of Personality and Social Psychology, 57, 749–761.
(2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54–67.
(2007). Design, evaluation and analysis of questionnaires for survey research. New York, NY: Wiley.
(2008). Measurement invariance: Review of practice and implications. Human Resource Management Review, 18, 210–222.
(1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78–107.
(1980). Evaluating planned organizational change: A method for assessing alpha, beta, and gamma change. Academy of Management Review, 5, 109–121.
(1992). Intrinsic, extrinsic and amotivational styles as predictors of behavior: A prospective study. Journal of Personality, 60, 599–620.
(2002). Toward a further understanding of and improvement in measurement invariance methods and procedures. Organizational Research Methods, 5, 139–158.
(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices and recommendations for organizational research. Organizational Research Methods, 3, 1–70.
(1993). Assessing newcomers’ changing commitments during the first 6 months of work. Journal of Applied Psychology, 78, 557–568.
(1992). The nature and origins of mass opinion. Cambridge: Cambridge University Press.
