Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

Evaluating the Construct Validity of Objective Personality Tests Using a Multitrait-Multimethod-Multioccasion-(MTMM-MO)-Approach

Published Online:January 01, 2014https://doi.org/10.1027/1015-5759/a000212

Abstract

Although Objective Personality Tests (OPTs) have a long history in psychology and the field of psychological assessment, their validity, and reliability have not yet been sufficiently studied. In this study, we examined the convergent and discriminant validity of objective (personality) tests, Implicit Association Tests (IATs), and self-report measures for the assessment of conscientiousness and intelligence. Moreover, the convergent and discriminant validity of these measures was assessed on the trait (stable) and occasion specific (momentary) level by using the multimethod latent state-trait (MM-LST) model proposed by Courvoisier, Nussbeck, Eid, and Cole (2008) which allows for the decomposition of different sources of variance. Data from 367 students assessed on three different measurement occasions was incorporated. Results indicate generally low convergence of OPTs with data gained by other approaches. Additional analyses revealed that the OPTs used assess stable rather than momentary components of the constructs. Reliabilities of different tests ranged from .54 to .95. Furthermore, a substantial amount of trait method specificity revealed that different methods assess trait components that are not shared between OPTs and other measures. Data on the criterion validity of the objective conscientiousness test revealed that it is related to the punctuality of test takers in the laboratory.

References

  • Allen, M. P. (1974). Construction of composite measures by the canonical factor regression method. In H. L. Costner, (Ed.), Sociological methodology 1973–1974 (pp. 51–78). San Francisco, CA: Jossey-Bass. First citation in articleGoogle Scholar

  • Amthauer, R. , Brocke, B. , Liepmann, D. , Beauducel, A. (2001). Intelligenz-Struktur-Test 2000 R [Test of intelligence structure 2000 R]. Göttingen, Germany: Hogrefe. First citation in articleGoogle Scholar

  • Back, M. D. , Krause, S. , Hirschmuller, S. , Stopfer, J. M. , Egloff, B. , & Schmukle, S. C. (2009). Unraveling the three faces of self-esteem: A new information-processing sociometer perspective. Journal of Research in Personality, 43, 933–937. doi: 10.1016/j.jrp.2009.04.002 First citation in articleCrossref, Google Scholar

  • Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. First citation in articleCrossref, Google Scholar

  • Bentler, P. M. (1995). EQS structural equations program manual. Encino, CA: Multivariate Software. First citation in articleGoogle Scholar

  • Bentler, P. , Chou, C.-P. (1987). Practical issues in structural modeling. Sociological Methods & Research, 17, 78–117. First citation in articleCrossref, Google Scholar

  • Bosson, J. K. , Brown, R. P. , Zeigler-Hill, V. , Swann, W. B. Jr. (2003). Self-enhancement tendencies among people with high self-esteem: The moderating role of implicit self-esteem. Self and Identity, 2, 169–187. doi: 10.1080/15298860309029 First citation in articleCrossref, Google Scholar

  • Brand, M. , Fujiwara, E. , Borsutzky, S. , Kalbe, E. , Kessler, J. , & Markowitsch, H. J. (2005). Decision-making deficits of Korsakoff patients in a new gambling task with explicit rules: Associations with executive functions. Neuropsychology, 19, 267–277. First citation in articleCrossref, Google Scholar

  • Brand, M. , Heinze, K. , Labudda, K. , Markowitsch, H. J. (2008). The role of strategies in deciding advantageously in ambiguous and risky situations. Cognitive Processing, 9, 159–173. First citation in articleCrossref, Google Scholar

  • Brickenkamp, R. (2002). Test d2 Aufmerksamkeits-Belastungs-Test. Manual. 9., überarbeitete und neu normierte Auflage [The d2 Test of Attention. Manual]. (9th ed.). Göttingen, Germany: Hogrefe. First citation in articleGoogle Scholar

  • Campbell, D. T. , Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. First citation in articleCrossref, Google Scholar

  • Carroll, J. B. (1993). Human cognitive abilities. New York, NY: Cambridge University Press. First citation in articleCrossref, Google Scholar

  • Cattell, J. M. (1890). Mental tests and measurements. Mind, 15, 373–381. First citation in articleCrossref, Google Scholar

  • Cattell, R. B. (1941). An objective test of character-temperatment: I. Journal of General Psychology, 25, 59–73. First citation in articleCrossref, Google Scholar

  • Cattell, R. B. (1944). An objective test of character-temperatment: II. Journal of Social Psychology, 19, 99–114. First citation in articleCrossref, Google Scholar

  • Cattell, R. B. (1946). Description and measurement of personality. New York, NY: World Book. First citation in articleGoogle Scholar

  • Cattell, R. B. (1957). Personality and motivation structure and measurement. New York, NY: World Book. First citation in articleGoogle Scholar

  • Cattell, R. B. (1968). What is “objective” in “objective personality tests”? In W. L. Barnette, (Ed.), Readings in psychological tests and measurements (pp. 163–168). Homewood, IL: Dorsey Press. First citation in articleGoogle Scholar

  • Cattell, R. B. , Kline, P. (1977). The scientific analysis of personality and motivation. London, UK: Academic Press. First citation in articleGoogle Scholar

  • Cattell, R. B. , Warburton, F. W. (1967). Objective personality and motivation tests: A theoretical introduction and practical compendium. Chicago, IL: University of Illinois Press. First citation in articleGoogle Scholar

  • Costa, P. T. , McCrae, R. R. (1992). Neo PI-R professional manual. Odessa, FL: Psychological Assessment Resources. First citation in articleGoogle Scholar

  • Courvoisier, D. S. . (2006). Unfolding the constituents of psychological scores: Development and application of mixture and multitrait-multimethod LST models (Doctoral dissertation), University of Geneva, Switzerland. Retrieved from doc.rero.ch/record/9221/files/these.pdf First citation in articleGoogle Scholar

  • Courvoisier, D. S. , Nussbeck, F. W. , Eid, M. , Cole, D. A. (2008). Analyzing the convergent and discriminant validity of states and traits: Development and applications of multimethod latent state-trait models. Psychological Assessment, 20, 270–280. doi: 10.1037/a0012812 First citation in articleCrossref, Google Scholar

  • Cronbach, L. J. , Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302. First citation in articleCrossref, Google Scholar

  • Dislich, F. X. R. , Imhoff, R. , Banse, R. , Altstötter-Gleich, C. , Zinkernagel, A. , Schmitt, M. (2012). Discrepancies between implicit and explicit self-concepts of intelligence predict performance on tests of intelligence. European Journal of Personality, 26, 212–220. First citation in articleCrossref, Google Scholar

  • Dislich, F. X. R. , Zinkernagel, A. , Ortner, T. M. , Schmitt, M. (2010). Convergence of direct, indirect, and objective risk taking measures in the domain of gambling: The moderating role of impulsiveness and self-control. Journal of Psychology, 218, 20–27. First citation in articleAbstract, Google Scholar

  • Egloff, B. , Schwerdtfeger, A. , Schmukle, S. C. (2005). Temporal stability of the Implicit Association Test-Anxiety. Journal of Personality Assessment, 84, 82–88. First citation in articleCrossref, Google Scholar

  • Eid, M. (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241–261. First citation in articleCrossref, Google Scholar

  • Eid, M. , Lischetzke, T. , Nussbeck, F. W. , Trierweiler, L. I. (2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38–60. First citation in articleCrossref, Google Scholar

  • Eid, M. , Nussbeck, F. W. , Geiser, C. , Cole, D. A. , Gollwitzer, M. , Lischetzke, T. (2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods. Psychological Methods, 13, 230–253. First citation in articleCrossref, Google Scholar

  • Elliot, S. , Lawty-Jones, M. , Jackson, C. (1996). Effects of dissimulation on self-report and objective measures of personality. Personality and Individual Differences, 21, 335–343. First citation in articleCrossref, Google Scholar

  • Fazio, R. H. , Olson, M. A. (2003). Implicit measures in social cognition research: Their meaning and uses. Annual Review of Psychology, 54, 297–327. First citation in articleCrossref, Google Scholar

  • Finney, S. J. , DiStefano, C. (2006). Non-normal and categorical data in structural equation modeling. In G. R. Hancock, R. O. Mueller, (Eds.), Structural equation modeling: A second course (pp. 269–314). Greenwich, CT: Information Age Publishing. First citation in articleGoogle Scholar

  • Friese, M. , Hofmann, W. , Schmitt, M. (2008). When and why do implicit measures predict behaviour? Empirical evidence for the moderating role of opportunity, motivation, and process reliance. European Review of Social Psychology, 19, 285–338. doi: 10.1080/10463280802556958 First citation in articleCrossref, Google Scholar

  • Geiser, C. , Eid, M. , West, S. G. , Lischetzke, T. , Nussbeck, F. W. (2012). A comparison of method effects in two confirmatory factor models for structurally different methods. Structural Equation Modeling, 19, 409–436. doi: 10.1080/10705511.2012.687658 First citation in articleCrossref, Google Scholar

  • Geiser, C. , Keller, B. , Lockhart, G. , Eid, M. , Cole, D. , Koch, T. (2014). Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-trait analyses. Behaviour Research Methods. doi: 10.3758/s13428-014-0457-z First citation in articleGoogle Scholar

  • Greenwald, A. G. , McGhee, D. E. , Schwartz, J. K. L. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464–1480. First citation in articleCrossref, Google Scholar

  • Greenwald, A. G. , Nosek, B. A. , Banaji, M. R. (2003). Understanding and using the Implicit Association Test I: An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197–216. First citation in articleCrossref, Google Scholar

  • Greenwald, A. G. , Poehlman, T. A. , Uhlmann, E. L. , Banaji, M. R. (2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17–41. doi: 10.1037/a0015575 First citation in articleCrossref, Google Scholar

  • Gschwendner, T. , Hofmann, W. , Schmitt, M. (2008). Differential stability: The effects of acute and chronic construct accessibility on the temporal stability of the Implicit Association Test. Journal of Individual Differences, 29, 70–79. doi: 10.1027/1614-0001.29.2.70 First citation in articleLink, Google Scholar

  • Häcker, H. , Schmidt, L. R. , Schwenkmezger, P. , Utz, H. E. (1975). Objektive Testbatterie, OA-TB 75 [Objective Test Battery, OA-TB 75]. Weinheim, Germany: Beltz. First citation in articleGoogle Scholar

  • Häcker, H. , Schwenkmezger, P. , Utz, H. (1979). Über die Verfälschbarkeit von Persönlichkeitsfragebogen und Objektiven Persönlichkeitstests unter SD-Instruktion und in einer Auslesesituation [About fakeability of personality questionnaires under SD-instruction and in a situation of selection]. Diagnostica, 25, 7–23. First citation in articleGoogle Scholar

  • Hofmann, W. , Gawronski, B. , Gschwendner, T. , Le, H. , Schmitt, M. (2005). A meta-analysis on the correlation between the implicit association test and explicit self-report measures. Personality and Social Psychology Bulletin, 31, 1369–1385. doi: 10.1177/0146167205275613 First citation in articleCrossref, Google Scholar

  • Hofmann, W. , Gschwendner, T. , Castelli, L. , Schmitt, M. (2008). Implicit and explicit attitudes and interracial interaction: The moderating role of situationally available control resources. Group Processes & Intergroup Relations, 11, 69–87. First citation in articleCrossref, Google Scholar

  • Hofmann, W. , Gschwendner, T. , Wiers, R. , Friese, M. , Schmitt, M. (2008). Working memory capacity and self-regulatory behavior: Towards an individual differences perspective on behavior determination by automatic versus controlled processes. Journal of Personality and Social Psychology, 95, 962–977. First citation in articleCrossref, Google Scholar

  • Hornke, L. F. , Küppers, A. , Etzel, S. (2000). Design and evaluation of an adaptive matrices test. Diagnostica, 46, 182–188. First citation in articleLink, Google Scholar

  • Hu, L.-T. , & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424–453. doi: 10.1037/1082-989x.3.4.424 First citation in articleCrossref, Google Scholar

  • Hu, L.-T. , & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi: 10.1080/10705519909540118 First citation in articleCrossref, Google Scholar

  • Hundleby, J. D. , Pawlik, K. , Cattell, R. B. (1965). Personality factors in objective test devices. San Diego, CA: Knapp. First citation in articleGoogle Scholar

  • Karpinski, A. , Steinman, R. B. (2006). The single category implicit association test as a measure of implicit social cognition. Journal of Personality and Social Psychology, 91, 16–32. First citation in articleCrossref, Google Scholar

  • Kubinger, K. D. , Ebenhöh, J. (1996). Arbeitshaltungen – Kurze Testbatterie: Anspruchsniveau, Frustrationstoleranz, Leistungsmotivation, Impulsivität/Reflexivität [Working Style – A short test-battery: Level of aspiration, achievement motivation, frustration tolerance, impulsiveness/reflexiveness]. Frankfurt/M.: Swets Test Services [Software and Manual] First citation in articleGoogle Scholar

  • Lejuez, C. W. , Aklin, W. M. , Jones, H. A. , Richards, J. B. , Strong, D. R. , Kahler, C. W. , Read, J. P. (2003). The balloon analogue risk task (BART) differentiates smokers and nonsmokers. Experimental and Clinical Psychopharmacology, 11, 26–33. First citation in articleCrossref, Google Scholar

  • Lejuez, C. W. , Read, J. P. , Kahler, C. W. , Richards, J. B. , Ramsey, S. E. , Stuart, G. L. , … Brown, R. A. (2002). Evaluation of a behavioral measure of risk taking: The Balloon Analogue Risk Task (BART). Journal of Experimental Psychology-Applied, 8, 75–84. doi: 10.1037/1076-898x.8.2.75 First citation in articleCrossref, Google Scholar

  • Lischetzke, T. , Eid, M. , Nussbeck, F. W. (2002, September). Unterschiedliche Definitionen von Methodeneffekten in MTMM Modellen und ihre Implikationen für die Analyse der Validität [Different definitions of method effects in MTMM models and their implications for validity analysis]. Paper presented at the 43rd meeting of the German Psychological Association, Berlin, Germany First citation in articleGoogle Scholar

  • Little, T. D. , Cunningham, W. A. , Shahar, G. , & Widaman, K. F. (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9, 151–173. doi: 10.1207/s15328007sem0902_1 First citation in articleCrossref, Google Scholar

  • Matthies, S. , Philipsen, A. , Svaldi, J. (2012). Risky decision making in adults with ADHD. Journal of Behavior Therapy and Experimental Psychiatry, 43, 938–946. First citation in articleCrossref, Google Scholar

  • Muthén, L. K. , Muthén, B. O. (1998–2012). Mplus user’s guide (3rd eds.). Los Angeles, CA: Muthén & Muthén. First citation in articleGoogle Scholar

  • Nussbeck, F. W. , Eid, M. , Lischetzke, T. (2006). Analysing multitrait-multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: What sample size is needed for valid results? British Journal of Mathematical & Statistical Psychology, 59, 195–213. doi: 10.1348/00071100sx67490 First citation in articleCrossref, Google Scholar

  • Ortner, T. M. , Horn, R. , Kersting, M. , Krumm, S. , Kubinger, K. D. , Proyer, R. T. , … Westhoff, K. (2007). Standortbestimmung und Zukunft Objektiver Persönlichkeitstests. [Current situation and future of objective personality tests] Report Psychologie, 32, 64–75. First citation in articleGoogle Scholar

  • Ortner, T. M. , Kubinger, K. D. , Schrott, A. , Radinger, R. , Litzenberger, M. (2006). Belastbarkeits-Assessment: Computerisierte Objektive Persönlichkeits-Testbatterie – Deutsch (BAcO-D). [Stress resistance assessment: Computerized Objective Test-battery – German version] Frankfurt/M.: Harcourt Assessment [Test: Software und Manual] First citation in articleGoogle Scholar

  • Ortner, T. M. , Proyer, R. T. (2014). Objective Personality Tests. Manuscript submitted for publication First citation in articleGoogle Scholar

  • Ostendorf, F. (1994). Zur Taxonomie deutscher Dispositionsbegriffe. In W. Hager, M. Hasselhorn, (Eds.) [A taxonomy of German dispositional concepts] Handbuch deutschsprachiger Wortnormen [Handbook of German word norms]. (pp. 382–441). Göttingen, Germany: Hogrefe. First citation in articleGoogle Scholar

  • Ostendorf, F. , Angleitner, A. (2003). NEO-Persönlichkeitsinventar nach Costa und McCrae, Revidierte Fassung (NEO-PI-R) [NEO-Personality Inventory revisited by Costa and McCrae]. Göttingen, Germany: Hogrefe. First citation in articleGoogle Scholar

  • Pohl, S. , & Steyer, R. (2010). Modeling common traits and method effects in multitrait-multimethod analysis. Multivariate Behavioral Research, 45, 45–72. First citation in articleCrossref, Google Scholar

  • Pohl, S. , Steyer, R. , & Kraus, K. (2008). Modelling method effects as individual causal effects. Journal of the Royal Statistical Society Series A, 171, 41–63. First citation in articleGoogle Scholar

  • Proyer, R. T. (2007). Convergence of conventional and behavior-based measures: Towards a multimethod approach in the assessment of vocational interests. Psychology Science Quarterly, 49, 168–183. First citation in articleGoogle Scholar

  • Proyer, R. T. , Häusler, J. (2007). Assessing behavior in standardized settings: The role of objective personality tests. International Journal of Clinical and Health Psychology, 7, 537–546. First citation in articleGoogle Scholar

  • R Development Core Team . (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing Retrieved from www.R-project.org First citation in articleGoogle Scholar

  • Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Nielsen & Lydiche. First citation in articleGoogle Scholar

  • Rothermund, K. , Wentura, D. (2004). Underlying processes in the Implicit Association Test: Dissociating salience from associations. Journal of Experimental Psychology: General, 133, 139–165. First citation in articleCrossref, Google Scholar

  • Rubio, V. J. , Hernández, J. M. , Zaldivar, F. , Marquez, O. , Santacreu, J. (2010). Can we predict risk-taking behavior? Two behavioral tests for predicting guessing tendencies in a multiple-choice test. European Journal of Psychological Assessment, 26, 87–94. First citation in articleLink, Google Scholar

  • Santacreu, J. , Rubio, V. J. , Hernández, J. M. (2006). The objective assessment of personality: Cattells’s T-data revisited and more. Psychology Science, 48, 53–68. First citation in articleGoogle Scholar

  • Schermelleh-Engel, K. , Moosbrugger, H. , Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research-Online, 8, 23–74. doi: citeulike-article-id:7222182 First citation in articleGoogle Scholar

  • Schmidt, L. R. (1975). Objektive Persönlichkeitsmessung in diagnostischer und klinischer Psychologie [Objective personality measurement in psychological assessment and clinical psychology]. Weinheim, Germany: Beltz. First citation in articleGoogle Scholar

  • Schmidt-Atzert, L. (2004). Objektiver Leistungsmotivations Test (OLMT) [Objective Achievement Motivation Test]. Mödling, Austria: Dr. G. Schuhfried [Software and Manual] First citation in articleGoogle Scholar

  • Schmidt-Atzert, L. (2007). Objektiver Leistungsmotivations Test (OLMT) [Objective Achievement Motivation Test] Mödling, Austria: Dr. G. Schuhfried [Software and Manual] First citation in articleGoogle Scholar

  • Schmitt, M. (2006). Conceptual, theoretical, and historical foundations of multimethod assessment. In M. Eid, E. Diener, (Eds.), Handbook of multimethod measurement in psychology (pp. 9–25). New York, NY: American Psychological Association. First citation in articleCrossref, Google Scholar

  • Schmitt, M. , Hofmann, W. , Gschwendner, T. , Gerstenberg, F. X. , Zinkernagel, A. (in press). A model of moderated convergence between direct, indirect, and behavioral measures of personality traits. In A. J. R. Van de Vijver, T. M. Ortner, (Eds.), Behavior based assessment: Going beyond self report in the personality, affective, motivation, and social domains. Göttingen, Germany: Hogrefe. First citation in articleGoogle Scholar

  • Schmukle, S. C. , Back, M. D. , Egloff, B. (2008). Validity of the five-factor model for the implicit self-concept of personality. European Journal of Psychological Assessment, 24, 263–272. doi: 10.1027/1015-5759.24.4.263 First citation in articleLink, Google Scholar

  • Schmukle, S. C. , Egloff, B. (2005). A latent state-trait analysis of implicit and explicit personality measures. European Journal of Psychological Assessment, 21, 100–107. doi: 10.1027/1015-5759.21.2.100 First citation in articleLink, Google Scholar

  • Skinner, N. S. F. , Howarth, E. (1973). Cross-media independence of questionnaire and objective test personality factors. Multivariate Behavioral Research, 8, 23–40. First citation in articleCrossref, Google Scholar

  • Steffens, M. C. , König, S. S. (2006). Predicting spontaneous big five behavior with implicit association tests. European Journal of Psychological Assessment, 22, 13–20. doi: 10.1027/1015-5759.22.1.13 First citation in articleLink, Google Scholar

  • Steiger, J. H. (1990). Structural model evaluation and modification - An interval estimation approach. Multivariate Behavioral Research, 25, 173–180. doi: 10.1207/s15327906mbr2502_4 First citation in articleCrossref, Google Scholar

  • Steyer, R. , Schmitt, M. , Eid, M. (1999). Latent state-trait theory and research in personality and individual differences. European Journal of Personality, 13, 389–408. First citation in articleCrossref, Google Scholar

  • Strack, F. , & Deutsch, R. (2004). Reflective and impulsive determinants of social behavior. Personality and Social Psychology Review, 8, 220–247. First citation in articleCrossref, Google Scholar

  • Wilson, T. D. , Lindsey, S. , Schooler, T. Y. (2000). A model of dual attitudes. Psychological Review, 107, 101–126. First citation in articleCrossref, Google Scholar

  • Ziegler, M. , Schmidt-Atzert, L. , Bühner, M. , Krumm, S. (2007). Faking susceptibility of different measurement methods: Questionnaire, semi-projective, and objective. Psychology Science, 49, 291–307. First citation in articleGoogle Scholar

  • Ziegler, M. , Schmukle, S. C. , Egloff, B. , & Bühner, M. (2010). Investigating Measures of Achievement Motivation(s). Journal of Individual Differences, 31, 15–21. doi: 10.1027/1614-0001/a000002 First citation in articleLink, Google Scholar