Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

Forced-Choice Versus Likert Responses on an Occupational Big Five Questionnaire

Published Online:March 21, 2019https://doi.org/10.1027/1614-0001/a000285

Abstract

Abstract. Conventional self-report measures are prone to response biases, which distort measurement in any applied assessment. The forced-choice (FC) format was proposed as a potential remedy for these biases. The purpose of these studies was to develop and evaluate a FC questionnaire for the occupational context based on the five factor model of personality. A single-stimulus Likert questionnaire was contextualized for occupational settings and psychometrically optimized in Study 1 (N = 401). Considering optimal design strategies, we subsequently used this questionnaire to construct and validate a FC questionnaire in Study 2 (N = 517). Methodological add-ons to established approaches were applied to achieve decent confirmatory model fit. The new questionnaire shows good psychometric qualities and strong validity. We make suggestions for further applications and studies.

References

  • Ackerman, P. L. (2009). Personality and intelligence. In P. J. CorrG. MatthewsEds., The Cambridge handbook of personality psychology (pp. 162–174). New York, NY: Cambridge University Press. https://doi.org/10.1017/CBO9780511596544.013 First citation in articleGoogle Scholar

  • Ackerman, P. L. & Heggestad, E. D. (1997). Intelligence, personality, and interests: Evidence for overlapping traits. Psychological Bulletin, 121, 219–245. https://doi.org/10.1037/0033-2909.121.2.219 First citation in articleCrossref, Google Scholar

  • Alliger, G. M., Lilienfeld, S. O. & Mitchell, K. E. (1996). The susceptibility of overt and covert integrity tests to coaching and faking. Psychological Science, 7, 32–39. https://doi.org/10.1111/j.1467-9280.1996.tb00663.x First citation in articleCrossref, Google Scholar

  • Anguiano-Carrasco, C., MacCann, C., Geiger, M., Seybert, J. M. & Roberts, R. D. (2015). Development of a forced-choice measure of typical-performance emotional intelligence. Journal of Psychoeducational Assessment, 33, 83–97. https://doi.org/10.1177/0734282914550387 First citation in articleCrossref, Google Scholar

  • Bäckström, M. & Björklund, F. (2013). Social desirability in personality inventories: Symptoms, diagnosis and prescribed cure. Scandinavian Journal of Psychology, 54, 152–159. https://doi.org/10.1111/sjop.12015 First citation in articleCrossref, Google Scholar

  • Barrick, M. R., Mount, M. K. & Judge, T. A. (2001). Personality and performance at the beginning of the new millennium: What do we know and where do we go next? International Journal of Selection and Assessment, 9, 9–30. https://doi.org/10.1111/1468-2389.00160 First citation in articleCrossref, Google Scholar

  • Bates, D. & Maechler, M. (2016). Matrix: Sparse and dense matrix classes and methods. R package version 1.2-6 [R-package]. Retrieved from https://CRAN.R-project.org/package=Matrix First citation in articleGoogle Scholar

  • Benit, N. & Soellner, R. (2013). Scientist-practitioner gap in Deutschland: Eine empirische Studie am Beispiel psychologischer Testverfahren [Scientist-practitioner gap in Germany: An empirical study exemplified by psychological tests]. Zeitschrift für Arbeits-und Organisationspsychologie A&O, 57, 145–153. https://doi.org/10.1026/0932-4089/a000111 First citation in articleLink, Google Scholar

  • Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. T. & Smith, M. A. (2006). A meta-analytic investigation of job applicant faking on personality measures. International Journal of Selection and Assessment, 14, 317–335. https://doi.org/10.1111/j.1468-2389.2006.00354.x First citation in articleCrossref, Google Scholar

  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. LordM. R. NovickEds., Statistical theories of mental test scores. Reading, MA: Addison-Wesley. First citation in articleGoogle Scholar

  • Borghuis, J., Denissen, J. J., Oberski, D., Sijtsma, K., Meeus, W. H., Branje, S., … Bleidorn, W. (2017). Big Five personality stability, change, and codevelopment across adolescence and early adulthood. Journal of Personality and Social Psychology, 113, 641–657. https://doi.org/10.31234/osf.io/8pnvk First citation in articleCrossref, Google Scholar

  • Borkenau, P. & Ostendorf, F. (1990). Comparing exploratory and confirmatory factor analysis: A study on the 5-factor model of personality. Personality and Individual Differences, 11, 515–524. https://doi.org/10.1016/0191-8869(90)90065-y First citation in articleCrossref, Google Scholar

  • Brown, A. & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460–502. https://doi.org/10.1177/0013164410375112 First citation in articleCrossref, Google Scholar

  • Brown, A. & Maydeu-Olivares, A. (2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44, 1135–1147. https://doi.org/10.3758/s13428-012-0217-x First citation in articleCrossref, Google Scholar

  • Brown, A. & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36–52. https://doi.org/10.1037/a0030641 First citation in articleCrossref, Google Scholar

  • Bulheller, S.Häcker, H. (Eds.). (1998). Raven’s Progressive Matrices und Vocabulary Scales, von J. C. Raven, J. Raven und J. H. Court. Frankfurt, Germany: Swets & Zeitlinger. First citation in articleGoogle Scholar

  • Campbell, D. T. & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. https://doi.org/10.1037/h0046016 First citation in articleCrossref, Google Scholar

  • Cattell, R. B. (1944). Psychological measurement: Normative, ipsative, interactive. Psychological Review, 51, 292–303. https://doi.org/10.1037/h0057299 First citation in articleCrossref, Google Scholar

  • Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1–29. https://doi.org/10.18637/jss.v048.i06 First citation in articleCrossref, Google Scholar

  • Chamorro-Premuzic, T. & Furnham, A. (2005). Personality and intellectual competence. Mahwah, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Christiansen, N. D., Burns, G. N. & Montgomery, G. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18, 267–307. https://doi.org/10.1207/s15327043hup1803_4 First citation in articleCrossref, Google Scholar

  • Cochran, W. G. & Cox, G. M. (1992). Experimental designs (2nd ed.). New York, NY: Wiley. First citation in articleGoogle Scholar

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Colorni, A., Dorigo, M. & Maniezzo, V. (1991). Distributed optimization by ant colonies. Proceedings of the First European Conference on Artificial Life, 142, 134–142. First citation in articleGoogle Scholar

  • Credé, M., Bashshur, M. & Niehorster, S. (2010). Reference group effects in the measurement of personality and attitudes. Journal of Personality Assessment, 92, 390–399. https://doi.org/10.1080/00223891.2010.497393 First citation in articleCrossref, Google Scholar

  • DeYoung, C. G., Quilty, L. C. & Peterson, J. B. (2007). Between facets and domains: 10 aspects of the Big Five. Journal of Personality and Social Psychology, 93, 880–896. https://doi.org/10.1037/0022-3514.93.5.880 First citation in articleCrossref, Google Scholar

  • Donovan, J. J., Dwight, S. A. & Hurtz, G. M. (2003). An assessment of the prevalence, severity, and verifiability of entry-level applicant faking using the randomized response technique. Human Performance, 16, 81–106. https://doi.org/10.1207/s15327043hup1601_4 First citation in articleCrossref, Google Scholar

  • Dudley, N. M., Orvis, K. A., Lebiecki, J. E. & Cortina, J. M. (2006). A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. Journal of Applied Psychology, 91, 40–57. https://doi.org/10.1037/0021-9010.91.1.40 First citation in articleCrossref, Google Scholar

  • Ellingson, J. E., Sackett, P. R. & Hough, L. M. (1999). Social desirability corrections in personality measurement: Issues of applicant comparison and construct validity. Journal of Applied Psychology, 84, 155. https://doi.org/10.1037/0021-9010.84.2.155 First citation in articleCrossref, Google Scholar

  • Frey, A., Hartig, J. & Rupp, A. A. (2009). An NCME instructional module on booklet designs in large-scale assessments of student achievement: Theory and practice. Educational Measurement: Issues and Practice, 28, 39–53. https://doi.org/10.1111/j.1745-3992.2009.00154.x First citation in articleCrossref, Google Scholar

  • Friedman, H. H. & Amoo, T. (1999). Rating the rating scales. Journal of Marketing Management, 9, 114–123. First citation in articleGoogle Scholar

  • Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. MervieldeI. DearyF. De FruytF. OstendorfEds., Personality Psychology in Europe (pp. 7–28). Tilburg, The Netherlands: Tilburg University Press. First citation in articleGoogle Scholar

  • Görlich, Y. & Schuler, H. (2007). Azubi-TH. Arbeitsprobe zur berufsbezogenen Intelligenz [Azubi-TH. Work sample for measuring occupational intelligence]. Göttingen, Germany: Hogrefe. First citation in articleGoogle Scholar

  • Guenole, N., Brown, A. A. & Cooper, A. J. (2018). Forced-choice assessment of work-related maladaptive personality traits: Preliminary evidence from an application of Thurstonian item response modeling. Assessment, 25, 513–526. https://doi.org/10.1177/1073191116641181 First citation in articleCrossref, Google Scholar

  • Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74, 167. https://doi.org/10.1037/h0029780 First citation in articleCrossref, Google Scholar

  • Higham, N. J. (2002). Computing the nearest correlation matrix – a problem from finance. IMA Journal of Numerical Analysis, 22, 329–343. https://doi.org/10.1093/imanum/22.3.329 First citation in articleCrossref, Google Scholar

  • Hu, L. & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. https://doi.org/10.1080/10705519909540118 First citation in articleCrossref, Google Scholar

  • Johnson, C. E., Wood, R. & Blinkhorn, S. F. (1988). Spuriouser and spuriouser: The use of ipsative personality tests. Journal of Occupational Psychology, 61, 153–162. https://doi.org/10.1111/j.2044-8325.1988.tb00279.x First citation in articleCrossref, Google Scholar

  • Joubert, T., Inceoglu, I., Bartram, D., Dowdeswell, K. & Lin, Y. (2015). A comparison of the psychometric properties of the forced choice and Likert scale versions of a personality instrument. International Journal of Selection and Assessment, 23, 92–97. https://doi.org/10.1111/ijsa.12098 First citation in articleCrossref, Google Scholar

  • Judge, T. A., Rodell, J. B., Klinger, R. L., Simon, L. S. & Crawford, E. R. (2013). Hierarchical representations of the five-factor model of personality in predicting job performance: Integrating three organizing frameworks with two theoretical perspectives. Journal of Applied Psychology, 98, 875–925. https://doi.org/10.1037/a0033901 First citation in articleCrossref, Google Scholar

  • Kersting, M. (2005). AKZEPT! Fragebogen zur Messung der Akzeptanz diagnostischer Verfahren [AKZEPT! Questionnaire for measuring the acceptance of diagnostic procedures]. Aachen, Germany: Rheinisch-Westfälische Technische Hochschule Aachen. First citation in articleGoogle Scholar

  • Kersting, M. (2008). Zur Akzeptanz von Intelligenz- und Leistungstests [On the acceptance of intelligence and performance tests]. Report Psychologie, 33, 420–433. First citation in articleGoogle Scholar

  • Klehe, U.-C., Kleinmann, M., Hartstein, T., Melchers, K. G., König, C. J., Heslin, P. A. & Lievens, F. (2012). Responding to personality tests in a selection context: The role of the ability to identify criteria and the ideal-employee factor. Human Performance, 25, 273–302. https://doi.org/10.1080/08959285.2012.703733 First citation in articleCrossref, Google Scholar

  • König, C. J., Klehe, U.-C., Berchtold, M. & Kleinmann, M. (2010). Reasons for being selective when choosing personnel selection procedures. International Journal of Selection and Assessment, 18, 17–27. https://doi.org/10.1111/j.1468-2389.2010.00485.x First citation in articleCrossref, Google Scholar

  • Konstabel, K., Aavik, T. & Allik, J. (2006). Social desirability and consensual validity of personality traits. European Journal of Personality, 20, 549–566. https://doi.org/10.1002/per.593 First citation in articleCrossref, Google Scholar

  • Leite, W. L., Huang, I.-C. & Marcoulides, G. A. (2008). Item selection for the development of short forms of scales using an Ant Colony Optimization algorithm. Multivariate Behavioral Research, 43, 411–431. https://doi.org/10.1080/00273170802285743 First citation in articleCrossref, Google Scholar

  • Martin, B. A., Bowen, C.-C. & Hunt, S. T. (2002). How effective are people at faking on personality questionnaires? Personality and Individual Differences, 32, 247–256. https://doi.org/10.1016/s0191-8869(01)00021-6 First citation in articleCrossref, Google Scholar

  • McDonald, R. P. (1999). Test theory: A unified treatment. Hillsdale, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Meade, A. W. (2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77, 531–551. https://doi.org/10.1348/0963179042596504 First citation in articleCrossref, Google Scholar

  • Mussel, P. & Paelecke, M. (2018). BFAS-G. Big Five Aspect Scales – German. In Leibniz-Zentrum für Psychologische Information und Dokumentation (ZPID)Ed., Elektronisches Testarchiv (PSYNDEX Tests-Nr. 9007737). Trier, Germany: ZPID. https://doi.org/10.23668/psycharchives.2341 First citation in articleGoogle Scholar

  • Muthén, L. K. & Muthén, B. O. (1998). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén. First citation in articleGoogle Scholar

  • Olaru, G., Schroeders, U., Wilhelm, O. & Ostendorf, F. (2018). A confirmatory examination of age‐associated personality differences: Deriving age‐related measurement invariant solutions using ant colony optimization. Journal of Personality, 86, 1037–1049. https://doi.org/10.1111/jopy.12373 First citation in articleCrossref, Google Scholar

  • Olaru, G., Witthöft, M. & Wilhelm, O. (2015). Methods matter: Testing competing models for designing short-scale Big-Five assessments. Journal of Research in Personality, 59, 56–68. https://doi.org/10.1016/j.jrp.2015.09.001 First citation in articleCrossref, Google Scholar

  • Ones, D. S., Wiernik, B. M., Wilmot, M. P. & Kostal, J. W. (2016). Conceptual and methodological complexity of narrow trait measures in personality-outcome research: Better knowledge by partitioning variance from multiple latent traits and measurement artifacts. European Journal of Personality, 30, 319–321. https://doi.org/10/bp27 First citation in articleGoogle Scholar

  • Ostendorf, F. & Angleitner, A. (2004). NEO-PI-R: NEO-Persönlichkeitsinventar nach Costa und McCrae. Testmanual [NEO-PI-R: NEO personality inventory by Costa and McCrae. Test manual]. Göttingen, Germany: Hogrefe. First citation in articleGoogle Scholar

  • Parker, J. D., Bagby, R. M. & Summerfeldt, L. J. (1993). Confirmatory factor analysis of the Revised NEO Personality Inventory. Personality and Individual Differences, 15, 463–466. https://doi.org/10.1016/0191-8869(93)90074-d First citation in articleCrossref, Google Scholar

  • Piotrowski, C. & Armstrong, T. (2006). Current recruitment and selection practices: A national survey of Fortune 1000 firms. North American Journal of Psychology, 8, 489–496. First citation in articleGoogle Scholar

  • R Core Team. (2015). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. First citation in articleGoogle Scholar

  • Revelle, W. & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145–154. https://doi.org/10.1007/s11336-008-9102-z First citation in articleCrossref, Google Scholar

  • S & F Personalpsychologie. (n.d.). KFM, Unpublished test First citation in articleGoogle Scholar

  • S & F Personalpsychologie. (n.d.). LEXI, Unpublished test First citation in articleGoogle Scholar

  • S & F Personalpsychologie. (n.d.). MATRIX, Unpublished test First citation in articleGoogle Scholar

  • S & F Personalpsychologie. (n.d.). TAKE5, Unpublished test First citation in articleGoogle Scholar

  • Salgado, J. F. (1997). The Five Factor Model of personality and job performance in the European Community. Journal of Applied Psychology, 82, 30. https://doi.org/10.1037/0021-9010.82.1.30 First citation in articleCrossref, Google Scholar

  • Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Richmond, VA: Psychometric Society. First citation in articleGoogle Scholar

  • Sawilowsky, S. S. (2002). A quick distribution-free test for trend that contributes evidence of construct validity. Measurement and Evaluation in Counseling and Development, 35, 78. First citation in articleCrossref, Google Scholar

  • Schmidt, F. L., Oh, I. S. & Shaffer, J. A. (2016). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 100 years of research findings (Working paper). Retrieved from https://www.researchgate.net/publication/309203898 First citation in articleGoogle Scholar

  • Schroeders, U., Wilhelm, O. & Olaru, G. (2016). Meta-heuristics in short scale construction: Ant colony optimization and genetic algorithm. PLoS One, 11, e0167110. https://doi.org/10.1371/journal.pone.0167110 First citation in articleCrossref, Google Scholar

  • Shaffer, J. A. & Postlethwaite, B. E. (2012). A matter of context: A meta-analytic investigation of the relative validity of contextualized and noncontextualized personality measures. Personnel Psychology, 65, 445–494. https://doi.org/10.1111/j.1744-6570.2012.01250.x First citation in articleCrossref, Google Scholar

  • Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251. https://doi.org/10.1037//0033-2909.87.2.245 First citation in articleCrossref, Google Scholar

  • Steinmayr, R., Schütz, A., Hertel, J. & Schröder-Abé, M. (2011). Mayer-Salovey-Caruso Test zur Emotionalen Intelligenz [Mayer-Salovey-Caruso test of emotional intelligence]. Bern, Switzerland: Hans Huber. First citation in articleGoogle Scholar

  • Sticca, F., Goetz, T., Bieg, M., Hall, N. C., Eberle, F. & Haag, L. (2017). Examining the accuracy of students’ self-reported academic grades from a correlational and a discrepancy perspective: Evidence from a longitudinal study. PLoS One, 12, e0187367. https://doi.org/10.1371/journal.pone.0187367 First citation in articleCrossref, Google Scholar

  • Tett, R. P., Jackson, D. N. & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44, 703–742. https://doi.org/10.1111/j.1744-6570.1991.tb00696.x First citation in articleCrossref, Google Scholar

  • Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286. https://doi.org/10.1037/h0070288 First citation in articleCrossref, Google Scholar

  • Thurstone, L. L. (1931). Rank order as a psycho-physical method. Journal of Experimental Psychology, 14, 187–201. https://doi.org/10.1037/h0070025 First citation in articleCrossref, Google Scholar

  • Viswesvaran, C. & Ones, D. S. (1999). Meta-analyses of fakability estimates: Implications for personality measurement. Educational and Psychological Measurement, 59, 197–210. https://doi.org/10.1177/00131649921969802 First citation in articleCrossref, Google Scholar

  • Wetzel, E., Böhnke, J. R. & Brown, A. (2016). Response biases. In F. T. LeongD. BartramF. CheungK. F. GeisingerD. IliescuEds., The ITC international handbook of testing and assessment (pp. 349–363). New York, NY: Oxford University Press. https://doi.org/10.1093/med:psych/9780199356942.003.0024 First citation in articleGoogle Scholar

  • Wheeler, B. (2014). AlgDesign: Algorithmic experimental design [R-package]. Retrieved from https://CRAN.R-project.org/package=AlgDesign First citation in articleGoogle Scholar

  • Wildt, A. R. & Mazis, M. B. (1978). Determinants of scale response: Label versus position. Journal of Marketing Research, 15, 261–267. https://doi.org/10.2307/3151256 First citation in articleCrossref, Google Scholar

  • Worcester, R. M. & Burns, T. R. (1975). Statistical examination of relative precision of verbal scales. Journal of the Market Research Society, 17, 181–197. First citation in articleGoogle Scholar

  • Zeidner, M. & Matthews, G. (2000). Intelligence and personality. In R. SternbergEd., Handbook of intelligence (pp. 581–610). New York, NY: Cambridge University Press. https://doi.org/10.1017/cbo9780511807947.027 First citation in articleGoogle Scholar

  • Zibarras, L. D. & Woods, S. A. (2010). A survey of UK selection practices across different organization sizes and industry sectors. Journal of Occupational and Organizational Psychology, 83, 499–511. https://doi.org/10.1348/096317909x425203 First citation in articleCrossref, Google Scholar

  • Ziegler, M. (2015). “F*** you, I won’t do what you told me!” – response biases as threats to psychological assessment. European Journal of Psychological Assessment, 31, 153–158. https://doi.org/10.1027/1015-5759/a000292 First citation in articleLink, Google Scholar

  • Ziegler, M.MacCann, C.Roberts, R. D. (Eds.). (2011). New perspectives on faking in personality assessment. New York, NY: Oxford University Press. First citation in articleCrossref, Google Scholar

  • Zinbarg, R. E., Revelle, W., Yovel, I. & Li, W. (2005). Cronbach’s α, Revelle’s β, and Mcdonald’s ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123–133. https://doi.org/10.1007/s11336-003-0974-7 First citation in articleCrossref, Google Scholar