Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article

The Effect of Response Formats on Response Style Strength

An Experimental Comparison

Published Online:August 02, 2023https://doi.org/10.1027/1015-5759/a000779

Abstract

Abstract: Many researchers use self-report data to examine abilities, personalities, or attitudes. At the same time, there is a widespread concern that response styles, such as the tendency to give extreme, midscale, or acquiescent responses, may threaten data quality. As an alternative to post hoc control of response styles using psychometric models, a priori control using specific response formats may be a means to reduce biasing response style effects in self-report data in day-to-day research practice. Previous research has suggested that response styles were less influential in a Drag-and-Drop (DnD) format compared to the traditional Likert-type format. In this article, we further examine the advantage of the DnD format, test its generalizability, and investigate its underlying mechanisms. In two between-participants experiments, we tested different versions of the DnD format against the Likert format. We found no evidence for reduced response style influence in any of the DnD conditions, nor did we find any difference between the conditions in terms of the validity of the measures to external criteria. We conclude that adaptations of response formats, such as the DnD format, may be promising, but require more thorough examination before recommending them as a means to reduce response style influence in psychological measurement.

References

  • Adams, D. J., Bolt, D. M., Deng, S., Smith, S. S., & Baker, T. B. (2019). Using multidimensional item response theory to evaluate how response styles impact measurement. British Journal of Mathematical and Statistical Psychology, 72(3), 466–485. https://doi.org/10.1111/bmsp.12169 First citation in articleCrossref, Google Scholar

  • Ashton, M. C., Lee, K., Marcus, B., & De Vries, R. E. (2007). German lexical personality factors: Relations with the HEXACO model. European Journal of Personality, 21(1), 23–43. https://doi.org/10.1002/per.597 First citation in articleCrossref, Google Scholar

  • Ashton, M. C., Perugini, M., De Vries, R. E., Boies, K., Lee, K., Szarota, P., Di Blas, L., & De Raad, B. (2004). A six-factor structure of personality-descriptive adjectives: Solutions from psycholexical studies in seven languages. Journal of Personality and Social Psychology, 86(2), 356–366. https://doi.org/10.1037/0022-3514.86.2.356 First citation in articleCrossref, Google Scholar

  • Bachman, J. G., & O’Malley, P. M. (1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly, 48, 491–509. https://doi.org/10.1086/268845 First citation in articleCrossref, Google Scholar

  • Baumgartner, H., & Steenkamp, J.-B. E. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840 First citation in articleCrossref, Google Scholar

  • Böckenholt, U. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17, 665–678. https://doi.org/10.1037/a0028111 First citation in articleCrossref, Google Scholar

  • Böckenholt, U. (2017). Measuring response styles in Likert items. Psychological Methods, 22, 69–83. https://doi.org/10.1037/met0000106 First citation in articleCrossref, Google Scholar

  • Böckenholt, U. (2019). Assessing item-feature effects with item response tree models. British Journal of Mathematical and Statistical Psychology, 72(3), 486–500. https://doi.org/10.1111/bmsp.12163 First citation in articleCrossref, Google Scholar

  • Böckenholt, U., & Meiser, T. (2017). Response style analysis with threshold and multi-process IRT models: A review and tutorial. British Journal of Mathematical and Statistical Psychology, 70, 159–181. https://doi.org/10.1111/bmsp.12086 First citation in articleCrossref, Google Scholar

  • Bolt, D. M., & Johnson, T. R. (2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33, 335–352. https://doi.org/10.1177/0146621608329891 First citation in articleCrossref, Google Scholar

  • Bolt, D. M., Lu, Y., & Kim, J.-S. (2014). Measurement and control of response styles using anchoring vignettes: A model-based approach. Psychological Methods, 19, 528–541. https://doi.org/10.1037/met0000016 First citation in articleCrossref, Google Scholar

  • Bolt, D. M., & Newton, J. R. (2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71(5), 814–833. https://doi.org/10.1177/0013164410388411 First citation in articleCrossref, Google Scholar

  • Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112 First citation in articleCrossref, Google Scholar

  • Brown, A., & Maydeu-Olivares, A. (2012a). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44, 1135–1147. https://doi.org/10.3758/s13428-012-0217-x First citation in articleCrossref, Google Scholar

  • Brown, A., & Maydeu-Olivares, A. (2012b). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36–52. https://doi.org/10.1037/a0030641 First citation in articleCrossref, Google Scholar

  • Brummett, B. H., Babyak, M. A., Williams, R. B., Barefoot, J. C., Costa, P. T., & Siegler, I. C. (2006). NEO personality domains and gender predict levels and trends in body mass index over 14 years during midlife. Journal of Research in Personality, 40, 222–236. https://doi.org/10.1016/j.jrp.2004.12.002 First citation in articleCrossref, Google Scholar

  • Couch, A., & Keniston, K. (1960). Yeasayers and naysayers: Agreeing response set as a personality variable. The Journal of Abnormal and Social Psychology, 60, 151–174. https://doi.org/10.1037/h0040372 First citation in articleCrossref, Google Scholar

  • Cronbach, L. J. (1942). Studies of acquiescence as a factor in the true-false test. Journal of Educational Psychology, 33(6), 401–415. https://doi.org/10.1037/h0054677 First citation in articleCrossref, Google Scholar

  • De Boeck, P., & Partchev, I. (2012). IRTrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 48(1), 1–28. https://doi.org/10.18637/jss.v048.c01 First citation in articleGoogle Scholar

  • DeCastellarnau, A. (2018). A classification of response scale characteristics that affect data quality: A literature review. Quality and Quantity, 52(4), 1523–1559. https://doi.org/10.1007/s11135-017-0533-4 First citation in articleCrossref, Google Scholar

  • De Jong, M. G., Steenkamp, J.-B. E., Fox, J.-P., & Baumgartner, H. (2008). Using item response theory to measure extreme response style in marketing research: A global investigation. Journal of Marketing Research, 45, 104–115. https://doi.org/10.1509/jmkr.45.1.104 First citation in articleCrossref, Google Scholar

  • Falk, C. F., & Cai, L. (2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21, 328–347. https://doi.org/10.1037/met0000059 First citation in articleCrossref, Google Scholar

  • Frick, S., Brown, A., & Wetzel, E. (2021). Investigating the normativity of trait estimates from multidimensional forced-choice data. Multivariate Behavioral Research, 58(1), 1–29. https://doi.org/10.1080/00273171.2021.1938960 First citation in articleCrossref, Google Scholar

  • Greenleaf, E. A. (1992). Measuring extreme response style. Public Opinion Quarterly, 56, 328–351. https://doi.org/10.1086/269326 First citation in articleCrossref, Google Scholar

  • Guenole, N., Brown, A. A., & Cooper, A. J. (2018). Forced-choice assessment of work-related maladaptive personality traits: Preliminary evidence from an application of Thurstonian item response modeling. Assessment, 25(4), 513–526. https://doi.org/10.1177/1073191116641181 First citation in articleCrossref, Google Scholar

  • Hamilton, D. L. (1968). Personality attributes associated with extreme response style. Psychological Bulletin, 69, 192–203. https://doi.org/10.1037/h0025606 First citation in articleCrossref, Google Scholar

  • Henninger, M., & Meiser, T. (2020a). Different approaches to modeling response styles in divide-by-total IRT models (Part I): A model integration. Psychological Methods, 25, 560–576. https://doi.org/10.1037/met0000249 First citation in articleCrossref, Google Scholar

  • Henninger, M., & Meiser, T. (2020b). Different approaches to modeling response styles in divide-by-total IRT models (Part II): Applications and novel extensions. Psychological Methods, 25, 577–595. https://doi.org/10.1037/met0000268 First citation in articleCrossref, Google Scholar

  • Henninger, M., & Meiser, T. (2022). Quality control: Response style modeling. In R. J. TierneyF. RizviK. Erkican (Eds.). International encyclopedia of education (4th ed., pp. 331–340). Elsevier. https://doi.org/10.1016/B978-0-12-818630-5.10041-7 First citation in articleCrossref, Google Scholar

  • Hilbig, B. E., Thielmann, I., Hepp, J., Klein, S. A., & Zettler, I. (2015). From personality to altruistic behavior (and back): Evidence from a double-blind dictator game. Journal of Research in Personality, 55, 46–50. https://doi.org/10.1016/j.jrp.2014.12.004 First citation in articleCrossref, Google Scholar

  • Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114. https://doi.org/10.1007/s10869-011-9231-8 First citation in articleCrossref, Google Scholar

  • Hui, C. H., & Triandis, H. C. (1985). The instability of response sets. Public Opinion Quarterly, 49, 253–260. https://doi.org/10.1086/268918 First citation in articleCrossref, Google Scholar

  • Jeon, M., & De Boeck, P. (2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48, 1070–1085. https://doi.org/10.3758/s13428-015-0631-y First citation in articleCrossref, Google Scholar

  • Khorramdel, L., & von Davier, M. (2014a). Measuring response styles across the Big Five: A multiscale extension of an approach using multinomial processing trees. Multivariate Behavioral Research, 49, 161–177. https://doi.org/10.1080/00273171.2013.866536 First citation in articleCrossref, Google Scholar

  • Khorramdel, L., & von Davier, M. (2014b). Measuring response styles across the Big Five: A multiscale extension of an approach using multinomial processing trees. Multivariate Behavioral Research, 49, 161–177. https://doi.org/10.1080/00273171.2013.866536 First citation in articleCrossref, Google Scholar

  • Lee, K., & Ashton, M. C. (2006). Further assessment of the HEXACO Personality Inventory: Two new facet scales and an observer report form. Psychological Assessment, 18, 182–191. https://doi.org/10.1037/1040-3590.18.2.182 First citation in articleCrossref, Google Scholar

  • Levy, R., Xu, Y., Yel, N., & Svetina, D. (2015). A standardized generalized dimensionality discrepancy measure and a standardized model-based covariance for dimensionality assessment for multidimensional models. Journal of Educational Measurement, 52, 144–158. https://doi.org/10.1111/jedm.12070 First citation in articleCrossref, Google Scholar

  • Machunsky, M., & Meiser, T. (2006). Personal need for structure als differenzialpsychologisches Konstrukt in der Sozialpsychologie [Personal need for structure as a construct of dispositional differences in social psychology]. Zeitschrift für Sozialpsychologie, 37, 87–97. https://doi.org/10.1024/0044-3514.37.2.87 First citation in articleLink, Google Scholar

  • Maydeu-Olivares, A., & Brown, A. (2010). Item response modeling of paired comparison and ranking data. Multivariate Behavioral Research, 45, 935–974. https://doi.org/10.1080/00273171.2010.531231 First citation in articleCrossref, Google Scholar

  • McKeown, B., & Thomas, D. B. (1988). Q methodology. Sage. First citation in articleCrossref, Google Scholar

  • Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455. https://doi.org/10.1037/a0028085 First citation in articleCrossref, Google Scholar

  • Meiser, T., & Machunsky, M. (2008). The personal structure of personal need for structure: A mixture-distribution Rasch analysis. European Journal of Psychological Assessment, 24, 27–34. https://doi.org/10.1027/1015-5759.24.1.27 First citation in articleLink, Google Scholar

  • Meiser, T., Plieninger, H., & Henninger, M. (2019). IRTree models with ordinal and multidimensional decision nodes for response styles and trait-based rating responses. British Journal of Mathematical and Statistical Psychology, 72, 501–516. https://doi.org/10.1111/bmsp.12158 First citation in articleCrossref, Google Scholar

  • Moors, G. (2003). Diagnosing response style behavior by means of a latent-class factor approach. Socio-demographic correlates of gender role attitudes and perceptions of ethnic discrimination reexamined. Quality and Quantity, 37, 277–302. https://doi.org/10.1023/A:1024472110002 First citation in articleCrossref, Google Scholar

  • Moors, G. (2004). Facts and artefacts in the comparison of attitudes among ethnic minorities. A multigroup latent class structure model with adjustment for response style behavior. European Sociological Review, 20, 303–320. https://doi.org/10.1093/esr/jch026 First citation in articleCrossref, Google Scholar

  • Morren, M., Gelissen, J. P., & Vermunt, J. K. (2012). The impact of controlling for extreme responding on measurement equivalence in cross-cultural research. Methodology, 8, 159–170. https://doi.org/10.1027/1614-2241/a000048 First citation in articleLink, Google Scholar

  • Moshagen, M., Hilbig, B. E., & Zettler, I. (2014). Faktorenstruktur, psychometrische Eigenschaften und Messinvarianz der deutschsprachigen Version des 60-Item HEXACO Persönlichkeitsinventars [Factor structure, psychometric properties, and measurement invariance of the German-language version of the 60-item HEXACO personality inventory]. Diagnostica, 60, 86–97. https://doi.org/10.1026/0012-1924/a000112 First citation in articleLink, Google Scholar

  • Neuberg, S. L., & Newsom, J. T. (1993). Personal need for structure: Individual differences in the desire for simpler structure. Journal of Personality and Social Psychology, 65, 113–131. https://doi.org/10.1037/0022-3514.65.1.113 First citation in articleCrossref, Google Scholar

  • Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. https://doi.org/10.1016/j.jesp.2009.03.009 First citation in articleCrossref, Google Scholar

  • Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. RobinsonP. R. ShaverL. S. WrightsmanEds., Measures of personality and social psychological attitudes (pp. 17–59). Academic Press. https://doi.org/10.1016/B978-0-12-590241-0.50006-X First citation in articleCrossref, Google Scholar

  • Plieninger, H. (2017). Mountain or molehill? A simulation study on the impact of response styles. Educational and Psychological Measurement, 77, 32–53. https://doi.org/10.1177/0013164416636655 First citation in articleCrossref, Google Scholar

  • Plieninger, H., & Meiser, T. (2014). Validity of multiprocess IRT models for separating content and response styles. Educational and Psychological Measurement, 74, 875–899. https://doi.org/10.1177/0013164413514998 First citation in articleCrossref, Google Scholar

  • Plieninger, H., & Henninger, M. (2023, March 31). The effect of response formats on response style strength: An experimental comparison. https://doi.org/10.17605/OSF.IO/ZU5TH First citation in articleGoogle Scholar

  • Revelle, W. (2017). psych: Procedures for psychological, psychometric, and personality research ((Version 1.7.5). https://CRAN.R-project.org/package=psych First citation in articleGoogle Scholar

  • Robitzsch, A., Kiefer, T., & Wu, M. (2022). TAM: Test analysis modules (R package version 4.1–4). http://cran.r-project.org/package=TAM First citation in articleGoogle Scholar

  • Rost, J. (1991). A logistic mixture distribution model for polychotomous item responses. British Journal of Mathematical and Statistical Psychology, 44, 75–92. https://doi.org/10.1111/j.2044-8317.1991.tb00951.x First citation in articleCrossref, Google Scholar

  • Rost, J., Carstensen, C. H., & von Davier, M. (1999). Sind die Big Five Raschskalierbar? Eine Reanalyse der NEO-FFI-Normierungsdaten [Are the Big Five Rasch scaleable? A reanalysis of the NEO-FFI norm data]. Diagnostica, 45(3), 119–127. https://doi.org/10.1026//0012-1924.45.3.119 First citation in articleLink, Google Scholar

  • Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph). Psychometric Society. http://www.psychometrika.org/journal/online/MN17.pdf First citation in articleGoogle Scholar

  • Smith, R. (1995, April). Using item mean squares to evaluate fit to the Rasch model. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA, April 18–22. https://files.eric.ed.gov/fulltext/ED384617.pdf First citation in articleGoogle Scholar

  • Stoel, R. D., Garre, F. G., Dolan, C., & Van Den Wittenboer, G. (2006). On the likelihood ratio test in structural equation modeling when parameters are subject to boundary constraints. Psychological Methods, 11(4), 439–455. https://doi.org/10.1037/1082-989X.11.4.439 First citation in articleCrossref, Google Scholar

  • Sutin, A. R., Ferrucci, L., Zonderman, A. B., & Terracciano, A. (2011). Personality and obesity across the adult life span. Journal of Personality and Social Psychology, 101, 579–592. https://doi.org/10.1037/a0024286 First citation in articleCrossref, Google Scholar

  • Thurstone, L. L. (1928). Attitudes can be measured. American Journal of Sociology, 33, 529–554. https://doi.org/10.1086/214483 First citation in articleCrossref, Google Scholar

  • Thurstone, L. L. (1930). A scale for measuring attitude toward the movies. The Journal of Educational Research, 22, 89–94. https://doi.org/10.1080/00220671.1930.10880071 First citation in articleCrossref, Google Scholar

  • Van Vaerenbergh, Y., & Thomas, T. D. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25, 195–217. https://doi.org/10.1093/ijpor/eds021 First citation in articleCrossref, Google Scholar

  • van Herk, H., Poortinga, Y. H., & Verhallen, T. M. M. (2004). Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35, 346–360. https://doi.org/10.1177/0022022104264126 First citation in articleCrossref, Google Scholar

  • von Davier, M., Shin, H.-J., Khorramdel, L., & Stankov, L. (2018). The effects of vignette scoring on reliability and validity of self-reports. Applied Psychological Measurement, 42(4), 291–306. https://doi.org/10.1177/0146621617730389 First citation in articleCrossref, Google Scholar

  • Weijters, B., Cabooter, E. F. K., & Schillewaert, N. (2010). The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing, 27, 236–247. https://doi.org/10.1016/j.ijresmar.2010.02.004 First citation in articleCrossref, Google Scholar

  • Weijters, B., Geuens, M., & Schillewaert, N. (2010). The stability of individual response styles. Psychological Methods, 15, 96–110. https://doi.org/10.1037/a0018721 First citation in articleCrossref, Google Scholar

  • Weng, L.-J. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956–972. https://doi.org/10.1177/0013164404268674 First citation in articleCrossref, Google Scholar

  • Wetzel, E., Böhnke, J. R., Carstensen, C. H., Ziegler, M., & Ostendorf, F. (2013). Do individual response styles matter? Assessing differential item functioning for men and women in the NEO-PI-R. Journal of Individual Differences, 34, 69–81. https://doi.org/10.1027/1614-0001/a000102 First citation in articleLink, Google Scholar

  • Wetzel, E., & Carstensen, C. H. (2017). Multidimensional modeling of traits and response styles. European Journal of Psychological Assessment, 33, 352–364. https://doi.org/10.1027/1015-5759/a000291 First citation in articleLink, Google Scholar

  • Wetzel, E., & Frick, S. (2020). Comparing the validity of trait estimates from the multidimensional forced-choice format and the rating scale format. Psychological Assessment, 32(3), 239–253. https://doi.org/10.1037/pas0000781.supp First citation in articleCrossref, Google Scholar

  • Wetzel, E., Frick, S., & Greiff, S. (2020). The multidimensional forced-choice format as an alternative for rating scales. European Journal of Psychological Assessment, 36(4), 511–515. https://doi.org/10.1027/1015-5759/a000609 First citation in articleLink, Google Scholar

  • Wetzel, E., & Greiff, S. (2018). The world beyond rating scales: Why we should think more carefully about the response format in questionnaires. European Journal of Psychological Assessment, 34, 1–5. https://doi.org/10.1027/1015-5759/a000469 First citation in articleLink, Google Scholar

  • Wetzel, E., Lüdtke, O., Zettler, I., & Böhnke, J. R. (2016). The stability of extreme response style and acquiescence over 8 years. Assessment, 23, 279–291. https://doi.org/10.1177/1073191115583714 First citation in articleCrossref, Google Scholar

  • Zhao, K., & Smillie, L. D. (2015). The role of interpersonal traits in social decision making. Personality and Social Psychology Review, 19, 277–302. https://doi.org/10.1177/1088868314553709 First citation in articleCrossref, Google Scholar