Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

The Impact of Different Methods to Correct for Response Styles on the External Validity of Self-Reports

Published Online:September 29, 2022https://doi.org/10.1027/1015-5759/a000731

Abstract

Abstract: Response styles (RSs) such as acquiescence represent systematic respondent behaviors in self-report questionnaires beyond the actual item content. They distort trait estimates and contribute to measurement bias in questionnaire-based research. Although various approaches were proposed to correct the influence of RSs, little is known about their relative performance. Because different correction methods formalize the latent traits differently, it is unclear how model choice affects the external validity of the corrected measures. Therefore, the present study on N = 1,000 Dutch respondents investigated the impact of correcting responses to measures of self-esteem and the need for cognition using structural equation models with structured residuals, multidimensional generalized partial credit models, and multinomial processing trees. The study considered three RSs: extreme, midpoint, and acquiescence RS. The results showed homogeneous correlation patterns among the modeled latent and external variables, especially if they were not themselves subject to RSs. In that case, the IRT-based models, including an uncorrected model, still yielded consistent results. Nevertheless, the strength of the effect sizes showed variation.

References

  • Baumgartner, H., & Steenkamp, J.-B. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840 First citation in articleCrossref, Google Scholar

  • Böckenholt, U., & Meiser, T. (2017). Response style analysis with threshold and multi-process IRT models: A review and tutorial. British Journal of Mathematical and Statistical Psychology, 70(1), 159–181. https://doi.org/10.1111/bmsp.12086 First citation in articleCrossref, Google Scholar

  • Cacioppo, J. T., Petty, R. E., Feinstein, J. A., & Jarvis, W. B. G. (1996). Dispositional differences in cognitive motivation: The life and times of individuals varying in need for cognition. Psychological Bulletin, 119(2), 197–253. https://doi.org/10.1037/0033-2909.119.2.197 First citation in articleCrossref, Google Scholar

  • Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06 First citation in articleCrossref, Google Scholar

  • Diedenhofen, B., & Musch, J. (2015). cocor: A comprehensive solution for the statistical comparison of correlations. PLoS One, 10(4), 1–12. https://doi.org/10.1371/journal.pone.0121945 First citation in articleCrossref, Google Scholar

  • Falk, C. F., & Cai, L. (2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21(3), 328–347. https://doi.org/10.1037/met0000059 First citation in articleCrossref, Google Scholar

  • Franck, E., De Raedt, R., Barbez, C., & Rosseel, Y. (2008). Psychometric properties of the Dutch Rosenberg Self-Esteem Scale. Psychologica Belgica, 48(1), 25–35. https://doi.org/10.5334/pb-48-1-25 First citation in articleCrossref, Google Scholar

  • Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe, 7(1), 7–28. First citation in articleGoogle Scholar

  • Greenleaf, E. A. (1992). Measuring extreme response style. Public Opinion Quarterly, 56(3), 328–351. https://doi.org/10.1086/269326 First citation in articleCrossref, Google Scholar

  • He, J., Bartram, D., Inceoglu, I., & van de Vijver, F. J. R. (2014). Response styles and personality traits: A multilevel analysis. Journal of Cross-Cultural Psychology, 45(7), 1028–1045. https://doi.org/10.1177/0022022114534773 First citation in articleCrossref, Google Scholar

  • Henninger, M., & Meiser, T. (2020). Different approaches to modeling response styles in divide-by-total item response theory models (part 1): A model integration. Psychological Methods, 25(5), 560–576. https://doi.org/10.1037/met0000249 First citation in articleCrossref, Google Scholar

  • Hoeting, J., Madigan, D., Raftery, A., & Volinsky, C. (1999). Bayesian model averaging: A tutorial. Statistical Science, 14(4), 382–401. https://doi.org/10.1214/ss/1009212519 First citation in articleCrossref, Google Scholar

  • Kieruj, N. D., & Moors, G. (2011). Response style behavior: question format dependent or personal style? Quality & Quantity, 47(1), 193–211. https://doi.org/10.1007/s11135-011-9511-4 First citation in articleCrossref, Google Scholar

  • Kutscher, T., & Eid, M. (2020). The effect of rating scale length on the occurrence of inappropriate category use for the assessment of job satisfaction: An experimental online study. Journal of Well-Being Assessment, 4(1), 1–35. https://doi.org/10.1007/s41543-020-00024-2 First citation in articleCrossref, Google Scholar

  • Liu, M., Lee, S., & Conrad, F. G. (2015). Comparing extreme response styles between agree-disagree and item-specific scales. Public Opinion Quarterly, 79(4), 952–975. https://doi.org/10.1093/poq/nfv034 First citation in articleCrossref, Google Scholar

  • Lu, D., Ye, M., & Curtis, G. P. (2015). Maximum likelihood Bayesian model averaging and its predictive analysis for groundwater reactive transport models. Journal of Hydrology, 529, 1859–1873. https://doi.org/10.1016/j.jhydrol.2015.07.029 First citation in articleCrossref, Google Scholar

  • Meiser, T., Plieninger, H., & Henninger, M. (2019). IRTree models with ordinal and multidimensional decision nodes for response styles and trait‐based rating responses. British Journal of Mathematical and Statistical Psychology, 72(3), 501–516. https://doi.org/10.1111/bmsp.12158 First citation in articleCrossref, Google Scholar

  • Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159–176. https://doi.org/10.1177/014662169201600206 First citation in articleCrossref, Google Scholar

  • Plieninger, H. (2021). Developing and applying IR-tree models: Guidelines, caveats, and an extension to multiple groups. Organizational Research Methods, 24(3), 654–670. https://doi.org/10.1177/1094428120911096 First citation in articleCrossref, Google Scholar

  • Plieninger, H., & Heck, D. W. (2018). A new model for acquiescence at the interface of psychometrics and cognitive psychology. Multivariate Behavioral Research, 53(5), 633–654. https://doi.org/10.1080/00273171.2018.1469966 First citation in articleCrossref, Google Scholar

  • Primi, R., De Fruyt, F., Santos, D., Antonoplis, S., & John, O. P. (2020). True or false? Keying direction and acquiescence influence the validity of socio-emotional skills items in predicting high school achievement. International Journal of Testing, 20(2), 97–121. https://doi.org/10.1080/15305058.2019.1673398 First citation in articleCrossref, Google Scholar

  • Primi, R., Santos, D., De Fruyt, F., & John, O. P. (2019). Comparison of classical and modern methods for measuring and correcting for acquiescence. British Journal Mathematical and Statistical Psychology, 72(3), 447–465. https://doi.org/10.1111/bmsp.12168 First citation in articleCrossref, Google Scholar

  • Pullmann, H., & Allik, J. (2000). The Rosenberg Self-Esteem Scale: Its dimensionality, stability and personality correlates in Estonian. Personality and Individual Differences, 28(4), 701–715. https://doi.org/10.1016/s0191-8869(99)00132-4 First citation in articleCrossref, Google Scholar

  • R Core Team. (2021). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/ First citation in articleGoogle Scholar

  • Reckase, M. D. (2009). Multidimensional item response theory. Springer. https://doi.org/10.1007/978–0-387–89976-3 First citation in articleCrossref, Google Scholar

  • Revelle, W. (2020). psych: Procedures for personality and psychological research [Computer software]. Northwestern University. First citation in articleGoogle Scholar

  • Robitzsch, A., Kiefer, T., & Wu, M. (2021). TAM: Test analysis modules [Computer software]. https://CRAN.R-project.org/package=TAM First citation in articleGoogle Scholar

  • Rosenberg, M. (1965). Society and the adolescent self-image. Princeton University Press. First citation in articleCrossref, Google Scholar

  • Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02 First citation in articleCrossref, Google Scholar

  • Scharl, A., & Gnambs, T. (2022, June 14). Data and supplemental materials for “The impact of different methods to correct for response styles on the external validity of self-reports.” https://osf.io/jmkqg First citation in articleGoogle Scholar

  • Scherpenzeel, A. C., & Das, M. (2010). True longitudinal and probability-based internet panels: Evidence from the Netherlands. In M. DasP. EsterL. KaczmirekEds., Social and behavioral research and the internet: Advances in applied methods and research strategies (1st ed., pp. 77–104). https://doi.org/10.4324/9780203844922-4 First citation in articleCrossref, Google Scholar

  • Schmitt, D. P., & Allik, J. (2005). Simultaneous administration of the Rosenberg Self-Esteem Scale in 53 Nations: Exploring the universal and culture-specific features of global self-esteem. Journal of Personality and Social Psychology, 89(4), 623–642. https://doi.org/10.1037/0022-3514.89.4.623 First citation in articleCrossref, Google Scholar

  • Sinclair, S. J., Blais, M. A., Gansler, D. A., Sandberg, E., Bistis, K., & LoCicero, A. (2010). Psychometric properties of the Rosenberg Self-Esteem Scale: Overall and across demographic groups living within the United States. Evaluation & The Health Professions, 33(1), 56–80. https://doi.org/10.1177/0163278709356187 First citation in articleCrossref, Google Scholar

  • Soto, C. J., & John, O. P. (2019). Optimizing the length, width, and balance of a personality scale: How do internal characteristics affect external validity? Psychological Assessment, 31(4), 444–459. https://doi.org/10.1037/pas0000586 First citation in articleCrossref, Google Scholar

  • Stan Development Team. (2021). RStan: The R interface to Stan [Computer software]. https://mc-stan.org First citation in articleGoogle Scholar

  • Van Vaerenbergh, Y., & Thomas, T. D. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25(2), 195–217. https://doi.org/10.1093/ijpor/eds021 First citation in articleCrossref, Google Scholar

  • Weijters, B., Cabooter, E., & Schillewaert, N. (2010). The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing, 27(3), 236–247. https://doi.org/10.1016/j.ijresmar.2010.02.004 First citation in articleCrossref, Google Scholar

  • Wetzel, E., Böhnke, J. R., & Rose, N. (2016). A simulation study on methods of correcting for the effects of extreme response style. Educational and Psychological Measurement, 76(2), 304–324. https://doi.org/10.1177/0013164415591848 First citation in articleCrossref, Google Scholar

  • Wetzel, E., & Carstensen, C. H. (2017). Multidimensional modeling of traits and response styles. European Journal of Psychological Assessment, 33, 352–364. https://doi.org/10.1027/1015-5759/a000291 First citation in articleLink, Google Scholar

  • Zhang, Y., & Wang, Y. (2020). Validity of three IRT models for measuring and controlling extreme and midpoint response styles. Frontiers in Psychology, 11, Article 271. https://doi.org/10.3389/fpsyg.2020.00271 First citation in articleCrossref, Google Scholar

  • Zou, G. Y. (2007). Toward using confidence intervals to compare correlations. Psychological Methods, 12(4), 399–413. https://doi.org/10.1037/1082-989x.12.4.399 First citation in articleCrossref, Google Scholar