Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Multistudy Report

Confound It!

Social Desirability and the “Reverse-Scoring” Method Effect

Published Online:March 28, 2018https://doi.org/10.1027/1015-5759/a000459

Abstract

Abstract. Many investigators have noted “reverse-coding” method factors when exploring response pattern structure with psychological inventory data. The current article probes for the existence of a confound in these investigations, whereby an item’s level of saturation with socially desirable content tends to covary with the item’s substantive scale keying. We first investigate its existence, demonstrating that 15 of 16 measures that have been previously implicated as exhibiting a reverse-scoring method effect can also be reasonably characterized as exhibiting a scoring key/social desirability confound. A second set of analyses targets the extent to which the confounding variable may confuse interpretation of factor analytic results and documents strong social desirability associations. The results suggest that assessment developers perhaps consider the social desirability scale value of indicators when constructing scale aggregates (and possibly scales when investigating inter-construct associations). Future investigations would ideally disentangle the confound via experimental manipulation.

References *Measures and data were included in the current manuscript procedure.

  • Alessandri, G., Vecchione, M., Fagnani, C., Bentler, P. M., Barbaranelli, C., Medda, E., … Caprara, G. V. (2010). Much more than model fitting? Evidence for the heritability of method effect associated with positively worded items of the Life Orientation Test Revised. Structural Equation Modeling, 17, 642–653. https://doi.org/10.1080/10705511.2010.510064 First citation in articleCrossref, Google Scholar

  • Alicke, M. D., & Sedikides, C. (2009). Self-enhancement and self-protection: What they are and what they do. European Review of Social Psychology, 20, 1–48. First citation in articleCrossref, Google Scholar

  • Allen, N. J., & Meyer, J. P. (1990). The measurement and antecedents of affective, continuance and normative commitment to the organization. Journal of Occupational Psychology, 63, 1–18. First citation in articleCrossref, Google Scholar

  • Allport, G. W. (1937). Personality: A psychological interpretation. New York, NY: Holt. First citation in articleGoogle Scholar

  • Barnette, J. J. (2000). Effects of stem and Likert response option reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively worded stems. Educational and Psychological Measurement, 60, 361–370. First citation in articleCrossref, Google Scholar

  • Bachman, J. G., & O’Malley, P. M. (1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response styles. Public Opinion Quarterly, 48, 491–509. First citation in articleCrossref, Google Scholar

  • Bäckstrӧm, M., Bjӧrklund, F., & Larsson, M. R. (2009). Five-factor inventories have a major general factor related to social desirability which can be reduced by framing items neutrally. Journal of Research in Personality, 43, 335–344. First citation in articleCrossref, Google Scholar

  • *Bayazit, M., Hammer, T. H., & Wazeter, D. L. (2004). Methodological challenges in union commitment studies. Journal of Applied Psychology, 89, 738–747. First citation in articleCrossref, Google Scholar

  • Bergami, M., & Bagozzi, R. P. (2000). Self-categorization, affective commitment, and group self-esteem as distinct aspects of social identity in the organization. British Journal of Social Psychology, 39, 555–577. First citation in articleCrossref, Google Scholar

  • Bjelland, I., Dahl, A. A., Haug, T. T., & Neckelmann, D. (2001). The validity of the Hospital Anxiety and Depression Scale an updated literature review. Journal of Psychosomatic Research, 52, 69–77. First citation in articleCrossref, Google Scholar

  • Blake, D. D., Weathers, F. W., Nagy, L. M., Kaloupek, D. G., Gusman, F. D., Charney, D. S., & Keane, T. M. (1995). The development of a clinician‐administered PTSD scale. Journal of traumatic stress, 8, 75–90. First citation in articleCrossref, Google Scholar

  • Borkenau, P., & Ostendorf, F. (1989). Descriptive consistency and social desirability in self and peer reports. European Journal of Personality, 3, 31–45. First citation in articleCrossref, Google Scholar

  • Boyce, A. S., Conway, J. S., & Caputo, P. M. (2014). Development and validation of Aon Hewitt’s personality model and Adaptive Employee Personality Test (ADEPT-15). New York, NY: Aon Hewitt. First citation in articleGoogle Scholar

  • Brown, T. A. (2003). Confirmatory factor analysis of the Penn State Worry Questionnaire: Multiple factors or method effects? Behaviour Research and Therapy, 41, 1411–1426. https://doi.org/10.1016/S0005-7967(03)00059-7 First citation in articleCrossref, Google Scholar

  • Brown, T. A., Antony, M. M., & Barlow, D. H. (1992). Psychometric properties of the Penn State Worry Questionnaire in a clinical anxiety disorders sample. Behaviour Research and Therapy, 30, 33–37. First citation in articleCrossref, Google Scholar

  • Burns, G. N., Fillipowski, J. N., Morris, M. B., & Shoda, E. A. (2015). Impact of electronic warnings on online personality scores and test-taker reactions in an applicant simulation. Computers in Human Behavior, 48, 163–172. First citation in articleCrossref, Google Scholar

  • *Carleton, R. N., Thibodeau, M. A., Teale, M. J. N., Welch, P. G., Abrams, M. P., Robinson, T., & Asmundson, G. J. G. (2013). The Center for Epidemiologic Studies Depression Scale: A review with a theoretical and empirical examination of item content and factor structure. PLoS One, 8, e58067 First citation in articleCrossref, Google Scholar

  • Castillo, C., Macrini, L., Cheniaux, E., & Landeira-Fernandez, J. (2010). Psychometric properties and latent structure of the Portuguese version of the Penn State Worry Questionnaire. The Spanish Journal of Psychology, 13, 431–443. https://doi.org/10.1017/S113874160000398X First citation in articleCrossref, Google Scholar

  • Coleman, C. M. (2013). Effects of negative keying and wording in attitude measures: A mixed-methods study (Unpublished doctoral dissertation). James Madison University, Harrisonburg, VA. First citation in articleGoogle Scholar

  • Conrad, K. J., Wright, B. D., McKnight, P., McFall, M., Fontana, A., & Rosenheck, R. (2004). Comparing traditional and Rasch analyses of the Mississippi PTSD Scale: Revealing limitations of reverse-scored items. Journal of Applied Measurement, 5, 15–30. First citation in articleGoogle Scholar

  • Cook, J., & Wall, T. (1980). New work attitude measures of trust, organizational commitment and personal need non-fulfilment. Journal of Occupational Psychology, 53, 39–52. First citation in articleCrossref, Google Scholar

  • DiStefano, C., & Motl, R. W. (2009). Personality correlates of method effects due to negatively worded items on the Rosenberg Self-Esteem scale. Personality and Individual Differences, 46, 309–313. First citation in articleCrossref, Google Scholar

  • Dunlop, P. D., Telford, A. D., & Morrison, D. L. (2012). Not too little, but not too much: The perceived desirability of responses to personality items. Journal of Research in Personality, 46, 8–18. First citation in articleCrossref, Google Scholar

  • Edwards, A. L. (1953). The relationship between the judged desirability of a trait and the probability that the trait will be endorsed. Journal of Applied Psychology, 37, 90–93. First citation in articleCrossref, Google Scholar

  • Edwards, A. L. (1957). The social desirability variable in personality assessment and research. New York, NY: Dryden. First citation in articleGoogle Scholar

  • Freud, S. (1961). Instincts and their vicissitudes. In J. StracheyEd., The standard edition of the complete psychological works of Sigmund Freud (Vol. 14, pp. 111–142). London, UK: Hogarth Press. (Original work published 1915). First citation in articleGoogle Scholar

  • Gana, K., Saada, Y., Bailly, N., Joulain, M., Hervé, C., & Alaphilippe, D. (2009). Longitudinal factorial invariance of the Rosenberg Self-Esteem Scale: Determining the nature of method effects due to item wording. Journal of Research in Personality, 47, 406–416. https://doi.org/10.1016/j.jrp.2013.03.011 First citation in articleCrossref, Google Scholar

  • Gordon, M. E., Philpot, J. W., Burt, R. E., Thompson, C. A., & Spiller, W. E. (1980). Commitment to the union: Development of a measure and an examination of its correlates. Journal of Applied Psychology, 65, 479–499. First citation in articleCrossref, Google Scholar

  • Greenberger, E., Chen, C., Dmitrieva, J., & Farruggia, S. P. (2003). Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: Do they matter? Personality and Individual Differences, 35, 1241–1254. https://doi.org/10.1016/S0191-8869(02)00331-8 First citation in articleCrossref, Google Scholar

  • *Hammer, T. H., Bayazit, M., & Wazeter, D. L. (2009). Union leadership and member attitudes: A multi-level analysis. Journal of Applied Psychology, 94, 392–410. First citation in articleCrossref, Google Scholar

  • Herzog, W., & Boomsma, A. (2009). Small-sample robust estimators of noncentrality-based and incremental model fit. Structural Equation Modeling, 61, 1–27. First citation in articleCrossref, Google Scholar

  • Huebner, S. (2001). Manual for the Multidimensional Students’ Life Satisfaction Scale. Columbia, SC: University of South Carolina. First citation in articleGoogle Scholar

  • Keane, T. M., Caddell, J. M., & Taylor, K. L. (1988). Mississippi Scale for Combat-Related Posttraumatic Stress Disorder: Three studies in reliability and validity. Journal of Consulting and Clinical Psychology, 56, 85–90. First citation in articleCrossref, Google Scholar

  • Kuncel, N. R., & Tellegen, A. (2009). A conceptual and empirical reexamination of the measurement of the social desirability of items: Implications for detecting desirable response style and scale development. Personnel Psychology, 62, 201–228. First citation in articleCrossref, Google Scholar

  • Lindwall, M., Barkoukis, V., Grano, C., Lucidi, F., Raudsepp, L., Liukkonen, J., & Thøgersen-Ntoumani, C. (2012). Method effects: The problem with negatively versus positively keyed items. Journal of Personality Assessment, 94, 196–204. https://doi.org/10.1080/00223891.2011.645936 First citation in articleCrossref, Google Scholar

  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Magazine, S. L., Williams, L. J., & Williams, M. L. (1996). A confirmatory factor analysis examination of reverse coding effects in Meyer and Allen’s Affective and Continuance Commitment Scales. Educational and Psychological Measurement, 56, 241–250. First citation in articleCrossref, Google Scholar

  • Mathews, B. P., & Shepherd, J. L. (2002). Dimensionality of Cook and Wall’s (1980) British Organizational Commitment Scale revisited. Journal of Occupational and Organizational Psychology, 75, 369–375. https://doi.org/10.1348/096317902320369767 First citation in articleCrossref, Google Scholar

  • McCrae, R., & Costa, P. (1983). Social desirability scales: More substance than style. Journal of Consulting and Clinical Psychology, 51, 882–888. First citation in articleCrossref, Google Scholar

  • McPherson, J., & Mohr, P. (2005). The role of item extremity in the emergence of keying-related factors: an exploration with the life orientation test. Psychological methods, 10, 120–131. https://doi.org/10.1037/1082-989X.10.1.120 First citation in articleCrossref, Google Scholar

  • Merritt, S. M. (2012). The two-factor solution to Allen and Meyer’s (1990) affective commitment scale: Effects of negatively worded items. Journal of Business and Psychology, 27, 421–436. First citation in articleCrossref, Google Scholar

  • Meyer, T. J., Miller, M. L., Metzger, R. L., & Borkovec, T. D. (1990). Development and validation of the Penn state worry questionnaire. Behaviour Research and Therapy, 28, 487–495. First citation in articleCrossref, Google Scholar

  • Morgeson, F. P., & Humphrey, S. E. (2006). The Work Design Questionnaire (WDQ): Developing and validating a comprehensive measure for assessing job design and the nature of work. Journal of Applied Psychology, 91, 1321. First citation in articleCrossref, Google Scholar

  • Mowday, R. T. (1999). Reflections on the study and relevance of organizational commitment. Human Resource Management Review, 8, 387–401. First citation in articleCrossref, Google Scholar

  • Mowday, R. T., Steers, R. M., & Porter, L. W. (1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14, 224–247. First citation in articleCrossref, Google Scholar

  • Mullen, S. P., Gothe, N. P., & McAuley, E. (2013). Evaluation of the factor structure of the Rosenberg Self-esteem Scale in older adults. Personality and Individual Differences, 54, 153–157. https://doi.org/10.1016/j.paid.2012.08.009 First citation in articleCrossref, Google Scholar

  • Palmer, B., Gignac, G., Bates, T., & Stough, C. (2003). Examining the structure of the trait meta-mood scale. Australian Journal of Psychology, 55, 154–158. First citation in articleCrossref, Google Scholar

  • Paulhus, D. L., & Buckels, E. (2012). Classic self-deception revisited. In S. StracheyT. D. WilsonEds., Handbook of self-knowledge (pp. 363–378). New York, NY: The Guilford Press. First citation in articleGoogle Scholar

  • Peabody, D. (1967). Trait inferences: Evaluative and descriptive aspects. Journal of Personality and Social Psychology, 7, 1–18. https://doi.org/10.1037/h0025230 First citation in articleCrossref, Google Scholar

  • Porter, L. W., Steers, R. M., Mowday, R. T., & Boulian, P. V. (1974). Organizational commitment, job satisfaction, and turnover among psychiatric technicians. Journal of Applied Psychology, 59, 603–609. First citation in articleCrossref, Google Scholar

  • Prentice, R. L. (1976). A generalization of the probit and logit methods for dose response curves. Biometrics, 32, 761–768. First citation in articleCrossref, Google Scholar

  • R Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https//www.R-project.org/ First citation in articleGoogle Scholar

  • Radloff, L. S. (1977). The CES-D Scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385–401. First citation in articleCrossref, Google Scholar

  • Richins, M. L., & Dawson, S. (1992). A consumer values orientation for materialism and its measurement: Scale development and validation. Journal of Consumer Research, 19, 303. First citation in articleCrossref, Google Scholar

  • Rosenberg, M. (1965). Rosenberg self-esteem scale (RSE). Acceptance and commitment therapy. Measures package, 61, 52. First citation in articleGoogle Scholar

  • Rosenberg, M. (1979). Conceiving the self. New York, NY: Basic Books. First citation in articleGoogle Scholar

  • Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1–36. First citation in articleCrossref, Google Scholar

  • Salovey, P., & Mayer, J. D. (1990). Emotional intelligence. Imagination, cognition and personality, 9, 185–211. First citation in articleCrossref, Google Scholar

  • Salovey, P., Mayer, J. D., Goldman, S. L., Turvey, C., & Palfai, T. (1995). Emotional attention, clarity, and repair: Exploring emotional intelligence using the Trait Meta-Mood scale. Emotion, Disclosure, and Health, 125, 125–154. First citation in articleCrossref, Google Scholar

  • Sawatzky, R., Ratner, P. A., Johnson, J. L., Kopec, J. A., & Zumbo, B. D. (2009). Sample heterogeneity and the measurement structure of the Multidimensional Students’ Life Satisfaction Scale. Social Indicators Research, 94, 273–296. https://doi.org/10.1007%2Fs11205-008-9423-4 First citation in articleCrossref, Google Scholar

  • Scheier, M. F., Carver, C. S., & Bridges, M. W. (1994). Distinguishing optimism from neuroticism (and trait anxiety, self-mastery, and self-esteem): A re-evaluation of the Life Orientation Test. Journal of Personality and Social Psychology, 67, 1063–1078. First citation in articleCrossref, Google Scholar

  • Schriesheim, C. A., & Eisenbach, R. J. (1995). An exploratory and confirmatory factor-analytic investigation of item wording effects on the obtained factor structures of survey questionnaire measures. Journal of Management, 21, 1177–1193. First citation in articleCrossref, Google Scholar

  • Schmitt, N., & Stults, D. M. (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9, 367–373. First citation in articleCrossref, Google Scholar

  • Schriesheim, C. A., & Hill, K. D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41, 1101–1114. First citation in articleCrossref, Google Scholar

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. First citation in articleCrossref, Google Scholar

  • Smith, D. B., & Ellingson, J. E. (2002). Substance versus style: A new look at social desirability in motivating contexts. Journal of Applied Psychology, 87, 211–219. First citation in articleCrossref, Google Scholar

  • Spielberger, C. D., Jacobs, G., Russell, S., & Crane, R. S. (1983). Assessment of anger: The state-trait anger scale. Advances in Personality Assessment, 2, 159–187. First citation in articleGoogle Scholar

  • Stogdill, R. M. (1963). Manual for the Leader Behavior Description Questionnaire-Form XII: An experimental revision. Columbus, OH: Bureau of Business Research, College of Commerce and Administration, Ohio State University. First citation in articleGoogle Scholar

  • *Stride, C., Wall, T. D., & Catley, N. (2007). Measures of job satisfaction, organizational commitment, mental health and job-related well-being: A benchmarking manual (2nd ed.). West Sussex, UK: Wiley. First citation in articleGoogle Scholar

  • Taylor, S. E., & Brown, J. D. (1988). Illusion and well-being: A social psychological perspective on mental health. Psychological Bulletin, 103, 193–210. First citation in articleCrossref, Google Scholar

  • Tomás, J. M., Oliver, A., Galiana, L., Sancho, P., & Lila, M. (2013). Explaining method effects associated with negatively worded items in trait and state global and domain-specific self-esteem scales. Structural Equation Modeling, 20, 299–313. https://doi.org/10.1080/10705511.2013.769394 First citation in articleCrossref, Google Scholar

  • Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. B. JabineM. L. StrafJ. M. TanurR. TourangeauEds., Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press. First citation in articleGoogle Scholar

  • Tourangeau, R., & Rasinski, K. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299–314. First citation in articleCrossref, Google Scholar

  • Vautier, S., & Pohl, S. (2009). Do balanced scales assess bipolar constructs? The case of the STAI scales. Psychological Assessment, 21, 187–193. https://doi.org/10.1037/a0015312 First citation in articleCrossref, Google Scholar

  • Weijters, B., & Baumgartner, H. (2012). Misresponse to reversed and negated items in surveys: A review. JMR, Journal of Marketing Research, 49, 737–747. First citation in articleCrossref, Google Scholar

  • Weijters, B., Baumgartner, H., & Schillewaert, N. (2013). Reversed item bias: An integrative model. Psychological Methods, 18, 320–334. First citation in articleCrossref, Google Scholar

  • Wong, N., Rindfleisch, A., & Burroughs, J. E. (2003). Do reverse-worded items confound measures in cross-cultural consumer research? The case of the Material Values Scale. Journal of Consumer Research, 30, 72–91. https://doi.org/10.1086/374697 First citation in articleCrossref, Google Scholar

  • Wouters, E., Le Roux Booysen, F., Ponnet, K., & Baron Van Loon, F. (2012). Wording effects and the factor structure of the Hospital Anxiety & Depression Scale in HIV/AIDS patients on antiretroviral treatment in South Africa. PLoS One, 7, e34881. https://doi.org/10.1371/journal.pone.0034881 First citation in articleCrossref, Google Scholar

  • Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12, 1100–1122. First citation in articleCrossref, Google Scholar

  • Ziegler, M. (2011). Applicant faking: A look into the black box. The Industrial and Organizational Psychologist, 49, 29–36. First citation in articleGoogle Scholar

  • Zigmond, A. S., & Snaith, R. P. (1983). The Hospital Anxiety and Depression Scale. Acta Psychiatrica Scandinavica, 67, 361–370. First citation in articleCrossref, Google Scholar