Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

The Occupational Performance Assessment–Response Distortion (OPerA-RD) Scale

Measuring Bias in Performance Evaluation

Published Online:March 25, 2022https://doi.org/10.1027/1866-5888/a000301

Abstract

Abstract. The ubiquity and consequences of job performance evaluations necessitate accurate responding. This paper describes two studies designed to develop (Study 1) and provide initial validation (Study 2) for a new measure specifically designed to assist in this context: the Occupational Performance Assessment–Response Distortion (OPerA-RD) scale. This 20-item scale is contextualized to the workplace and was developed by identifying items that could detect over- and under-reporting of job performance by self- or other-report in four independent faking samples. Initial validation of the OPerA-RD was supported by expected differences between within-group faking and control conditions in subsequent samples, specifically over- and under-reporting of job performance by self- or other-reports. Implications for research and applied settings are discussed.

References

  • Aguinis, H., Villamor I., & Ramani, R. S. (2021). MTurk research: Review and recommendations. Journal of Management, 47(4), 823–837. 10.1177/0149206320969787 First citation in articleCrossref, Google Scholar

  • Ben-Porath, Y. S., & Tellegen, A. (2008). Minnesota multiphasic personality inventory–2 restructured form; Manual for administration, scoring, and interpretation. University of Minnesota Press. First citation in articleGoogle Scholar

  • Berry, C. M., Carpenter, N. C., & Barratt, C. L. (2012). Do other-reports of counterproductive work behavior provide an incremental contribution over self-reports? A meta-analytic comparison. Journal of Applied Psychology, 97(3), 613–636. 10.1037/a0026739 First citation in articleCrossref, Google Scholar

  • Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. T., & Smith, M. A. (2006). A meta-analytic investigation of job applicant faking on personality measures. International Journal of Selection and Assessment, 14(4), 317–335. 10.1111/j.1468-2389.2006.00354.x First citation in articleCrossref, Google Scholar

  • Blanch, A., Aluja, A., Gallart, S., & Dolcet, J.-M. (2009). A review on the use of NEO-PI-R validity scales in normative, job selection, and clinical samples. European Journal of Psychiatry, 23(2), 121–129. 10.4321/s0213-61632009000200006 First citation in articleCrossref, Google Scholar

  • Blasberg, S. A., Rogers, K. H., & Paulhus, D. L. (2013). The Bidimensional Impression Management Index (BIMI): Measuring agentic and communal forms of impression management. Journal of Personality Assessment, 96(5), 523–531. 10.1080/00223891.2013.862252 First citation in articleCrossref, Google Scholar

  • Bolino, M., Long, D., & Turnley, W. (2016). Impression management in organizations: Critical questions, answers, and areas for future research. Annual Review of Organizational Psychology and Organizational Behavior, 3(1), 377–406. 10.1146/annurev-orgpsych-041015-062337 First citation in articleCrossref, Google Scholar

  • Bommer, W. H., Johnson, J. L., Rich, G. A., Podsakoff, P. M., & Mackenzie, S. B. (1995). On the interchangeability of objective and subjective measures of employee performance: A meta-analysis. Personnel Psychology, 48(3), 587–605. 10.1111/j.1744-6570.1995.tb01772.x First citation in articleCrossref, Google Scholar

  • Campbell, J. P. (2012). Behavior, performance, and effectiveness in the twenty-first century. In S. Kozlowski (Ed.), The Oxford handbook of organizational psychology, (Vol. 1, pp. 159–194). Oxford University Press. First citation in articleCrossref, Google Scholar

  • Campbell, J. P., & Wiernik, B. M. (2015). The modeling and assessment of work performance. Annual Review of Organizational Psychology and Organizational Behavior, 2(1), 47–74. 10.1146/annurev-orgpsych-032414-111427 First citation in articleCrossref, Google Scholar

  • Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017). Amazon Mechanical Turk in organizational psychology: An evaluation and practical recommendations. Journal of Business Psychology, 32(4), 347–361. 10.1007/s10869-016-9458-5 First citation in articleCrossref, Google Scholar

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum. First citation in articleGoogle Scholar

  • Connelly, B. S., & Ones, D. S. (2010). An other perspective on personality: Meta-analytic integration of observers' accuracy and predictive validity. Psychological Bulletin, 136(6), 1092–1122. 10.1037/a0021212 First citation in articleCrossref, Google Scholar

  • Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24(4), 349–354. 10.1037/h0047358 First citation in articleCrossref, Google Scholar

  • Detrick, P., & Chibnall, J. T. (2008). Positive response distortion by police officer applicants: Association of Paulhus deception scales with MMPI-2 and inwald personality inventory validity scales. Assessment, 15(1), 87–96. 10.1177/1073191107306082 First citation in articleCrossref, Google Scholar

  • Donovan, J. J., Dwight, S. A., & Hurtz, G. M. (2003). An assessment of the prevalence, severity, and verifiability of entry-level applicant faking using the randomized response technique. Human Performance, 16(1), 81–106. 10.1207/s15327043hup1601_4 First citation in articleCrossref, Google Scholar

  • Goffin, R. D., & Christiansen, N. D. (2003). Correcting personality tests for faking: A review of popular personality tests and an initial survey of researchers. International Journal of Selection and Assessment, 11(4), 340–344. 10.1111/j.0965-075x.2003.00256.x First citation in articleCrossref, Google Scholar

  • Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93–104. 10.1037/0003-066x.59.2.93 First citation in articleCrossref, Google Scholar

  • Goulet-Pelletier, J.-C., & Cousineau, D. (2018). A review of effect sizes and their confidence intervals, Part I: The Cohen's d family. The Quantitative Methods for Psychology, 14(4), 242–265. 10.20982/tqmp.14.4.p242 First citation in articleCrossref, Google Scholar

  • Heggestad, E. D., & Gordon, H. L. (2008). An argument for context-specific personality assessments. Industrial and Organizational Psychology, 1(3), 320–322. 10.1111/j.1754-9434.2008.00056.x First citation in articleCrossref, Google Scholar

  • Heggestad, E. D., Scheaf, D. J., Banks, G. C., Monroe Hausfeld, M., Tonidandel, S., & Williams, E. B. (2019). Scale adaptation in organizational science research: A review and best-practice recommendations. Journal of Management, 45(6), 2596–2627. 10.1177/0149206319850280 First citation in articleCrossref, Google Scholar

  • Kirkpatrick, D. L. (1996). Great ideas revisited: Techniques for evaluating training programs – Revisiting Kirkpatrick’s four level model. Training and Development, 50, 54–59. First citation in articleGoogle Scholar

  • Kline, R. B. (2013). Beyond significance testing: Statistics reform in the behavioral sciences (2nd ed.). American Psychological Association. First citation in articleCrossref, Google Scholar

  • Knouse, S. B. (1989). Impression management and the letter of recommendation. In R. A. GiacaloneP. Rosenfeld (Eds.), Impression management in the organization (pp. 283–296). Erlbaum. First citation in articleGoogle Scholar

  • Koehn, M. A., Okan, C., & Jonason, P. K. (2018). A primer on the dark triad traits. Australian Journal of Psychology, 71(1), 7–15. 10.1111/ajpy.12198 First citation in articleCrossref, Google Scholar

  • Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, mechanical turk, and other convenience samples. Industrial and Organizational Psychology, 8(2), 142–164. 10.1017/iop.2015.13 First citation in articleCrossref, Google Scholar

  • LeBreton, J. M., Shiverdecker, L. K., & Grimaldi, E. M. (2018). The dark triad and workplace behavior. Annual Review of Organizational Psychology and Organizational Behavior, 5(1), 387–414. 10.1146/annurev-orgpsych-032117-104451 First citation in articleCrossref, Google Scholar

  • Li, A., & Bagger, J. (2006). Using the BIDR to distinguish the effects of impression management and self-deception on the criterion validity of personality measures: A meta-analysis. International Journal of Selection and Assessment, 14(2), 131–141. 10.1111/j.1468-2389.2006.00339.x First citation in articleCrossref, Google Scholar

  • Marcus, B. (2009). ‘Faking’ from the applicant’s perspective: A theory of self-presentation in personnel selection settings. International Journal of Selection and Assessment, 17(4), 417–430. 10.1111/j.1468-2389.2009.00483.x First citation in articleCrossref, Google Scholar

  • McGill, D. A., van der Vleuten, C. P. M., & Clarke, M. J. (2011). Supervisor assessment of clinical and professional competence of medical trainees: A reliability study using workplace data and a focused analytical literature review. Advances in Health Sciences Education, 16(3), 405–425. 10.1007/s10459-011-9296-1 First citation in articleCrossref, Google Scholar

  • McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136(3), 450–470. 10.1037/a0019216 First citation in articleCrossref, Google Scholar

  • Merriam-Webster. (n.d.). Opus. In Merriam-Webster.com dictionary. Retrieved May 6, 2020. https://www.merriam-webster.com/dictionary/opus First citation in articleGoogle Scholar

  • Murphy, K. R. (2008). Explaining the weak relationship between job performance and ratings of job performance. Industrial and Organizational Psychology, 1(2), 148–160. 10.1111/j.1754-9434.2008.00030.x First citation in articleCrossref, Google Scholar

  • Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Sage. First citation in articleGoogle Scholar

  • O’Boyle, E. H. Jr., Forsyth, D. R., Banks, G. C., & McDaniel, M. A. (2012). A meta-analysis of the dark triad and work behavior: A social exchange perspective. Journal of Applied Psychology, 97(3), 557–579. 10.1037/a0025679 First citation in articleCrossref, Google Scholar

  • Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81(6), 660–679. 10.1037/0021-9010.81.6.660 First citation in articleCrossref, Google Scholar

  • Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. RobinsonP. ShaverL. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). Academic Press. 10.1016/b978-0-12-590241-0.50006-x First citation in articleCrossref, Google Scholar

  • Paulhus, D. L. (1998). Manual for the Balanced Inventory of Desirable Responding (BIDR-7). MHS. First citation in articleGoogle Scholar

  • Paulhus, D. L. (2002). Socially desirable responding: The evolution of a construct. In H. I. BraunD. N. JacksonD. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (pp. 49–69). Erlbaum. First citation in articleGoogle Scholar

  • Peck, J. A., & Levashina, J. (2017). Impression management and interview and job performance ratings: A meta-analysis of research design with tactics in mind. Frontiers in Psychology, 8, 201. 10.3389/fpsyg.2017.00201 First citation in articleCrossref, Google Scholar

  • Pransky, G., Finkelstein, S., Berndt, E., Kyle, M., Mackell, J., & Tortorice, D. (2006). Objective and self‐report work performance measures: A comparative analysis. Industrial Journal of Productivity and Performance Management, 55(5), 390–399. 10.1108/17410400610671426 First citation in articleCrossref, Google Scholar

  • Rothstein, M. G., & Goffin, R. D. (2006). The use of personality measures in personnel selection: What does current research support? Human Resource Management Review, 16(2), 155–180. 10.1016/j.hrmr.2006.03.004 First citation in articleCrossref, Google Scholar

  • Shaffer, J. A., & Postlethwaite, B. E. (2012). A matter of context: A meta-analytic investigation of the relative validity of contextualized and noncontextualized personality measures. Personnel Psychology, 65(3), 445–494. 10.1111/j.1744-6570.2012.01250.x First citation in articleCrossref, Google Scholar

  • Uziel, L. (2010). Rethinking social desirability scales: From impression management to interpersonally oriented self-control. Perspectives on Psychological Science, 5(3), 243–262. 10.1177/1745691610369465 First citation in articleCrossref, Google Scholar

  • Williams, K. M., Sitarenios, G., Rzepa, S., & Wheldon, H. (2013). Clinical testing case studies: Prevention, detection, and management of falsified test development data. In J. A. WollackJ. J. Fremer (Eds.), Handbook of test security (pp. 285–297). Routledge. First citation in articleGoogle Scholar

  • Woo, S. E., Keith, M., & Thornton, M. A. (2015). Amazon Mechanical Turk for industrial and organizational psychology: Advantages, challenges, and practical recommendations. Industrial and Organizational Psychology, 8(2), 171–179. 10.1017/iop.2015.21 First citation in articleCrossref, Google Scholar