Incentives and Alternative Rating Approaches
Roads to Greater Accuracy in Job Performance Assessment?
Abstract
Ratings of others’ performance are central in applied psychology. We investigated rater incentives and two approaches to Behavior Observation Scale (BOS) rating on rating accuracy. Raters (N = 147) were randomly assigned to one of three accuracy-incentive conditions and completed one of two BOSs 48 hr after observing videotaped performances. A serial BOS asked raters to assess one ratee at a time, across behaviors. A parallel BOS had raters consider all ratees on one behavior at a time. The serial (one ratee at a time) approach was generally more accurate than the parallel (one behavior at a time) approach. However, an accuracy incentive prior to observation mitigated the negative effects of the parallel approach. Overall, a serial BOS seems well suited for developmental appraisal.
References
(1983). Is memory schematic? Psychological Bulletin, 93, 203–231.
(1988). The impact of rating scale format on rater accuracy: An evaluation of the mixed standard scale. Journal of Management, 14, 415–423.
(1995). On the interchangeability of objective and subjective measures of employee performance: A meta-analysis. Personnel Psychology, 48, 587–605.
(1977). Consistency of rating accuracy and rating errors in the judgment of human performance. Organizational Behavior and Human Performance, 20, 238–252.
(2002). Effects of procedure and outcome accountability on interview validity. Journal of Applied Psychology, 87, 185–191.
(2007). Social comparison: The end of a theory and the emergence of a field. Organizational Behavior and Human Decision Processes, 102, 3–21.
(1994). Performance appraisal: Alternative perspectives. Cincinnati, OH: South-Western.
(1955). Processes affecting scores on “understanding of others” and “assumed similarity”. Psychological Bulletin, 52, 177–193.
(1996). A cognitive approach to performance appraisal: A program of research. New York, NY: Routledge.
(2000). Design performance appraisal systems to improve performance. In , Blackwell Handbook of Principles of Organizational Behavior (pp. 60–72). Oxford, UK: Blackwell.
(1996). Organization of information in memory and the performance appraisal process: Evidence from the field. Journal of Applied Psychology, 81, 717–737.
(1989). Organization of information used for performance appraisals: Role of diary-keeping. Journal of Applied Psychology, 74, 124–129.
(2009). Forced rankings: Pros, cons, and practices. In , Performance management: Putting research into action (pp. 411–443). San Francisco, CA: Jossey-Bass.
(2006). Trust us: New perspectives on performance appraisal. In , Performance measurement: Current perspectives and future challenges (pp. 321–337). Mahwah, NJ: Erlbaum.
(2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
(1996). Criterion validation of two approaches to performance appraisal: The behavioral observation scale and the relative percentile method. Journal of Business and Psychology, 11, 23–33.
(2009). Taking advantage of social comparisons in performance appraisal: The Relative Percentile Method. Human Resource Management, 48, 251–268.
(2011). Is it all relative? Comparative judgments and the possible improvement of self-ratings and ratings of others. Perspectives in Psychological Science, 6, 48–60.
(2008). Inaccurate performance ratings are a reflection of larger organizational issues. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 190–193.
(1986). The relationship between memory and judgment depends on whether the judgment task is memory-based or on-line. Psychological Review, 93, 258–268.
(1986). The relationship between supervisory ratings and results-oriented measures of performance: A meta-analysis. Personnel Psychology, 39, 811–826.
(2001). Can performance-feedback accuracy be improved? Effects of rater priming and rating-scale format on rating accuracy. Journal of Applied Psychology, 86, 134–144.
(1998). Are financial incentives related to performance? A meta-analytic review of empirical research. Journal of Applied Psychology, 83, 777–787.
(2006). Performance measurement reconsidered: An examination of frequency estimation as a basis for performance assessment. In , Performance measurement: Current perspectives and future challenges (pp. 77–110). Mahwah, NJ: Erlbaum.
(1980). Performance rating. Psychological Bulletin, 87, 72–107.
(1977). Behavioral observation scales for performance appraisal purposes. Personnel Psychology, 30, 255–268.
(1994). Increasing productivity through performance appraisal (2nd ed.). Reading, MA: Addison-Wesley.
(1991). Cognitive theory in industrial and organizational psychology. In , Handbook of industrial and organizational psychology (2nd ed., Vol. 2, pp. 1–62). Palo Alto, CA: Consulting Psychologists Press.
(1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.
(2007). Do within-dimension ratings in assessment centers really lead to improved construct validity? A meta-analytic reassessment. Zeitschrift für Personalpsychologie, 6, 141–149.
(1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
(1991). Effective teaching behaviors in the college classroom. In , Higher education: Handbook of theory and research (Vol. 7, pp. 135–172). New York, NY: Agathon.
(1988). A comparison of criteria for test validation: A meta-analytic investigation. Personnel Psychology, 41, 517–535.
(2007). Absolute vs relative performance rating formats: Implications for fairness and organizational justice. International Journal of Selection and Assessment, 15, 302–316.
(1993). The influence of rater motivation on assimilation effects and accuracy in performance ratings. Organizational Behavior and Human Decision Processes, 55, 41–60.
(1996). A second look at the relationship between rating and behavioral accuracy in performance appraisal. Journal of Applied Psychology, 81, 3–10.
(1989). An investigation of the validity of expert true score estimates in appraisal research. Journal of Applied Psychology, 74, 143–151.
(1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497–506.
(1997). Differences in accuracy of absolute and comparative performance appraisal methods. Organizational Behavior and Human Decision Processes, 70, 95–103.
(1986). The practicality of behavioral observation scales, behavioral expectation scales, and trait scales. Personnel Psychology, 39, 619–628.
(1990). The effect of performance appraisal salience on recall and ratings. Organizational Behavior and Human Decision Processes, 46, 217–239.
(1993). Processing objective and question order effects on the causal relation between memory and judgment in performance appraisal: The tip of the iceberg. Journal of Applied Psychology, 78, 232–241.
