Abstract
Summary: This paper presents a meta-analysis of published job analysis interrater reliability data in order to predict the expected levels of interrater reliability within specific combinations of moderators, such as rater source, experience of the rater, and type of job descriptive information. The overall mean interrater reliability of 91 reliability coefficients reported in the literature was .59. The results of experienced professionals (job analysts) showed the highest reliability coefficients (.76). The method of data collection (job contact versus job description) only affected the results of experienced job analysts. For this group higher interrater reliability coefficients were obtained for analyses based on job contact (.87) than for those based on job descriptions (.71). For other rater categories (e.g., students, organization members) neither the method of data collection nor training had a significant effect on the interrater reliability. Analyses based on scales with defined levels resulted in significantly higher interrater reliability coefficients than analyses based on scales with undefined levels. Behavior and job worth dimensions were rated more reliable (.62 and .60, respectively) than attributes and tasks (.49 and .29, respectively). Furthermore, the results indicated that if nonprofessional raters are used (e.g., incumbents or students), at least two to four raters are required to obtain a reliability coefficient of .80. These findings have implications for research and practice.
References
References marked with an asterisk indicate studies included in the meta-analysis.
References
Arvey, R.D. Davis, G.A. McGowen, S.L. Dipboye, R.L. (1982). Potential sources of bias in job analytic processes.. Academy of Management Journal, 25, 618– 629* Banks, M.H. Miller, R.L. (1984). Reliability and convergent validity of the Job Components Inventory.. Journal of Occupational Psychology, 57, 181– 184Borman, W.C. (1996). The Occupational Information Network: An updated dictionary of occupational titles.. Military Psychology, 8, 263– 265* Butler, S.K. Harvey, R.J. (1988). A comparison of holistic versus decomposed rating of Position Analysis Questionnaire work dimensions.. Personnel Psychology, 41, 761– 771* Cain, P.S. Green, B.F. (1983). Reliabilities of selected ratings available from the dictionary of occupational titles.. Journal of Applied Psychology, 68, 155– 165Cascio, W.F. (1995). Whither industrial and organizational psychology in a changing world of work?. American Psychologist, 50, 928– 939Cascio, W.F. (1998). Applied psychology in human resource management. . London, Prentice Hall.Cornelius, E.T. Carron, T.J. Collins, M.N. (1979). Job analysis models and job classification.. Personnel Psychology, 32, 693– 708* Cornelius, E.T. Lyness, K.S. (1980). A comparison of holistic and decomposed judgment strategies in job analyses by job incumbents.. Journal of Applied Psychology, 65, 155– 163* Cornelius, E.T. DeNisi, A.S. Blencoe, A.G. (1984a). Expert and naive raters using the PAQ: Does it matter?. Personnel Psychology, 37, 453– 464* Cornelius, E.T. Schmidt, F.L. Carron, T.J. (1984b). Job classification approaches and the implementation of validity generalization results.. Personnel Psychology, 37, 247– 260Cunningham, J.W. (1996). Ability requirement factors for general job elements.. Military Psychology, 8, 219– 234* Cunningham, J.W. Powell, T.E. Wimpee, W.E. Wilson, M.A. Ballentine, R.D. (1996). Ability-requirement factors for general job elements.. Military Psychology, 8, 219– 234* Davis, K.R. Sauser, W.I. (1991). Effects of alternative weighting methods in a policy-capturing approach to job evaluation: A review and empirical investigation.. Personnel Psychology, 44, 85– 127* DeNisi, A.S. Cornelius, E.T. Blencoe, A.G. (1987). Further investigation of common knowledge effects on job analysis ratings.. Journal of Applied Psychology, 72, 262– 268* Doverspike, D. Carlisi, A.M. Barrett, G.V. (1983). Generalizability analysis of a point-method job evaluation instrument.. Journal of Applied Psychology, 68, 476– 483* Doverspike, D. Barrett, G.V. (1984). An internal bias analysis of a job evaluation instrument.. Journal of Applied Psychology, 69, 648– 662* Earles, J.A. Driskill, W.E. Dittmar, M.J. (1996). Methodology for identifying abilities for job classification: An application of job analysis.. Military Psychology, 8, 179– 193Feldman, J.M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal.. Journal of Applied Psychology, 66, 127– 148Fiske, S.T. Neuberg, S.L. (1990). A continuum of impression formation, from category-based to individuating processes: Influences of information and motivation on attention and interpretation.. In M.P. Zanna (Ed.), Advances in experimental social psychology (Vol.2, pp.1-74). New York: Academic Press.* Fleishman, E.A. Mumford, M.D. (1988). Ability Requirement Scales.. In S. Gael (Ed.), The job analysis handbook for business, industry and government (pp.917-935). New York: Wiley.Fleishman, E.A. Mumford, M.D. (1991). Evaluating classifications of job behavior: A construct validation of the ability requirements scales.. Personnel Psychology, 44, 523– 575* Fraser, S.L. Cronshaw, S.F. Alexander, R.A. (1984). Generalizability analysis of a point method job evaluation instrument: A field study.. Journal of Applied Psychology, 69, 643– 647* Friedman, L. Harvey, R.J. (1986). Can raters with reduced job descriptive information provide accurate Position Analysis Questionnaire (PAQ) ratings?. Personnel Psychology, 39, 779– 789* Frieling, E. Kannheiser, W. Lindberg, R. (1974). Some results with the German form of the Position Analysis Questionnaire.. Journal of Applied Psychology, 59, 741– 747Goldstein, I.L. Zedeck, S. Schneider, B. (1993). An exploration of the job analysis-content validity process.. In Schmitt, N., & Borman, W.C. (Eds.), Personnel selection in organizations (pp.3-34). San Francisco: Jossey-Bass.Gordon, M.E. Slade, L.A. Schmitt, N. (1986). The “science of the sophomore” revisited: From conjecture to empiricism.. Academy of Management Review, 11, 191– 207* Geyer, P.D. Hice, J. Hawk, J. Boese, R. Brannon, Y. (1989). Reliabilities of ratings available from the Dictionary of Occupational Titles.. Personnel Psychology, 42, 547– 560* Gomez-Mejia, L.R. Page, R.C. Tornow, W.W. (1982). A comparison of the practical utility of traditional, statistical and hybrid job evaluation approaches.. Academy of Management Journal, 25, 790– 809Green, S.B. Stutzman, T. (1986). An evaluation of methods to select respondents to structured job analysis questionnaires.. Personnel Psychology, 39, 543– 564Guion, R.M. (1993). The need for change: Six persistent themes.. In Schmitt, N., & Borman, W.C. (Eds.), Personnel selection in organizations (pp.481-496). San Francisco: Jossey-Bass.* Hahn, D.C. Dipboye, R.L. (1988). Effects of training and information on the accuracy and reliability of job evaluations.. Journal of Applied Psychology, 73, 146– 153Harvey, R.J. (1991). Job analysis.. In M.D. Dunette, & L.M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol.2, pp.71-162). Palo Alto, CA: Consulting Psychologists Press.Harvey, R.J. Hayes, T.L. (1986). Monte Carlo baselines for interrater reliability correlations using the Position Analysis Questionnaire.. Personnel Psychology, 39, 345– 357* Harvey, R.J. Lozada-Larsen, S.R. (1988). Influence of amount of job descriptive information on job analysis rating accuracy.. Journal of Applied Psychology, 73, 457– 461Harvey, T.L. Hayes, R.J. (1986). Monte Carlo baselines for interrater reliability correlations using the Position Analysis Questionnaire.. Personnel Psychology, 39, 345– 357Hayes, T.L. (1988). Do demographic and experience characteristics of incumbents do predict job analysis ratings?. In R.J. Harvey (Chair), Troublesome questions in job analysis. Symposium presented at the annual conference of the Society for Industrial and Organizational Psychology, Dallas.Hedges, L.V. Olkin, I. (1985). Statistical methods for meta-analysis. . Orlando, FL: Academic Press.Hogan, R.T. (1991). Personality and personality measures.. In M.D. Dunette, & L.M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol.4, pp.873-919). Palo Alto, CA: Consulting Psychologists Press.* Hogan, J.C. Ogden, G.D. Gebhardt, D.L. Fleishman, E.A. (1980). Reliability and validity of methods for evaluating perceived physical effort.. Journal of Applied Psychology, 65, 672– 679* Huber, V.L. (1991). Comparison of supervisor-incumbent and female-male multidimensional job evaluation ratings.. Journal of Applied Psychology, 76, 115– 121* Hughes, G.L. Prien, E.P. (1989). Evaluation of task and job skill linkage judgments used to develop test specifications.. Personnel Psychology, 42, 283– 292Hunter, J.E. Schmidt, F.L. Jackson, G.B. (1982). Meta-analysis: Cumulating findings across research. . Beverly Hills. CA: Sage.Hunter, J.E. Hunter, R.F. (1984). Validity and utility of alternative predictors of job performance.. Psychological Bulletin, 96, 72– 98Johnson, B.T. Mullen, B. Salas, E. (1995). Comparison of three major meta-analytic approaches.. Journal of Applied Psychology, 80, 94– 106* Jones, A.P. Main, D.S. Butler, M.C. Johnson, L.A. (1982). Narrative job descriptions as potential sources of job analysis ratings.. Personnel Psychology, 35, 813– 828Lawler, E.E. (1994). From job-based to competency-based organizations.. Journal of Organizational Behavior, 15, 3– 15* Madigan, R.M. (1985). Comparable worth judgments: A measurement properties analysis.. Journal of Applied Psychology, 70, 137– 147* Mallamad, S.M. Levine, J.M. Fleishman, E.A. (1980). Identifying ability requirements by decision flow diagrams.. Human Factors, 22, 57– 68* McCormick, E.J. Jeanneret, P.R. Mecham, R.C. (1972). A study of job characteristics and job dimensions as based on the Position Analysis Questionnaire.. Journal of Applied Psychology, 56, 347– 368McDaniel, M.A. Whetzel, D.L. Schmidt, F.L. Maurer, S.D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis.. Journal of Applied Psychology, 79, 599– 616Morgeson, F.P. Campion, P. (1997). Social and cognitive sources of potential inaccuracy in job analysis.. Journal of Applied Psychology, 5, 627– 655Mullen, B. (1989). Advanced basic meta-analysis. . London, Erlbaum.Offermann, L.R. Gowing, M.K. (1993). Personnel selection in the future: The impact of changing demographics and the nature of work.. In N. Schmitt & W.C. Borman (Eds.), Personnel selection in organizations (pp.385-417). San Francisco: Jossey-Bass.Roe, R.A. (1996). Naar een nieuw paradigma voor personeelsselectie ‘Toward a new paradigm of personnel selsction’.. In R. Bouwen, K. de Witte, & J. Verboven (Eds.), Organizeren en veranderen ‘To organize and to change’ (pp.287-312). Leuven: Garant.Rosenthal, R. (1991). Meta-analytic procedures for social research. . Beverly Hills, CA: Sage.Rosenthal, R. Rosnov, R.L. (1991). Essentials of behavioral research. Methods and Data Analysis. . London: McGraw-Hill.Rosenthal, R. Rubin, D. (1978). Interpersonal expectancy effects: The first 345 studies.. Behavioral and Brain Sciences, 3, 377– 415Sanchez, J. (1990). The effects of job experience on judgments of task importance. . Paper presented at the annual conference of the Society for Industrial and Organizational Psychology, Miami.* Sanchez, J.I. Fraser, S.L (1992). On the choice of scales for task analysis.. Journal of Applied Psychology, 77, 545– 553* Sanchez, J.I. Levine, E.L. (1989). Determining important tasks within jobs: A policy-capturing approach.. Journal of Applied Psychology, 74, 336– 342* Sanchez, J.I. Levine, E.L. (1994). The impact of raters' cognition on judgment accuracy: An extension to the job analysis domain.. Journal of Business and Psychology, 9, 47– 57Schmidt, F.L. Hunter, J.E. (1977). Development of a general solution to the problem of validity generalization.. Journal of Applied Psychology, 62, 529– 541Schmidt, F.L. Hunter, J.E. (1981). Employment testing, old theories and new research findings.. American Psychologist, 36, 1128– 1137Schmidt, F.L. Hunter, J.E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.. Psychological Bulletin, 124, 262– 274* Schmitt, N.S. Fine, S.A. (1983). Interrater reliability of judgments of functional levels and skill requirements of jobs based on written task statements.. Journal of Occupational Psychology, 56, 121– 127Schuler, H. (1989). Some advantages and problems of job analysis.. In M.Smith, & I.T. Robertson (Eds.), Advances in selection and assessment (pp.31-42). London: Wiley.Schneider, R.J. Hough, L.M. (1995). Personality and industrial/organizational psychology.. In C.L. Cooper & I.T. Robertson (Eds.), International review of industrial and organizational psychology (pp.75-129). New York: Wiley.* Schwab, D.P. Heneman, H.G. (1986). Assessment of a consensus based multiple information source job evaluation system.. Journal of Applied Psychology, 71, 354– 356Smith, M. (1994). A theory of the validity of predictors in selection.. Journal of Occupational and Organizational Psychology, 67, 13– 31* Smith, J.E. Hakel, M.D. (1979). Convergence among data sources, response bias, and reliability and validity of a structured job analysis questionnaire.. Personnel Psychology, 32, 677– 692* Sparrow, J. (1989). The utility of PAQ in relating job behaviors to traits.. Journal of Occupational Psychology, 62, 151– 162Sparrow, P.R. Bognanno, M. (1993). Competency requirement forecasting: Issues for international selection and assessment.. International Journal of Selection and Assessment, 1, 50– 58Spector, P.E. Brannick, M.T. Coovert, M.D. (1989). Job analysis.. In C.L. Cooper & I.T. Robertson (Eds.), International review of industrial and organizational psychology (pp.281- 329). New York: Wiley.* Surette, M.A. Aamodt, M.G. Johnson, D.L. (1990). Effects of analyst training and amount of available job related information on job analysis ratings.. Journal of Business and Psychology, 4, 439– 451* Taylor, L.R. (1978). Empirically derived job families as a foundation for the study of validity generalization, Study I: The construction of job families based on the component and overall dimensions of the PAQ.. Personnel Psychology, 31, 325– 340* Taylor, L.R. Colbert, G.A. (1978). Empirically derived job families as a foundation for the study of validity generalization: Study II.. The Construction of job families based on company-specific PAQ job dimensions.. Personnel Psychology, 31, 341– 353Viswesvaran, C. Ones, D.S. Schmidt, F.L. (1996). Comparative analysis of the reliability of job performance ratings.. Journal of Applied Psychology, 81, 557– 574Voskuijl, O.F. Meyer, J. (1999). The use of student data in job analysis research.. European Journal of psychological Assessment, 15, 127– 132* Webb, N.M. Shavelson, R.S. (1981). Generalizability of general education development ratings of jobs in the United States.. Journal of Applied Psychology, 66, 186– 192Wexley, K.N. Silverman, S.B. (1978). An examination of differences between managerial effectiveness and response patterns on a structured job analysis questionnaire.. Journal of Applied Psychology, 63, 646– 649Wilson, M.A. (1997). The validity of task coverage ratings by incumbents and supervisors: Bad news.. Journal of Business and Psychology, 12, 85– 95Wohlers, A.J. London, M. (1989). Ratings of managerial characteristics: Evaluation difficulty, co-worker agreement, and self-awareness.. Personnel Psychology, 42, 235– 261
