Does Gender Matter in Grant Peer Review?
An Empirical Investigation Using the Example of the Austrian Science Fund
Abstract
One of the most frequently voiced criticisms of the peer review process is gender bias. In this study we evaluated the grant peer review process (external reviewers’ ratings, and board of trustees’ final decision: approval or no approval for funding) at the Austrian Science Fund with respect to gender. The data consisted of 8,496 research proposals (census) across all disciplines from 1999 to 2009, which were rated on a scale from 1 to 100 (poor to excellent) by 18,357 external reviewers in 23,977 reviews. In line with the current state of research, we found that the final decision was not associated with applicant’s gender or with any correspondence between gender of applicants and reviewers. However, the decisions on the grant applications showed a robust female reviewer salience effect. The approval probability decreases (up to 10%), when there is parity or a majority of women in the group of reviewers. Our results confirm an overall gender null hypothesis for the peer review process of men’s and women’s grant applications in contrast to claims that women’s grants are systematically downrated.
References
(2009). A note on comparing the estimates of models for cluster-correlated or longitudinal data with binary or ordinal outcomes. Psychometrika, 74, 97–105. doi: 10.1007/S11336-008-9080-1
(2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199–245.
(2007a). Reliability, fairness and predictive validity of committee peer review – evaluation of the selection of post-graduate fellowship holders by the Boehringer Ingelheim Fonds. B.I.F. Futura, 19, 7–19.
(2007b). Gatekeepers of science – effects of external reviewers’ attributes on the assessments of fellowship applications. Journal of Informetrics, 1, 83–91. doi: 10.1016/j.joi.2006.09.005
(2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1, 226–238. doi: 10.1016/j.joi.2007.03.001
(2008). How to detect indications of potential sources of bias in peer review: A generalized latent variable modeling approach exemplified by a gender study. Journal of Informetrics, 2, 280–287.
(2011). A meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5, 346–359. doi: 10.1016/j.joi.2011.01.006
(2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Sciences, 108, 3157–3167. doi: 10.1073/pnas.1014871108
(2010). Factors influencing approval probability in FWF decision-making procedures. Vienna, Austria: Fonds zur Förderung der wissenschaftlichen Forschung (FWF).
(2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.
(2005). The gender similarities hypothesis. American Psychologist, 60, 581–592. doi: 10.1037/0003-066x.60.6.581
(2003). A multilevel cross-classified modelling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society Series A (Statistics in Society), 166, 279–300.
(1995). Federal, research impact assessment – axioms, approaches, applications. Scientometrics, 34, 163–206.
(2006). SAS for mixed models (2nd ed.). Cary, NC: SAS Institute.
(2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58, 127–137. doi: 10.1046/j.0039-0402.2003.00252.x
(2009). Do women have less success in peer review? Nature, 459, 602.
(2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79, 1290–1326. doi: 10.3102/0034654309334143
(2006). Scientists’ perceptions of organizational justice and self-reported misbehaviors. Journal of Empirical Research on Human Research Ethics, 1, 51–66. doi: 10.1525/jer.2006.1.1.51
(1963). Visibility in small groups. The Journal of Social Psychology, 61, 311–325.
(1979). Effects of household sex composition on the salience of one’s gender in the spontaneous self-concept. Journal of Experimental Psychology, 15, 77–90. doi: 10.1016/0022-1031(79)90020-9
(2007). A simulation study of sample size for multilevel logistic regression models. BMC Medical Research Methodology, 7, doi: 10.1186/1471-2288-7-34
(2003). Group composition: Explaining similarities and differences among group members. In , The Sage handbook of social psychology (pp. 367–380). London, UK: Sage.
(2008). The social psychology of gender. London, UK: Guilford Press.
(2011). Rethinking observed power concept, practice, and implications. Methodology-European Journal of Research Methods for the Behavioral and Social Sciences, 7, 81–87. doi: 10.1027/1614-2241/A000025
(2009). Judging merit. New York, NY: Psychology Press.
(2008). Majority, minority, and parity: Effects of gender and group size on perceived group variability. Social Psychology Quarterly, 71, 114–142. doi: 10.1111/1467-985X.00278
(1998). Is there gender bias in research fellowships awarded by the NHMRC7? The Medical Journal of Australia, 169, 21623–21624.
.(1997). Women and peer review. An audit of the Wellcome Trust’s decision-making on grants. London, UK: The Wellcome Trust.
(1997). Nepotism and sexism in peer-review. Nature, 387, 341–343. doi: 10.1038/387341a0
(2000). Real science. What it is, and what it means. Cambridge, UK: Cambridge University Press.
