Result-Blind Peer Reviews and Editorial Decisions
A Missing Pillar of Scientific Culture
Abstract
The present article suggests a possible way to reduce the file drawer problem in scientific research (Rosenthal, 1978, 1979), that is, the tendency for “nonsignificant” results to remain hidden in scientists’ file drawers because both authors and journals strongly prefer statistically significant results. We argue that peer-reviewed journals based on the principle of rigorous evaluation of research proposals before results are known would address this problem successfully. Even a single journal adopting a result-blind evaluation policy would remedy the persisting problem of publication bias more efficiently than other tools and techniques suggested so far. We also propose an ideal editorial policy for such a journal and discuss pragmatic implications and potential problems associated with this policy. Moreover, we argue that such a journal would be a valuable addition to the scientific publication outlets, because it supports a scientific culture encouraging the publication of well-designed and technically sound empirical research irrespective of the results obtained. Finally, we argue that such a journal would be attractive for scientists, publishers, and research agencies.
References
2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678.
(2012). We knew the future all along: Scientific hypothesizing is much more accurate than other forms of precognition-A satire in one part. Perspectives on Psychological Science, 7, 307–309.
(1969). Über die Anwendung des Signifikanztests bei theorietestenden Experimenten
([Application of significance tests in theory-testing experiments] . Psychologische Beiträge, 11, 275–285.1972). Der Signifikanztest in der psychologischen Forschung
([The test of significance in psychological research] . Frankfurt, Germany: Akademische Verlagsgesellschaft.1962). Statistical power of abnormal-social psychological research: A review. Journal of Abnormal Psychology, 65, 145–153.
(1988). Statistical power analysis for the social sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
(1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
(1990). Improving editorial procedures. American Psychologist, 45, 665–666.
(1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–34.
(2010). Experimental Psychology: A note on statistical analysis. Experimental Psychology, 57, 1–4.
(2013). A comprehensive review of reporting practices in psychology journals: Are effect sizes really enough? Theory & Psychology, 23, 98–122. doi: 10.1177/0959354312436870
(2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In , The SAGE handbook of quantitative methodology for the social sciences (pp. 391–408). Thousand Oaks, CA: Sage.
(1996). Models for estimating the number of unpublished studies. Statistics in Medicine, 15, 2493–2507.
(1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
(2010). Letter from the new editor-in-chief. European Psychologist, 15, 1–2.
(1990). The traditional editorial statement. Journal of Experimental Psychology: General, 119, 3–4.
(2009). p(rep) misestimates the probability of replication. Psychonomic Bulletin & Review, 16, 424–429.
(1988). Selection models and the file drawer problem. Statistical Science, 3, 109–135.
(2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345–353.
(1988). Improving what is published: a model in search for an editor. American Psychologist, 43, 635–642.
(2006). The case of the misleading funnel plot. British Medical Journal, 333, 597–600.
(2010). The truth wears off. Is there something wrong with the scientific method? The New Yorker, December 13, 2010.
(2001). A historical introduction to the philosophy of science (4th ed.). Oxford, UK: Oxford University Press.
(2010). Are scientists nearsighted gamblers? The misleading nature of impact factors. Frontiers in Psychology, 1, 1–2.
(1999). Detecting biases in meta-analyses: A case study of fluctuating asymmetry and sexual selection. The American Naturalist, 154, 220–233.
(1959). The logic of scientific discovery. London, UK: Routledge.
(2011). Is there evidence of publication bias in JDM research? Judgment and Decision Making, 6, 870–881.
(2012). Psychology’s woes and a partial cure: The value of replication. APS Observer, 25, 27–29.
(1978). Combining results of independent studies. Psychological Bulletin, 85, 185–193.
(1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641.
(2009). Bayesian t-tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237.
(2000). Publication bias: The “file-drawer” problem in scientific inference. Journal of Scientific Exploration, 14, 91–106.
(2012). The ironic effect of significant results on the credibility of multiple study articles. Psychological Methods 17, 551–566. doi: 10.1037/a0029487
. (2011). Unpublished results hide the decline effect. Nature, 470, 437.
(1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 107, 309–316.
(2011). False positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
(2012). Introduction to the Special Section: Data, data, everywhere … especially in my file drawer. Perspectives on Psychological Science, 7, 58–59.
(1959). Publication decisions and their possible effects on inferences drawn from tests of significance – or vice versa. Journal of the American Statistical Association, 54, 30–34.
(2005). In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology, 58, 894–901.
(1994). The pivotal role of replication in psychology: Empirically evaluating the replicability of sample results. Journal of Personality, 62, 157–176.
(1982). Judgment under uncertainty: Heuristics and biases. In , Judgment under uncertainty: Heuristics and biases (pp. 3–20). Cambridge, UK: Cambridge University Press.
(1970). A proposal for a new editorial policy in the social sciences. The American Statistician, 16, 16–19.
(1999). Statistical Methods in Psychology Journals: Guidelines and Explanations. American Psychologist, 54, 594–604.
(October 3, 2012). Nobel laureate challenges psychologists to clean up their act. Nature News and Comment. www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 (retrieved 05 October 2012).
(