Abstract
Abstract. Summarizing and organizing research in narrative reviews is a classic procedure for cumulating research. In recent years narrative reviews have been increasingly, though not completely, replaced by meta-analyses. Using a case study of a prominent narrative review of the behavioral priming literature (Bargh, Schwader, Hailey, Dyer, & Boothby, 2012), we show that narrative reviews run the risk of drawing a picture that tends to be too good to be true, when the effect-sizes of the papers cited in the narrative review are compared to meta-analyses of the respective topic. We shortly discuss the reasons for this, emphasizing two sources of bias that may inflict narrative reviews to a larger degree than meta-analyses, namely bias in study selection, and bias in study aggregation.
References
(2006). What have we been priming all these years? On the development, mechanisms, and ecology of nonconscious social behavior. European Journal of Social Psychology, 36, 147–168.
(2012, May 11). Priming effects replicate just fine, thanks [Web log comment]. Psychology Today. Retrieved from https://www.psychologytoday.com/blog/the-natural-unconscious/201205/priming-effects-replicate-just-fine-thanks
(1996). Automaticity of social behavior: Direct effect of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71, 230–244.
(2012). Automaticity in social-cognitive processes. Trends in Cognitive Sciences, 16, 593–605.
(2002). The malleability of automatic stereotypes and prejudice. Personality and Social Psychology Review, 6, 242–261.
(2012). Sequential priming measures of implicit social cognition: A meta-analysis of associations with behavior and explicit attitudes. Personality and Social Psychology Review, 16, 330–350.
(2011). A single exposure to the American flag shifts support toward Republicanism up to 8 months later. Psychological Science, 22, 1011–1018.
(2013). Mere exposure to money increases endorsement of free-market systems and social inequality. Journal of Experimental Psychology: General, 142, 301–306.
(2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9, 40–48.
(2006). Automatic social behavior as motivated preparation to interact. Journal of Personality and Social Psychology, 90, 893–910.
(2003). Comparison of systematic and narrative reviews: The example of the atypical antipsychotics. Epidemiologia e Psichiatria Sociale, 12, 146–153.
(1989). Integrating research: A guide for literature reviews (2nd ed.). Newbury Park, CA: Sage.
(2012). The overview of reviews: Unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. The American Psychologist, 67, 446–462.
(2004). A meta-analysis of priming effects on impression formation supporting a general model of informational biases. Personality and Social Psychology Review, 8, 2–27.
(2009). Implicit measures: A normative analysis and review. Psychological Bulletin, 135, 347.
(2014). Welcome back theory!. Perspectives on Psychological Science, 9, 72–75.
(2001). The perception-behavior expressway: Automatic effects of social perception on social behavior. Advances in Experimental Social Psychology, 33, 1–40.
(2012). Behavioral priming: It’s all in the mind, but whose mind. PLoS One, 7, e29081.
(2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463.
(1986). On the automatic activation of attitudes. Journal of Personality and Social Psychology, 50, 229–238.
(2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151–156.
(1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3–8.
(2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91–108.
(2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17–41.
(2013). Two failures to replicate high-performance-goal priming effects. PLoS One, 8, e72467.
(2013). On the automatic activation of attitudes: A quarter century of evaluative priming research. Psychological Bulletin, 139, 1062–1089.
(2002). The nonconsciousness of self-consciousness. Journal of Personality and Social Psychology, 83, 406–424.
(2014). Content and process priming: A review. Journal of Consumer Psychology, 24, 96–118.
(2010). Priming Christian religious concepts increases racial prejudice. Social Psychological and Personality Science, 1, 119–126.
(2012, September 26). A proposal to deal with questions about priming effects. [Open letter]. Retrieved from http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf
(2007). The impact of review articles. Laboratory Investigation, 87, 1174–1185.
(2014). An intersensory interaction account of priming effects – and their absence. Perspectives on Psychological Science, 9, 49–58.
(2014). Investigating variation in replicability: A “Many Labs” replication project. Social Psychology, 45, 142–152.
(2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One, 9, e105825.
(2015). Theory building, replication, and behavioral priming: Where do we need to go from here? Perspectives on Psychological Science, 10, 408–414.
(1990). Repetition priming and automaticity: Common underlying mechanisms? Cognitive Psychology, 22, 1–35.
(1973). The sociology of science: Theoretical and empirical investigations. Chicaco, IL: University of Chicago press.
(2014). Semantic priming well established. Science, 345, 523.
(1971). Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90, 227–234.
(2014). Unconscious influences on decision making: A critical review. The Behavioral and Brain Sciences, 37, 1–19.
(2015). Fooling ourselves. Nature, 526, 182–185.
(2016). An assessment of the magnitude of effect sizes evidence from 30 years of meta-analysis in management. Journal of Leadership & Organizational Studies, 23, 66–81.
(2013). Ten simple rules for writing a literature review. PLoS Computational Biology, 9, e1003149.
(2008). Structural priming: A critical review. Psychological Bulletin, 134, 427–459.
(2016). Estimating the difference between published and unpublished effect sizes. A meta-review. Review of Educational Research, 86, 207–236.
(2015). Why many priming results don’t (and won’t) replicate: A quantitative analysis Unpublished Manuscript. Tübingen, Germany: University of Tübingen.
(2015). Do subtle reminders of money change people’s political views? Journal of Experimental Psychology: General, 144, e73–e85.
(1994).
Parametric measures of effect size . In H. CooperL. V. HedgesEds., The handbook of research synthesis (pp. 231–244). New York, NY: Sage.(2015). meta: General Package for Meta-Analysis. R package (Version 4.3-2) [Computer software]. Retrieved from https://CRAN.R-project.org/package=meta
(2007). God is watching you. Priming God concepts increases prosocial behavior in an anonymous economic game. Psychological Science, 18, 803–809.
(2015). Religious priming: A meta-analysis with a focus on prosociality. Personality and Social Psychology Review. Advance online publication. doi: 10.1177/1088868314568811
(2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
(2014). p-Curve and effect size correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666–681.
(2014). Religious magnanimity: Reminding people of their religious belief system reduces hostility after threat. Journal of Personality and Social Psychology, 107, 432–453.
(2016). Selection bias, vote counting, and money-priming effects: A comment on Rohrer, Pashler, and Harris (2015) and Vohs (2015). Journal of Experimental Psychology: General, 145, 655–663.
(2009). Mechanisms of masked priming: A meta-analysis. Psychological Bulletin, 135, 452–477.
(2015). Money priming can change people’s thoughts, feelings, motivations, and behaviors: An update on 10 years of experiments. Journal of Experimental Psychology: General, 144, e86–e93.
(2006). The psychological consequences of money. Science, 314, 1154–1156.
(2016). From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological Bulletin, 142, 472–497.
(2009). Multiple mechanisms of prime-to-behavior effects. Social and Personality Psychology Compass, 3, 566–581.
