Skip to main content
Original Article

Power-Enhanced Funnel Plots for Meta-Analysis

The Sunset Funnel Plot

Published Online:https://doi.org/10.1027/2151-2604/a000392

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

References

  • Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101. https://doi.org/10.2307/2533446 First citation in articleCrossrefGoogle Scholar

  • Chevance, A., Schuster, T., Steele, R., Ternès, N., & Platt, R. W. (2015). Contour plot assessment of existing meta-analyses confirms robust association of statin use and acute kidney injury risk. Journal of Clinical Epidemiology, 68, 1138–1143. https://doi.org/10.1016/j.jclinepi.2015.05.030 First citation in articleCrossrefGoogle Scholar

  • Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. https://doi.org/10.1111/j.0006-341X.2000.00455.x First citation in articleCrossrefGoogle Scholar

  • Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634. https://doi.org/10.1136/bmj.315.7109.629 First citation in articleCrossrefGoogle Scholar

  • Forstmeier, W., Wagenmakers, E. J., & Parker, T. H. (2017). Detecting and avoiding likely false-positive finding: A practical guide. Biological Reviews, 92, 1941–1968. https://doi.org/10.1111/brv.12315 First citation in articleCrossrefGoogle Scholar

  • Ioannidis, J. P., Stanley, T. D., & Doucouliagos, H. (2017). The power of bias in economics research. The Economic Journal, 127, F236–F265. https://doi.org/10.1111/ecoj.12461 First citation in articleCrossrefGoogle Scholar

  • Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245–253. https://doi.org/10.1177/1740774507079441 First citation in articleCrossrefGoogle Scholar

  • Kossmeier, M., Tran, U. S., & Voracek, M. (2018). metaviz. [R software package]. Retrieved from https://CRAN.R-project.org/package=metaviz First citation in articleGoogle Scholar

  • Kossmeier, M., Tran, U., & Voracek, M. (2019). Visual inference for the funnel plot in meta-analysis. Zeitschrift für Psychologie, 227, 83–89. https://doi.org/10.1027/2151-2604/a000358 First citation in articleLinkGoogle Scholar

  • Langan, D., Higgins, J. P., Gregory, W., & Sutton, A. J. (2012). Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis. Journal of Clinical Epidemiology, 65, 511–519. https://doi.org/10.1016/j.jclinepi.2011.10.009 First citation in articleCrossrefGoogle Scholar

  • Lau, J., Ioannidis, J. P., Terrin, N., Schmid, C. H., & Olkin, I. (2006). Evidence based medicine: The case of the misleading funnel plot. British Medical Journal, 333, 597–600. https://doi.org/10.1136/bmj.333.7568.597 First citation in articleCrossrefGoogle Scholar

  • Light, R. J., & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press. First citation in articleCrossrefGoogle Scholar

  • Mathie, R. T., Ramparsad, N., Legg, L. A., Clausen, J., Moss, S., Davidson, J. R., … McConnachie, A. (2017). Randomised, double-blind, placebo-controlled trials of non-individualised homeopathic treatment: Systematic review and meta-analysis. Systematic Reviews, 6, 63. https://doi.org/10.1186/s13643-017-0445-3 First citation in articleCrossrefGoogle Scholar

  • Muncer, S. J., Craigie, M., & Holmes, J. (2003). Meta-analysis and power: Some suggestions for the use of power in research synthesis. Understanding Statistics, 2, 1–12. https://doi.org/10.1207/S15328031US0201_01 First citation in articleCrossrefGoogle Scholar

  • Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61, 991–996. https://doi.org/10.1016/j.jclinepi.2007.11.010 First citation in articleCrossrefGoogle Scholar

  • Pietschnig, J., Penke, L., Wicherts, J. M., Zeiler, M., & Voracek, M. (2015). Meta-analysis of associations between human brain volume and intelligence differences: How strong are they and what do they mean? Neuroscience & Biobehavioral Reviews, 57, 411–432. https://doi.org/10.1016/j.neubiorev.2015.09.017 First citation in articleCrossrefGoogle Scholar

  • R Core Team. (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/ First citation in articleGoogle Scholar

  • Renkewitz, F., & Keiner, M. (2019). How to detect publication bias in psychological research: A comparative evaluation of six statistical methods. Zeitschrift für Psychologie, 227, 261–279. https://doi.org/10.1027/2151-2604/a000386 First citation in articleLinkGoogle Scholar

  • Rücker, G., Schwarzer, G., & Carpenter, J. (2008). Arcsine test for publication bias in meta-analyses with binary outcomes. Statistics in Medicine, 27, 746–763. https://doi.org/10.1002/sim.2971 First citation in articleCrossrefGoogle Scholar

  • Schild, A. H. E., & Voracek, M. (2013). Less is less: A systematic review of graph use in meta‐analyses. Research Synthesis Methods, 4, 209–219. https://doi.org/10.1002/jrsm.1076 First citation in articleGoogle Scholar

  • Schimmack, U. (2016). The replicability-index: Quantifying statistical research integrity. Retrieved from https://replicationindex.wordpress.com/2016/01/31/a-revised-introduction-to-the-r-index/ First citation in articleGoogle Scholar

  • Simmonds, M. (2015). Quantifying the risk of error when interpreting funnel plots. Systematic Reviews, 4, 24. https://doi.org/10.1186/s13643-015-0004-8 First citation in articleCrossrefGoogle Scholar

  • Sterne, J. A., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. Journal of Clinical Epidemiology, 54, 1046–1055. https://doi.org/10.1016/S0895-4356(01)00377-8 First citation in articleCrossrefGoogle Scholar

  • Sterne, J. A., Gavaghan, D., & Egger, M. (2000). Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53, 1119–1129. https://doi.org/10.1016/S0895-4356(00)00242-0 First citation in articleCrossrefGoogle Scholar

  • Sterne, J. A., Sutton, A. J., Ioannidis, J. P., Terrin, N., Jones, D. R., Lau, J., … Tetzlaff, J. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal, 343, d4002. https://doi.org/10.1136/bmj.d4002 First citation in articleCrossrefGoogle Scholar

  • Szucs, D., & Ioannidis, J. P. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15, e2000797. https://doi.org/10.1371/journal.pbio.2000797 First citation in articleCrossrefGoogle Scholar

  • Terrin, N., Schmid, C. H., & Lau, J. (2005). In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology, 58, 894–901. https://doi.org/10.1016/j.jclinepi.2005.01.006 First citation in articleCrossrefGoogle Scholar

  • Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-24277-4 First citation in articleCrossrefGoogle Scholar