Avoiding Methodological Biases in Meta-Analysis
Use of Online Versus Offline Individual Participant Data (IPD) in Educational Psychology
Abstract
Abstract. Individual participant data (IPD) meta-analysis is the gold standard of meta-analyses. This paper points out several advantages of IPD meta-analysis over classical meta-analysis, such as avoiding aggregation bias (e.g., ecological fallacy or Simpson’s paradox) and shows how its two main disadvantages (time and cost) can be overcome through Internet-based research. Ideally, we recommend carrying out IPD meta-analyses that consider online versus offline data gathering processes and examine data quality. Through a comprehensive literature search, we investigated whether IPD meta-analyses published in the field of educational psychology already follow these recommendations; this was not the case. For this reason, the paper demonstrates characteristics of ideal meta-analysis on teachers’ judgment accuracy and links it to recent meta-analyses on that topic. The recommendations are important for meta-analysis researchers and for readers and reviewers of meta-analyses. Our paper is also relevant to current discussions within the psychological community on study replication.
References
2015). Editorial. Psychological Bulletin, 141, 1–5.
(1969).
(A typology of ecological fallacies . In M. DoganS. RokanEds., Quantitative Ecological Analysis in the Social Sciences (pp. 69–86). Cambridge, MA: MIT Press.2001). Judgment of factors influencing interest: An Australian study. Journal of Vocational Education Research, 26, 1–13.
(Batinic B.Reips U.-D.Bošnjak M.Eds.. (2002). Online Social Sciences. Seattle, WA: Hogrefe & Huber.
2002). Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: Ecological bias rears its ugly head. Statistics in Medicine, 31, 371–387. doi: 10.1002/sim.1023
(2014). Things I have learned about meta-analysis since 1990: Reducing bias in search of “The Big Picture”. Canadian Journal of Learning and Instruction, 40, 17.
(1975). Sex bias in graduate admission: Data from Berkeley. Science, 187, 398–404.
(2014). Ein national gefördertes Onlinelabor als Infrastruktur für die psychologische Forschung
([A nationally funded online laboratory as infrastructure for psychological research] . Psychologische Rundschau, 65, 75–85. doi: 10.1026/0033-3042/a0001981959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.
(1975). Experimental evidence of dietary factors and hormone-dependent cancers. Cancer Research, 35, 3374–3383.
(1993). The Cochrane collaboration: Preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences, 703, 156–163.
(2015). The association between alcohol use and engagement in casual sexual relationships and experiences: A meta-analytical review of non-experimental studies. Archives of Sexual Behavior, 44, 837–856.
(2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological Methods, 14, 165–176.
(2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–1000.
(2015). Get real in individual participant data (IPD) meta-analysis: A review of the methodology. Research Synthesis Methods, 6, 293–309. doi: 10.1002/jrsm.1160
. (2014). Social desirability is the same in offline, online, and paper surveys: A meta-analysis. Computers in Human Behavior, 36, 487–495.
(2015). A systematic review of the impact of multiple language teaching, prior language experience and acquisition order on student’s language proficiency in primary and secondary school. Copenhagen, Denmark: Danish Clearinghouse for Educational Research, Department of Education, Aarhus University.
(1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.
(2016). One hundred years of research: Prudent aspirations. Educational Researcher, 45, 69–72. doi: 10.3102/0013189X16639026
(2000). Meta-analysis using multilevel models with an application to the study of class size effects. Applied Statistics, 49, 399–412.
(Hammond K. R.Stewart T. R.Eds.. (2001). The essential Brunswik: Beginnings, explications, applications. Oxford, UK: University Press.
1996). Aggregation and the estimated effects of school resources. Review of Economics and Statistics, 78, 611–627.
(2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London, UK: Routledge.
(1989). Judgments of academic achievement: A review of literature. Review of Educational Research, 59, 297–313. doi: 10.3102/00346543059003297
(2010). Meta-research: The art of getting it wrong. Research Synthesis Methods, 1, 169–184. doi: 10.1002/jrsm.19
(2002). Commentary: Meta-analysis of individual participants’ data in genetic epidemiology. American Journal of Epidemiology, 156, 204–210.
(2008). Determinants of linear judgment: A meta-analysis of lens studies. Psychological Bulletin, 134, 404–426. doi: 10.1037/0033-2909.134.3.404
(2010). Flesh on the bones: A critical meta-analytic perspective on lens studies. Mannheim, Germany: MADOC.
(2016). Teachers as judges: A psychometric (re)evaluation of teacher’s judgment accuracy [Working paper]. Zurich, Switzerland: University of Zurich.
(2009). A meta-analysis of judgment achievement defined by the lens model equation. Swiss Journal of Psychology, 68, 99–112. doi: 10.1024/1421-0185.68.2.99
(2008). Internet-basierte Messung Sozialer Erwünschtheit: Theoretische Grundlagen und Experimentelle Untersuchung
([Internet-based measurement of social desirability] . Saarbrücken, Germany: VDM Verlag Dr. Müller.2013). A critical meta-analysis of Lens Model studies in human judgment and decision-making. PLoS One, 8, e83528. doi: 10.1371/journal.pone.0083528
(1997). Quantitative synthesis in systematic reviews. Annals of Internal Medicine, 127, 820–826.
(1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. The American Psychologist, 48, 1181–1209.
(2005). The strengths and limitations of meta-analyses based on aggregate data. BMC Medical Research Methodology, 5, 14. doi: 10.1186/1471-2288-5-14
(2007). Sleep, sex, and the Web: Surveying the difficult-to-reach clinical population suffering from sexsomnia. Behavior Research Methods, 39, 233–236.
(1996). Symposium on school quality and educational outcomes. Review of Economics and Statistics, 78, 559–561.
(2015). Estimating the reproducibility of psychological science. Science, 349, 943. doi: 10.1126/science.aac4716
(2012). Combining individual participant and aggregated data in a meta-analysis with correlational studies. Research Synthesis Methods, 3, 257–268.
(2016). Overviews in educational research: A systematic review and analysis. Review of Educational Research. Advance online publication. doi: 10.3102/0034654316631117
(2002). Standards for Internet-based experimenting. Experimental Psychology, 49, 243–256.
(2006).
(Web-based methods . In M. EidE. DienerEds., Handbook of multimethod measurement in psychology (pp. 73–85). Washington, DC: American Psychological Association. doi: 10.1037/11383-0062008).
(How Internet-mediated research changes science . In A. BarakEd., Psychological aspects of cyberspace: Theory, research, applications (pp. 268–294). Cambridge, UK: Cambridge University Press.Reips U.-D.Bošnjak M.Eds.. (2001). Dimensions of internet science. Lengerich, Germany: Pabst.
2015). Methodological challenges in the use of the Internet for scientific research: Ten solutions and recommendations. Studia Psychologica, Advance online publication
(2007). Evidence synthesis combining individual patient data and aggregate data: A systematic review identified current practice and possible methods. Journal of Clinical Epidemiology, 60, 431–439. doi: S0895-4356(06)00403-3
(1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.
(2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82. doi: 10.1146/annurev.psych.52.1.59
(2008). Publication bias as a threat to the validity of meta-analytic results. Journal of Experimental Criminology, 4, 61–81.
(2004). Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. Journal of Clinical Epidemiology, 57, 683–697.
(2014). Methods of meta-analysis: Correcting error and bias in research findings. Los Angeles, CA: Sage.
(2016). The crisis of confidence in research findings in Psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology, 4, 32–37. doi: 10.1037/arc0000029
(2015). Introduction to the special issue on the origins of modern meta-analysis. Research Synthesis Methods, 6, 219–220. doi: 10.1002/jrsm.1148
(2015). A decade of individual participant data meta-analyses: A review of current practice. Contemporary Clinical Trials, 45, 76–83. doi: 10.1016/j.cct.2015.06.012
(2005). Meta-analysis of individual patient data from randomized trials: A review of methods used in practice. Clinical Trials, 2, 209–217. doi: 10.1191/1740774505cn087oa
(1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Series B, 13, 238–241.
(2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Education Research, 75, 417–453.
(2015). Preferred reporting items for a systematic review and meta-analysis of individual participant data: The PRISMA-IPD statement. Journal of the American Medical Association (JAMA), 313, 1657–1665. doi: 10.1001/jama.2015.3656
(1993). Meta-analysis of the literature or of individual patient data: Is there a difference? The Lancet, 341, 418–422.
(2002). To IPD or not to IPD? Evaluation & the Health Professions, 25, 76–97.
(1999). Die Soziale-Erwünschtheits-Skala-17 (SES-17): Entwicklung und erste Befunde zu Reliabilität und Validität
([The Social Desirability Scale-17 (SDS-17): Development and first findings on reliability and validity] . Diagnostica, 45, 173–177. doi: 10.1026/0012-1924.45.4.1732012). Accuracy of teachers’ judgments of students’ academic achievement: A meta-analysis. Journal of Educational Psychology, 104, 743–763. doi: 10.1037/a0027627
(2008). Meta-analysis of individual- and aggregate-level data. Statistics in Medicine, 27, 651–669. doi: 10.1002/sim.2916
(2015). Phantom effects in school composition research: Consequences of failure to control biases due to measurement error in traditional multilevel models. School Effectiveness and School Improvement, 26, 75–101.
(2015). Individual Participant Data (IPD) meta-analysis of randomized controlled trials: Guidance on their use. PLoS Medicine, 12, e1001855. doi: 10.1371/journal.pmed.1001855
(2016). Statistical harmonization methods in Individual Participants Data meta-analysis are highly needed. Biometrics & Biostatistics International Journal, 3, 00064. doi: 10.15406/bbij.2016.03.00064
(2007). Random-effects models and moderator analyses in meta-analysis. Zeitschrift für Psychologie/Journal of Psychology, 215, 104–121.
(2010). Outlier and influence diagnostics for meta-analysis. Research Synthesis Methods, 1, 112–125. doi: 10.1002/jrsm.11
(