Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Review Article

Avoiding Methodological Biases in Meta-Analysis

Use of Online Versus Offline Individual Participant Data (IPD) in Educational Psychology

Published Online:October 28, 2016https://doi.org/10.1027/2151-2604/a000251

Abstract

Abstract. Individual participant data (IPD) meta-analysis is the gold standard of meta-analyses. This paper points out several advantages of IPD meta-analysis over classical meta-analysis, such as avoiding aggregation bias (e.g., ecological fallacy or Simpson’s paradox) and shows how its two main disadvantages (time and cost) can be overcome through Internet-based research. Ideally, we recommend carrying out IPD meta-analyses that consider online versus offline data gathering processes and examine data quality. Through a comprehensive literature search, we investigated whether IPD meta-analyses published in the field of educational psychology already follow these recommendations; this was not the case. For this reason, the paper demonstrates characteristics of ideal meta-analysis on teachers’ judgment accuracy and links it to recent meta-analyses on that topic. The recommendations are important for meta-analysis researchers and for readers and reviewers of meta-analyses. Our paper is also relevant to current discussions within the psychological community on study replication.

References

  • Albarracín, D. (2015). Editorial. Psychological Bulletin, 141, 1–5. First citation in articleCrossref, Google Scholar

  • Alker, H. S. (1969). A typology of ecological fallacies. In M. DoganS. RokanEds., Quantitative Ecological Analysis in the Social Sciences (pp. 69–86). Cambridge, MA: MIT Press. First citation in articleGoogle Scholar

  • Athanasou, J. A. & Cooksey, R. W. (2001). Judgment of factors influencing interest: An Australian study. Journal of Vocational Education Research, 26, 1–13. First citation in articleCrossref, Google Scholar

  • Batinic B.Reips U.-D.Bošnjak M.Eds.. (2002). Online Social Sciences. Seattle, WA: Hogrefe & Huber. First citation in articleGoogle Scholar

  • Berlin, J. A., Santanna, J., Schmid, C. H., Szczech, L. A. & Feldman, H. I. (2002). Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: Ecological bias rears its ugly head. Statistics in Medicine, 31, 371–387. doi: 10.1002/sim.1023 First citation in articleCrossref, Google Scholar

  • Bernard, R. M. (2014). Things I have learned about meta-analysis since 1990: Reducing bias in search of “The Big Picture”. Canadian Journal of Learning and Instruction, 40, 17. First citation in articleGoogle Scholar

  • Bickel, P. J., Hammel, E. A. & O’Connell, J. W. (1975). Sex bias in graduate admission: Data from Berkeley. Science, 187, 398–404. First citation in articleCrossref, Google Scholar

  • Bruder, M., Göritz, A. S., Reips, U.-D. & Gebhard, R. K. (2014). Ein national gefördertes Onlinelabor als Infrastruktur für die psychologische Forschung [A nationally funded online laboratory as infrastructure for psychological research]. Psychologische Rundschau, 65, 75–85. doi: 10.1026/0033-3042/a000198 First citation in articleLink, Google Scholar

  • Campbell, D. T. & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. First citation in articleCrossref, Google Scholar

  • Carroll, K. (1975). Experimental evidence of dietary factors and hormone-dependent cancers. Cancer Research, 35, 3374–3383. First citation in articleGoogle Scholar

  • Chalmers, I. (1993). The Cochrane collaboration: Preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences, 703, 156–163. First citation in articleCrossref, Google Scholar

  • Claxton, S. E., DeLuca, H. K. & van Dulmen, M. H. (2015). The association between alcohol use and engagement in casual sexual relationships and experiences: A meta-analytical review of non-experimental studies. Archives of Sexual Behavior, 44, 837–856. First citation in articleCrossref, Google Scholar

  • Cooper, H. & Patall, E. A. (2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological Methods, 14, 165–176. First citation in articleCrossref, Google Scholar

  • Curran, P. J. & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–1000. First citation in articleCrossref, Google Scholar

  • Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., … the getReal methods review group. (2015). Get real in individual participant data (IPD) meta-analysis: A review of the methodology. Research Synthesis Methods, 6, 293–309. doi: 10.1002/jrsm.1160 First citation in articleCrossref, Google Scholar

  • Dodou, D. & De Winter, J. C. F. (2014). Social desirability is the same in offline, online, and paper surveys: A meta-analysis. Computers in Human Behavior, 36, 487–495. First citation in articleCrossref, Google Scholar

  • Dyssegaard, C. B., Egeberg, J. H., Sommersel, H. B., Steenberg, K. & Vestergaard, S. (2015). A systematic review of the impact of multiple language teaching, prior language experience and acquisition order on student’s language proficiency in primary and secondary school. Copenhagen, Denmark: Danish Clearinghouse for Educational Research, Department of Education, Aarhus University. First citation in articleGoogle Scholar

  • Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8. First citation in articleCrossref, Google Scholar

  • Glass, G. V. (2016). One hundred years of research: Prudent aspirations. Educational Researcher, 45, 69–72. doi: 10.3102/0013189X16639026 First citation in articleCrossref, Google Scholar

  • Goldstein, H., Yang, M., Omar, R., Turner, R. & Thompson, S. (2000). Meta-analysis using multilevel models with an application to the study of class size effects. Applied Statistics, 49, 399–412. First citation in articleGoogle Scholar

  • Hammond K. R.Stewart T. R.Eds.. (2001). The essential Brunswik: Beginnings, explications, applications. Oxford, UK: University Press. First citation in articleGoogle Scholar

  • Hanushek, E., Rivkin, S. & Taylor, L. (1996). Aggregation and the estimated effects of school resources. Review of Economics and Statistics, 78, 611–627. First citation in articleCrossref, Google Scholar

  • Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London, UK: Routledge. First citation in articleGoogle Scholar

  • Hoge, D. H. & Coladarci, T. (1989). Judgments of academic achievement: A review of literature. Review of Educational Research, 59, 297–313. doi: 10.3102/00346543059003297 First citation in articleCrossref, Google Scholar

  • Ioannidis, J. P. A. (2010). Meta-research: The art of getting it wrong. Research Synthesis Methods, 1, 169–184. doi: 10.1002/jrsm.19 First citation in articleCrossref, Google Scholar

  • Ioannidis, J. P. A., Rosenberg, P. S., Goedert, J. & O’Brien, T. R. (2002). Commentary: Meta-analysis of individual participants’ data in genetic epidemiology. American Journal of Epidemiology, 156, 204–210. First citation in articleCrossref, Google Scholar

  • Karelaia, N. & Hogarth, R. (2008). Determinants of linear judgment: A meta-analysis of lens studies. Psychological Bulletin, 134, 404–426. doi: 10.1037/0033-2909.134.3.404 First citation in articleCrossref, Google Scholar

  • Kaufmann, E. (2010). Flesh on the bones: A critical meta-analytic perspective on lens studies. Mannheim, Germany: MADOC. First citation in articleGoogle Scholar

  • Kaufmann, E. (2016). Teachers as judges: A psychometric (re)evaluation of teacher’s judgment accuracy [Working paper]. Zurich, Switzerland: University of Zurich. First citation in articleGoogle Scholar

  • Kaufmann, E. & Athanasou, J. A. (2009). A meta-analysis of judgment achievement defined by the lens model equation. Swiss Journal of Psychology, 68, 99–112. doi: 10.1024/1421-0185.68.2.99 First citation in articleLink, Google Scholar

  • Kaufmann, E. & Reips, U.-D. (2008). Internet-basierte Messung Sozialer Erwünschtheit: Theoretische Grundlagen und Experimentelle Untersuchung [Internet-based measurement of social desirability]. Saarbrücken, Germany: VDM Verlag Dr. Müller. First citation in articleGoogle Scholar

  • Kaufmann, E., Reips, U.-D. & Wittmann, W. W. (2013). A critical meta-analysis of Lens Model studies in human judgment and decision-making. PLoS One, 8, e83528. doi: 10.1371/journal.pone.0083528 First citation in articleCrossref, Google Scholar

  • Lau, J., Ioannidis, J. P. A. & Schmid, C. H. (1997). Quantitative synthesis in systematic reviews. Annals of Internal Medicine, 127, 820–826. First citation in articleCrossref, Google Scholar

  • Lipsey, M. W. & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. The American Psychologist, 48, 1181–1209. First citation in articleCrossref, Google Scholar

  • Lyman, G. H. & Kuderer, N. M. (2005). The strengths and limitations of meta-analyses based on aggregate data. BMC Medical Research Methodology, 5, 14. doi: 10.1186/1471-2288-5-14 First citation in articleCrossref, Google Scholar

  • Mangan, M. & Reips, U.-D. (2007). Sleep, sex, and the Web: Surveying the difficult-to-reach clinical population suffering from sexsomnia. Behavior Research Methods, 39, 233–236. First citation in articleCrossref, Google Scholar

  • Moffitt, R. (1996). Symposium on school quality and educational outcomes. Review of Economics and Statistics, 78, 559–561. First citation in articleCrossref, Google Scholar

  • Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349, 943. doi: 10.1126/science.aac4716 First citation in articleCrossref, Google Scholar

  • Pigott, T., Williams, R. & Polanin, J. (2012). Combining individual participant and aggregated data in a meta-analysis with correlational studies. Research Synthesis Methods, 3, 257–268. First citation in articleCrossref, Google Scholar

  • Polanin, J. R., Maynard, B. R. & Dell, N. A. (2016). Overviews in educational research: A systematic review and analysis. Review of Educational Research. Advance online publication. doi: 10.3102/0034654316631117 First citation in articleCrossref, Google Scholar

  • Reips, U.-D. (2002). Standards for Internet-based experimenting. Experimental Psychology, 49, 243–256. First citation in articleLink, Google Scholar

  • Reips, U.-D. (2006). Web-based methods. In M. EidE. DienerEds., Handbook of multimethod measurement in psychology (pp. 73–85). Washington, DC: American Psychological Association. doi: 10.1037/11383-006 First citation in articleCrossref, Google Scholar

  • Reips, U.-D. (2008). How Internet-mediated research changes science. In A. BarakEd., Psychological aspects of cyberspace: Theory, research, applications (pp. 268–294). Cambridge, UK: Cambridge University Press. First citation in articleGoogle Scholar

  • Reips U.-D.Bošnjak M.Eds.. (2001). Dimensions of internet science. Lengerich, Germany: Pabst. First citation in articleGoogle Scholar

  • Reips, U.-D., Buchanan, T., Krantz, J. H. & McGraw, K. (2015). Methodological challenges in the use of the Internet for scientific research: Ten solutions and recommendations. Studia Psychologica, Advance online publication First citation in articleGoogle Scholar

  • Riley, R. D., Simmonds, M. C. & Look, M. P. (2007). Evidence synthesis combining individual patient data and aggregate data: A systematic review identified current practice and possible methods. Journal of Clinical Epidemiology, 60, 431–439. doi: S0895-4356(06)00403-3 First citation in articleCrossref, Google Scholar

  • Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357. First citation in articleCrossref, Google Scholar

  • Rosenthal, R. & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82. doi: 10.1146/annurev.psych.52.1.59 First citation in articleCrossref, Google Scholar

  • Rothstein, H. R. (2008). Publication bias as a threat to the validity of meta-analytic results. Journal of Experimental Criminology, 4, 61–81. First citation in articleCrossref, Google Scholar

  • Schmid, C. H., Stark, P. C., Berlin, J. A., Landais, P. & Lau, J. (2004). Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. Journal of Clinical Epidemiology, 57, 683–697. First citation in articleCrossref, Google Scholar

  • Schmidt, F. L. & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings. Los Angeles, CA: Sage. First citation in articleGoogle Scholar

  • Schmidt, F. L. & Oh, I.-S. (2016). The crisis of confidence in research findings in Psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology, 4, 32–37. doi: 10.1037/arc0000029 First citation in articleCrossref, Google Scholar

  • Shadish, W. R. (2015). Introduction to the special issue on the origins of modern meta-analysis. Research Synthesis Methods, 6, 219–220. doi: 10.1002/jrsm.1148 First citation in articleCrossref, Google Scholar

  • Simmonds, M., Stewart, G. & Stewart, L. (2015). A decade of individual participant data meta-analyses: A review of current practice. Contemporary Clinical Trials, 45, 76–83. doi: 10.1016/j.cct.2015.06.012 First citation in articleCrossref, Google Scholar

  • Simmonds, M. C., Higgins, J. P. T., Stewart, L. S., Tierney, J. F., Clarke, M. J. & Thompson, S. G. (2005). Meta-analysis of individual patient data from randomized trials: A review of methods used in practice. Clinical Trials, 2, 209–217. doi: 10.1191/1740774505cn087oa First citation in articleCrossref, Google Scholar

  • Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Series B, 13, 238–241. First citation in articleGoogle Scholar

  • Sirin, S. R. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Education Research, 75, 417–453. First citation in articleCrossref, Google Scholar

  • Stewart, L. A., Clarke, M., Rovers, M., Riley, R. D., Simmonds, M., Stewart, G. & Tierney, J. F. (2015). Preferred reporting items for a systematic review and meta-analysis of individual participant data: The PRISMA-IPD statement. Journal of the American Medical Association (JAMA), 313, 1657–1665. doi: 10.1001/jama.2015.3656 First citation in articleCrossref, Google Scholar

  • Stewart, L. A. & Parmar, M. K. B. (1993). Meta-analysis of the literature or of individual patient data: Is there a difference? The Lancet, 341, 418–422. First citation in articleCrossref, Google Scholar

  • Stewart, L. A. & Tierney, J. F. (2002). To IPD or not to IPD? Evaluation & the Health Professions, 25, 76–97. First citation in articleCrossref, Google Scholar

  • Stöber, J. (1999). Die Soziale-Erwünschtheits-Skala-17 (SES-17): Entwicklung und erste Befunde zu Reliabilität und Validität [The Social Desirability Scale-17 (SDS-17): Development and first findings on reliability and validity]. Diagnostica, 45, 173–177. doi: 10.1026/0012-1924.45.4.173 First citation in articleLink, Google Scholar

  • Südkamp, A., Kaiser, J. & Möller, J. (2012). Accuracy of teachers’ judgments of students’ academic achievement: A meta-analysis. Journal of Educational Psychology, 104, 743–763. doi: 10.1037/a0027627 First citation in articleCrossref, Google Scholar

  • Sutton, A. J., Kendrick, D. & Coupland, C. A. C. (2008). Meta-analysis of individual- and aggregate-level data. Statistics in Medicine, 27, 651–669. doi: 10.1002/sim.2916 First citation in articleCrossref, Google Scholar

  • Televantou, I., Marsh, H. W., Kyriakidies, L., Nagengast, B., Fletcher, J. & Malmberg, L.-E. (2015). Phantom effects in school composition research: Consequences of failure to control biases due to measurement error in traditional multilevel models. School Effectiveness and School Improvement, 26, 75–101. First citation in articleCrossref, Google Scholar

  • Tierney, J. F., Vale, C., Riley, R., Smith, C. T., Stewart, L., Clarke, M. & Rovers, M. (2015). Individual Participant Data (IPD) meta-analysis of randomized controlled trials: Guidance on their use. PLoS Medicine, 12, e1001855. doi: 10.1371/journal.pmed.1001855 First citation in articleCrossref, Google Scholar

  • Van den Heuvel, E. R. & Griffith, L. E. (2016). Statistical harmonization methods in Individual Participants Data meta-analysis are highly needed. Biometrics & Biostatistics International Journal, 3, 00064. doi: 10.15406/bbij.2016.03.00064 First citation in articleCrossref, Google Scholar

  • Viechtbauer, W. (2007). Random-effects models and moderator analyses in meta-analysis. Zeitschrift für Psychologie/Journal of Psychology, 215, 104–121. First citation in articleLink, Google Scholar

  • Viechtbauer, W. & Chueng, M. W.-L. (2010). Outlier and influence diagnostics for meta-analysis. Research Synthesis Methods, 1, 112–125. doi: 10.1002/jrsm.11 First citation in articleCrossref, Google Scholar