Replikationskrise, p-hacking und Open Science
Eine Umfrage zu fragwürdigen Forschungspraktiken in studentischen Projekten und Impulse für die Lehre
Abstract
Zusammenfassung. In den letzten Jahren gab es innerhalb der Psychologie eine intensive Auseinandersetzung mit den Auswirkungen der Replikationskrise sowie dem hieraus entstandenen Diskurs über die Weiterentwicklung der Disziplin. Als ein Grund für die mangelnde Replizierbarkeit psychologischer Forschung wurde die Verwendung fragwürdiger Forschungspraktiken (eng. QRPs) identifiziert. Während es umfangreiche Untersuchungen zur Prävalenz von QRPs unter Wissenschaftler*innen gibt, ist bisher wenig über die Verbreitung dieser Praktiken unter Studierenden bekannt. Mit der hier vorgestellten Arbeit wurde erstmals eine größere Befragung unter 1397 Psychologie-Studierenden im deutschsprachigen Raum durchgeführt, um die Verbreitung von QRPs in studentischen Projekten sowie den aktuellen Stand der akademischen Lehre in Bezug auf die Replikationskrise und Open Science zu erheben. Die gemeinsame Betrachtung der Lehre und des Einsatzes fragwürdiger Forschungspraktiken versprechen Aufschluss darüber, wie die psychologische Lehre mit dem empirischen Vorgehen der Studierenden zusammenhängt. Die Ergebnisse zeigen, dass QRPs auch in studentischen Projekten vorkommen, wobei große Unterschiede in der Verbreitung einzelner QRPs bestehen. Auch zwischen den verschiedenen Projekttypen zeigten sich Unterschiede, so war die Anwendung von QRPs in Experimentalpraktika am stärksten und in Masterarbeiten am schwächsten ausgeprägt. Unsere Daten weisen insgesamt darauf hin, dass die selbstberichtete Verbreitung von QRPs über den Studienverlauf abnimmt. Zudem scheint ein Großteil der Studierenden bereits mit der Thematik der Replikationskrise in der Lehre in Berührung gekommen zu sein. Deren Behandlung findet größtenteils in der Methodenlehre und weniger in inhaltlich spezialisierten Lehrveranstaltungen statt. Wir geben abschließend Impulse zur Weiterentwicklung der psychologischen Lehre, in denen die Prinzipien der Offenheit, Transparenz und Kollaboration beim Hervorbringen inhaltlich robuster Forschung bereits während des Studiums im Vordergrund stehen.
Abstract: In recent years, there has been an intensive debate within psychology about the conclusions that should be drawn because of the replication crisis. The use of questionable research practices (QRPs) was identified as one reason for problems concerning replicability. While there are extensive studies on the prevalence of QRPs among scientists, little is known about their occurrence among students. This article presents the first large-scale survey among 1,397 psychology students in the German-speaking countries to investigate the occurrence of QRPs in student projects and the current state of academic teaching regarding the replication crisis and open science. The joint examination of teaching and the use of questionable research practices serves to provide information on how the teaching of psychology is related to the empirical approach of students. The results reveal that questionable research practices do occur in student projects, albeit with large differences in the occurrence of specific QRPs. We also found differences in the incidence of QRPs between different project types: QRP usage was most frequent in empirical internships and least frequent in master’s theses. Our data suggest that the extent of reported QRPs generally decreases as students progress in their study programs. In addition, most students seem to have already come into contact with the replication crisis in teaching, mostly in methodology classes. Finally, we provide impulses for the further development of the teaching of psychology. The principles of openness, transparency, and collaboration play an important role in these recommendations, which are aimed at teaching and producing robust research from the very beginning.
Literatur
(2017). Questionable research practices among Italian research psychologists. PLoS ONE, 12 (3) https://doi.org/10.1371/journal.pone.0172792
(2020). Rmarkdown: dynamic documents for R. https://github.com/rstudio/rmarkdown
(2019). Statistical significance gives bias a free pass. European Journal of Clinical Investigation, 49 (12) https://doi.org/10.1111/eci.13176
(2020). Formr: a study framework allowing for automated feedback generation and complex longitudinal experience-sampling studies using R. Behavior Research Methods, 52, 376 – 387. https://doi.org/10.3758/s13428-019-01236-y
(2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543 – 554. https://doi.org/10.1177/1745691612459060
(2016). Editorial: Evidence on questionable research practices: the good, the bad, and the ugly. Journal of Business and Psychology, 31, 323 – 338. https://doi.org/10.1007/s10869-016-9456-7
(2019). Redesigning methods curricula for reproducibility. https://psyteachr.github.io/sips2019/
(2015). Fitting linear mixed-effects models using {lme4}. Journal of Statistical Software, 67 (1), 1 – 48. https://doi.org/10.18637/jss.v067.i01
(2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2, 637 – 644. https://doi.org/10.1038/s41562-018-0399-z
(2018). How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology, 45, 158 – 163. https://doi.org/10.1177/0098628318762900
(1994). The earth is round (p < .05). American Psychologist, 49, 997 – 1003. https://doi.org/10.1037/0003-066X.49.12.997
(2005). Empfehlungen der DGPs zur Einrichtung von B.Sc.-M.Sc.-Studiengängen in Psychologie an den Universitäten. https://www.dgps.de/index.php?id=143{\&}tx{\_}ttnews{\%}5Btt{\_}news{\%}5D=993{\&}cHash=03097507021fb333802bbeaf76963033
(2019). Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity. PLoS ONE, 14 (5), 1 – 23. https://doi.org/10.1371/journal.pone.0216125
(2018). Four reasons to prefer Bayesian analyses over significance testing. Psychonomic Bulletin and Review, 25, 207 – 218. https://doi.org/10.3758/s13423-017-1266-z
(2017). Academic research in the 21st century: maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environmental Engineering Science, 34 (1), 51 – 61. https://doi.org/10.1089/ees.2016.0223
(2018). The regression trap and other pitfalls of replication science—illustrated by the report of the Open Science Collaboration. Basic and Applied Social Psychology, 40 (3), 115 – 124. https://doi.org/10.1080/01973533.2017.1421953
(2016). Questionable research practices revisited. Social Psychological and Personality Science, 7 (1), 45 – 52. https://doi.org/10.1177/1948550615612150
(2016). Comment on „Estimating the reproducibility of psychological science”. Science, 351 (6277) https://doi.org/10.1126/science.aad7243
(2016). Qualitative research methods: when to use them and how to judge them. 31, 498 – 501. https://doi.org/10.1093/humrep/dev334
(2020). How to teach open science principles in the undergraduate curriculum – the Hagen cumulative science project. Psychology Learning & Teaching, 19 (1), 91 – 106. https://doi.org/10.1177/1475725719868149
(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524 – 532. https://doi.org/10.1177/0956797611430953
(2018). Editorial: Themenheft Replizierbarkeit. Psychologische Rundschau, 69, 1 – 2. https://doi.org/10.1026/0033-3042/a000385
(2018). Many Labs 2: investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443 – 490. https://doi.org/10.1177/2515245918810225
(2018). Positionspapier der Psychologie-Fachschaften-Konferenz (PsyFaKo) zum Thema „Replikationskrise und Open Science in der Psychologie“. https://psyfako.org/index.php/positionspapiere/
(2018). Questionable research practices in student final theses – Prevalence, attitudes, and the role of the supervisor’s perceived attitudes. PLoS ONE, 13 (8) https://doi.org/10.1371/journal.pone.0203470
(2017). {lmerTest} package: tests in linear mixed effects models. Journal of Statistical Software, 82 (13), 1 – 26. https://doi.org/10.18637/jss.v082.i13
(2011). Computational modeling in cognition: principles and practice. SAGE. https://doi.org/10.4135/9781483349428
(2019). Abandon statistical significance. The American Statistician, 73 (S1), 235 – 245. https://doi.org/10.1080/00031305.2018.1527253
(2018). Robust research needs many lines of evidence. Nature, 553, 399 – 401. https://doi.org/10.1038/d41586-018-01023-3
(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115, 2600 – 2606. https://doi.org/10.1073/pnas.1708274114
(2020). What is replication? PLoS Biology, 18 (3), 1 – 8. https://doi.org/10.1371/journal.pbio.3000691
(2014). Registered reports. Social Psychology, 45 (3), 137 – 141. https://doi.org/10.1027/1864-9335/a000192
(2015). Estimating the reproducibility of psychological science. Science, 349 (6251), aac4716. https://doi.org/10.1126/science.aac4716
(2020). Psychology as a robust science [Course Syllabus]. https://www.amyorben.com/docs/syllabus.pdf
(2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/
(2020). Teaching good research practices: protocol of a research master course. Psychology, Learning & Teaching, 19 (1), 45 – 59. https://doi.org/10.1177/1475725719858807
(2018). Forschungstransparenz als hohes wissenschaftliches Gut stärken. Psychologische Rundschau, 69, 37 – 44. https://doi.org/10.1026/0033-3042/a000386
(2017). Der Umgang mit Forschungsdaten im Fach Psychologie: Konkretisierung der DFG-Leitlinien. Psychologische Rundschau, 68, 20 – 35. https://doi.org/10.1026/0033-3042/a000341
(2016). Playing with data—or how to discourage questionable research practices and stimulate researchers to do things right. Psychometrika, 81 (1), 1 – 15. https://doi.org/10.1007/s11336-015-9446-0
(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359 – 1366. https://doi.org/10.1177/0956797611417632
(2014). Expectations for replications: are yours realistic? Perspectives on Psychological Science, 9, 305 – 318. https://doi.org/10.1177/1745691614528518
(2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9 (1), 59 – 71. https://doi.org/10.1177/1745691613514450
(2016). Inflation von falsch-positiven Befunden in der psychologischen Forschung. Psychologische Rundschau, 67, 163 – 174. https://doi.org/10.1026/0033-3042/a000296
(2017).
The need for Bayesian hypothesis testing in psychological science . In S. O. LilienfeldI. Waldman (Eds.), Psychological science under scrutiny: recent challenges and proposed solutions. John Wiley; Sons.(2019). Publishing research with undergraduate students via replication work: the collaborative replications and education project. 10. https://doi.org/10.3389/fpsyg.2019.00247
(2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid P-hacking. Frontiers in Psychology, 7 (NOV), 1 – 12. https://doi.org/10.3389/fpsyg.2016.01832
(2017). Tidyverse: easily install and load the ’tidyverse’. https://cran.r-project.org/package=tidyverse
