Seven Easy Steps to Open Science

Background

“Open science” is an umbrella term used to refer to the concepts of openness, transparency, rigor, reproducibility, replicability, and accumulation of knowledge, all of which are considered fundamental features of the scientific endeavor. In recent years, psychological researchers have begun to adopt reforms to make their work better align with these principles and to address the current “credibility revolution” (Vazire, 2018). For example, the Society for the Improvement of Psychological Science (SIPS; https://improvingpsych.org/mission/) is a membership society founded to further promote improved methods and practices in the psychological research field.

The proposed open science reforms are largely a response to realizations that standard research practices undermine fundamental principles of high-quality and open science (e.g., Ioannidis, 2005; Open Science Collaboration, 2015; Simmons, Nelson, & Simonsohn, 2011). Most scientists agree that there is a reproducibility crisis, at least to some extent (Baker, 2016). However, not all psychological scientists have adopted the best practices recommended by experts to make science more reproducible (Ioannidis, Munafò, Fusar-Poli, Nosek, & David, 2014; O’Boyle, Banks, & Gonzalez-Mulé, 2014). In part, this is because current incentive structures are misaligned with fundamental best practices¹ (Bakker, van Dijk, & Wicherts, 2012; Higginson & Munafò, 2016). Furthermore, there is confusion, disagreement, and misinformation about what the best practices are, whether and why they are necessary, and how to implement them (Houtkoop et al., 2018). In response to this, researchers have produced many excellent resources considering each major facet of open science and methodological reforms. These resources provide detailed instruction, context, and relevant empirical evidence. However, they are sometimes technical, are distributed across different journals and domains of psychology, or may be difficult to identify and access by a diverse range of researchers. Students and academics with little background knowledge of open science may not easily find and make use of such resources. Indeed, lack of information about the resources available and the incentives for adopting gold-standard scientific practices have recently been identified as primary reasons for researchers in psychology not using such improved scientific approaches (Washburn et al., 2018). Thus, an accessible and consolidated guide that outlines the best openly accessible resources related to improved practices in (psychological) science is needed.

Choosing the focus of such an overview is difficult because of the constant evolution of what is considered psychological best practice. Furthermore, recommendations differ across research aims and disciplines, and even reasonable researchers disagree about what exactly constitutes “best practices.” This review therefore focuses on seven broad topics chosen so they may be flexibly applied depending on researcher preference and the current context of their academic work. Following Corker’s (2018) framing, we view open science as a set of behaviors, and this paper therefore provides curious beginners with the information they need to start implementing these behaviors in their research.

Objectives

In this article, we provide a comprehensive and concise introduction to open scientific practices and highlight resources that can help students and researchers with no background knowledge begin implementing such best practices. Following the format of an annotated reading list introduced by Etz, Gronau, Dablander, Edelsbrunner, and Baribault (2018), we curate and summarize reviews, tutorials, and metascientific research related to seven topics, which were selected by identifying themes in a public, crowdsourced list of readings on reproducibility and replicability maintained by Dan Simons and Brent Roberts (https://docs.google.com/document/d/14lBD0aZDPij2Z6AOpAharOAtmt6ZBI0EuF3_tu8m66I/edit). The seven topics selected are: open access, open data, preregistration, reproducible analyses, replications, and teaching open science. Our aim was to create an annotated reading list that included all topics commonly considered to be open science practices and that have been described in published guides or meta-scientific papers. For each topic, we provide an accessible summary anchored by one publicly available, published, peer-reviewed article, and suggest additional readings. In doing so, we aim to make open science practices both understandable and actionable to readers.

It is important to note that the transparency and robustness added by many open science practices do not always guarantee increased rigor. We initially also included a section on better practices in statistics and methodology, as both the analytic approach and the methodology of a research project are central determinants of the replicability and reproducibility of its research claims. Although we focus here on the adoption of open science practices, we encourage the reader to also carefully plan data-collection and analysis, be aware of the assumptions of their statistical models, and have a proper understanding of the statistical tools they use. For example, it might be useful to consider possible misconceptions and corresponding clarifications regarding widely used frequentist statistical tools, such as p-values, confidence intervals, and power (Greenland et al., 2016; Kass et al., 2016). It is also possible to adopt alternative approaches, such as the Bayesian framework (Wagenmakers et al., 2018). Either way, by implementing better practices in statistics and methodology, a researcher can increase the scientific value of their work by substantially enhancing the credibility of the inferences made. Furthermore, if a researcher is uncertain whether they have engaged in the best methodological or statistical practices, they may be more reluctant to maximize the transparency of their work (e.g., by publishing data or code). Thus, best practices in methods and statistics underlie many of the other factors discussed in the following sections.

Beginning with a broad introduction to open science, we move on to discuss practices that directly engage in making the process or products of science more reproducible, replicable, and transparent. Finally, we discuss replication research and teaching, two practices that fundamentally align with and support these values.

Source: Munafò et al. (2017) – “A Manifesto for Reproducible Science”

Open science does not refer to one set of specific rules; instead, it is a collection of several research practices that variously manifest themselves in different research contexts (Corker, 2018). Open science practices are valuable not only because they are transparent but also because they help improve the quality and accumulation of scientific knowledge. The first source highlighted in this paper provides a skilled overview of open science and related topics concerning reproducibility and replicability (see Appendix for further resources: Corker, 2018; Fecher & Friesike, 2014; Spellman, Gilbert, & Corker, 2017). Munafò et al. define open science as “the process of making the content and process of producing evidence and claims transparent and accessible to others” (p. 5). For example, research transparency and accessibility are essential for evaluating the credibility of both statistical evidence and scientific claims. The credibility of evidence depends in part on its reproducibility; given the same quantitative evidence (i.e., data) and the same statistical analysis, can the same result be obtained? The credibility of scientific claims also depends in part on their replicability; if an experiment is repeated with the same procedures, therefore generating new data, will the same result be obtained (for discussion of these definitions, see Plesser, 2018)? Neither reproducibility nor replicability can be evaluated without both transparency and accessibility of the research process and the evidence generated from it. Thus, open science practices are crucial to the most basic aspects of the scientific process.

Munafò et al. call attention to many factors that undermine reproducibility and replicability. Some are the direct result of non-transparency (e.g., of missing study details or inaccessible data), whereas others stem from suboptimal methodological practices and research design. Such factors may include flexibility in statistical analysis or post hoc hypothesizing, when hypotheses are changed to better fit the results found. Figure 1 provides a powerful visualization of threats to scientific credibility that may occur at every stage of the research process. Although many of these threats have been known for decades (e.g., Cohen, 1965), reforms in psychology have only recently begun to address them more widely. Below, we highlight two instances identified by Munafò et al. where progress is being made: (1) developing and refining new tools to protect researchers against threats to credibility, such as self-deception, and (2) creating new incentive structures that reward researchers for being transparent and making research and related products (e.g., data or code) accessible. These and other tools outlined in Munafò et al. are given full focus later in this annotated reading list.

Munafò et al. note that publications remain the primary scientific currency and that journals are more likely to publish novel, positive, and straightforward results rather than negative results or replications, ultimately rewarding science susceptible to false-positive results. However, such problematic incentive structures are changing. Since implementing publication badges for papers adopting open scientific practices, such as an Open Data badge, the journal Psychological Science has observed a large increase in articles sharing their data (see Open Data, Materials, and Code section). Funders are also adopting new research transparency requirements, and new funding opportunities are available that specifically focus on promoting reproducible research, meta-science, or replication studies (see Replication Research section). Another key part of the changing incentive structure is university hiring and promotion practices, which can encourage the use of unhelpful metrics, such as the journal impact factor (McKiernan et al., 2019). Increasingly, solicitations for academic appointments include language valuing open science practices (see https://osf.io/7jbnt).

Understanding the concept of open science is key to understanding the broad value of each topic reviewed in this paper. Whether by increasing or supporting rigor and transparency, each of the practices covered in subsequent parts of this paper can help individual researchers and psychological science as a whole engage in more replicable and reproducible research.

Source: Tennant et al. (2016) – “The Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review”

A basic and essential part of research transparency is openness regarding the publication and dissemination process (see Appendix for further resources: Chan et al., 2002; COPE, OASPA, DOAJ, & WAME, 2018; Piwowar et al., 2018). Open Access (OA) refers to the unrestricted public availability of research products. Tennant et al., our second highlighted source, provide a thorough discussion of the history, forms, processes, and consequences of OA. The article discusses relevant empirical evidence related to the strengths and weaknesses of OA from the perspective of different stakeholders.

OA aims to remove barriers to accessing and distributing research and its relevant products. Enabled by advances in communication technology and coined by the Budapest Open Access Initiative (Chan et al., 2002), OA is defined as the free, public availability of a research product on the internet for distribution and re-use with acknowledgment. It therefore shares many functional similarities with the Creative Commons attribution licence CC-BY (Tennant et al., 2016). OA is typically used in reference to published journal articles, but any output could be OA, including student works and study materials, code, and data.

Most psychologists use the taxonomy described in Harnad et al. (2008) to define OA. Harnad distinguishes two non-exclusive routes to OA, while other taxonomies consider whether a paper has been made public by a fully OA journal or whether a work is explicitly licensed for reuse (Piwowar et al., 2018). The gold route refers to OA in the formal publication system: works are made publicly available at the point of publication (i.e., by the publishers themselves). The green route refers to authors self-archiving: works are made publicly available by the people who created them. Green OA includes self-archiving of works that have been peer-reviewed, as well as those that have not (e.g., pre-prints). Tennant et al. describe that across academia, only 25% of articles are OA through green or gold routes, despite 79% of journals indexed by SCOPUS endorsing author self-archiving in some format.

The accumulation of scientific knowledge is facilitated by wide access to research products so that anyone can access, build on, and respond to prior work. Tennant et al. discuss two specific effects of OA on academic research. First, OA works are used more: within academia, OA works are cited between 36% and 600% more than non-OA works, although opinions differ about the causal nature of this citation advantage. Outside of academia, OA works are given more coverage by journalists and discussed more in non-scientific settings (e.g., on social media). Second, OA works facilitate meta-research because they enable the use of automated text- and data-mining tools. Such options support meta-researchers in investigating research findings, which in turn helps us better understand what existing research can (and cannot) tell us.

Tennant et al. also discuss the complex economic impacts of OA on publishers, institutions, private research companies, and funders. They discuss the societal impact of OA, exploring the ethical implications of public access to research products, the importance of OA for people in low- and middle-income countries, and deceptive publishing practices related to the OA movement. Finally, they highlight connections between OA and the open science movement, and consider several directions for future research.

Those looking to make their research products OA have several options. Any product of research that authors own can be self-archived for free (green OA). To self-archive, authors should confirm ownership of the to-be-posted material, obtain permission from co-authors, and decide whether and how to licence the work before posting it on a public, stable institutional or subject repository, such as PsyArXiv (for a complete list see http://v2.sherpa.ac.uk/opendoar/). Authors can also make published works available (gold OA) by publishing in an exclusively OA journal (https://doaj.org/) or a hybrid journal that supports gold OA. For information about OA policies at specific journals, see the SHERPA RoMEO database (http://www.sherpa.ac.uk/romeo/index.php).

Source: Klein et al. (2018) – “A Practical Guide for Transparency in Psychological Science”

Another basic form of research transparency is making a study’s materials, data, and/or analysis code publicly available (see Appendix for further resources: Gilmore, Kennedy, & Adolph, 2018; Levenstein & Lyle, 2018). According to the sixth edition of the APA Publication Manual, researchers must be able and willing to share their raw data with editors during the manuscript review process and with “other qualified professionals,” for 5 years after publication of their paper (APA, 2010). This standard is well grounded in several excellent reasons for sharing data that make work more reproducible and replicable, with a focus on enabling more reproducible and replicable research. For example, sharing data enables: verification, the process of checking results to minimize errors and bias in published work, and analytic reproducibility, the process of checking which data analysis steps were executed for cleaning and analyzing the data at hand.

In practice, data sharing is not common. In a descriptive survey of 600 researchers, Houtkoop et al. (2018) found that only 10% of respondents had ever publicly shared data. The vast majority of researchers fail to share their data either because they do not see it as necessary or important, or because they do not know how. Klein et al. provide practical guidelines to help researchers navigate the process of preparing and sharing their research. The authors make a strong case for why sharing data should be common practice in science. Two central arguments are that (1) science is based on verifiability not trust – one wants to be informed about every detail in an analysis instead of relying solely on the authors word for it, and (2) analytic reproducibility can only be achieved when data are openly available – re-running analyses to identify errors is a key ingredient of a healthy research cycle. Open data, materials, and code thus help increase the credibility of the research process, and boost the efficiency of scientific discovery and verifiability.

Klein et al. move on to tackle the what, when, where, and how of sharing products of research. As a bare minimum, they recommend that the data files the analysis is based on are shared openly. Furthermore, any additional information about an experiment (e.g., its stimuli, tasks, descriptions of instruments) that can be shared is helpful, although researchers should ultimately aim to share detailed empirical protocols and computer code that enables a fully reproducible analysis. Anonymizing data and explicitly reporting why transparent sharing might not be possible (e.g., data collected from sensible populations) are also important steps in this process. Regarding when to share research products, the short answer is better late than never. The longer answer may be connected to the relevant standards of journals (e.g., some journals ask researchers to share data once a paper goes into press) or funding agencies (e.g., some funders mandate that the data should be made available with closure of a project). Researchers should also consider how they share their data. Data should be shared in a FAIR manner – findable, accessible, interoperable, and reusable. Findable and accessible are mainly concerned with where data are uploaded. Considerations include the availability of persistent DOIs, metadata, tracking of data re-use, licensing, access control, and long term availability. One service that provides these features is the Open Science Framework (https://osf.io). Interoperable and reusable highlight the importance of thinking about data format (proprietary, e.g., Microsoft Word, vs. non-proprietary, e.g., text files) and how such formats might change in the future. One important way to ensure that materials, data, and code are reusable is through detailed documentation. This helps future users (including the original researcher, at a later point in time) understand the functioning of the materials, the structure and format of the data, and how the code processes this data to arrive at the statistical results in a paper. Some practical recommendations on data management are provided by Schönbrodt, Gollwitzer, and Abele-Brehm (2017).

Sharing data, materials, and analysis code is a basic open science practice that researchers can engage in to the extent it makes sense for their research context. Doing so, as Klein et al. note, not only makes work more replicable and reproducible but also increases efficiency, reduces errors, and brings benefits to the broader scientific community by increasing collective efficiency and credibility.

Source: Wilson et al. (2017) – “Good Enough Practices in Scientific Computing”

Another fundamental open science practice is producing reproducible analyses. This entails providing the resources (i.e., open data and open code) that allow others to exactly generate the results reported in the final research product. Code and data must also remain safe, organized, and accessible over time. Our fourth highlighted source therefore focuses on what computational skills are necessary to be a productive and reproducible scientist in the 21st century (see Appendix for further resources: Brown et al., 2014; Goh, Hall, & Rosenthal, 2016; Poldrack et al., 2017; Software Carpentry Workshops). Although most researchers do not consider themselves software developers, the vast majority now use computers to manage and store their data, collaborate with colleagues via online networks, and oftentimes write code to conduct their data analysis. Wilson et al. highlight how – in an age where scientific work is mediated by technology – computing best practices are as integral to robust research practices as fundamental lab techniques. They outline the basic scientific computing skills that are “good enough” to ensure that research is not just reproducible but also efficient, transparent, and accessible in the future.

Wilson et al. cover six different aspects to scientific computing: data management, software, collaboration, project organization, tracking changes, and writing manuscripts. Although every aspect might not apply to every researcher, any academic will likely find many of these aspects applicable to their daily work. For example, the importance of saving raw data separately from any manipulated versions of such data is highlighted. Furthermore, the data themselves should be backed-up in multiple locations (see also Spellman et al., 2017). This practice sounds simple and commonplace, but errors in raw data storage have caused multiple high-profile retractions in recent years (including a study published in Nature where the co-authors could not agree on the nature of their original data file; for retraction, see Brown et al., 2013; for additional discussion, see Oransky, 2013).

Wilson et al. also suggest certain practices regarding project organization. For example, each research project should have its own directory (folder) on a computer. Relevant text documents, raw data, and cleaned data files should then be put in separate sub-directories. Further, they discuss different techniques to keep track of work, either by using version control software or by copying the entire project folder whenever substantial changes are implemented. Although these practices may seem excessive initially, copying entire folders allows work and progress to be backed up and tracked quickly and easily. Wilson et al. stress: Data are cheap, time is expensive.

Many labs use different software for their daily work; Wilson et al. highlight aspects that all researchers should strive to fulfill, regardless of the software they choose to use. All steps of data processing should be recorded, either by coding analyses in open-source programs like R or Python or by “pasting” SPSS analytical steps into a reproducible syntax file. Wilson et al. also highlight the importance of not duplicating code – to decrease the potential for errors and increase usability. When sharing materials and data, researchers should explicitly decide on licensing agreements (St. Laurent, 2004). Such licensing agreements and data sharing practices are especially important when submitting data to “reputable DOI-issuing repositories”, such as Figshare, Dryad, and Zenodo. Doing so will allow researchers to share their data and receive recognition for it when re-used by others. Overall, implementing good coding and data storage practices in daily research is crucial to ensure that analyses are reproducible in the future, not just by others but also by the researchers themselves.

Increasing numbers of funders and institutions are asking for data management plans (e.g., Swiss National Science Foundation, 2018) and supporting the move to more “coding based” data analysis options; however, researchers are often not informed about what is necessary to ensure that all aspects of their research are reproducible. Wilson et al. provide a key starting point for those unsure about how to begin or those wanting to improve. Because nearly every researcher now works with computers on a daily basis, but rarely has formal training to do so responsibly, this paper will be critical reading for many. Making analyses and data as reproducible, clear, sustainable, and accessible as is feasible is an important step toward implementing and encouraging open scientific principles.

Source: Wagenmakers, Wetzels, Borsboom, van der Maas, and Kievit (2012) – “An Agenda for Purely Confirmatory Research”

Preregistration is an open science practice that protects researchers from some of the influence of misaligned incentives, allowing them to be more transparent about their analytic decision-making. Our fifth highlighted source focuses on preregistration (see Appendix for further resources: Chambers, Feredoes, Muthukumaraswamy, & Etchells, 2014; Nosek, Ebersole, DeHaven, & Mellor, 2018; van’t Veer & Giner-Sorolla, 2016). In this paper, Wagenmakers et al. call for a widespread adoption of preregistration, arising from the need to promote purely confirmatory research and transparently demarcate exploratory research. They begin by outlining the main limitations and pressures that lead researchers to overstate the evidentiary value of their results: cognitive biases, particularly confirmation and hindsight bias, and the pressure to publish large quantities of predominantly positive results. They emphasize that biases are inherently human and that succumbing to publication pressures is rational given the academic system. Although this situation is understandable when examining the motivations of any individual researcher, on a field-wide level, it can produce a scientific literature that is uninterpretable, populated by results that are oftentimes inaccurate or less compelling than claimed. “Research findings that do not replicate are worse than fairy tales; with fairy tales the reader is at least aware that the work is fictional” (Wagenmakers et al., 2012, p. 633). Wagenmakers et al. argue that the best way to remedy this situation is to encourage researchers to commit to hypotheses and analysis plans before they interact with their data, thus promoting a scientific system less governed by biases and publication pressures.

To understand the power of preregistration, it is key to examine the distinction between confirmatory and exploratory research (De Groot, 1956/2014). For research to be confirmatory, hypotheses and analyses must be specified before data collection and/or at a time when the researcher has no direct access to the data. As the term “confirmatory research” suggests, this type of research confirms hypotheses. In contrast, exploratory research focuses on generating hypotheses. Both exploratory and confirmatory research are important for science to progress, but they must be correctly and transparently labeled. Wagenmakers et al. argue that “almost no psychological research is conducted in a purely confirmatory fashion” (Wagenmakers et al., 2012, p. 633). They list common researcher degrees of freedom (Simmons et al., 2011), that is, possible points of analytic flexibility, such as cherry-picking which dependent variables or results to report or constructing hypotheses that best fit the data (i.e., HARKing, Kerr, 1998; see Figure 1). The results of such exploration of the data – whether intentional or unintentional – are often falsely labeled as confirmatory, increasing false-positive rates. They argue that for statistical results to be reliable, the practice of presenting exploratory research as confirmatory must be comprehensively prevented. The solution lies in preregistration: researchers committing to the hypotheses, study design, and analyses before the data are accessible. In their paper, Wagenmakers et al. present an exemplary preregistered replication as an illustration of this practice.

Wagenmakers et al. contend that preregistration is a straightforward solution to prevent researchers from presenting exploratory results as confirmatory, thus counteracting cognitive and publication biases. However, preregistration is no panacea; vague preregistrations can leave room for unreported exploration (Veldkamp et al., 2018) and can still be subject to the pressures of publication bias. The concept of “registered reports” (Chambers et al., 2014) might offer a better solution. Here, the publishability of a study is decided based on the preregistered study design and analysis plan (Stage 1 review). Following successful peer-review of the study design and protocol prior to data collection, a study is given in principle acceptance by the journal. This means that the study will be published regardless of the outcome, so long as the process adheres to the peer reviewed preregistered study design and analysis plan. Any further exploration must be clearly stated as such. Thus, both the rigor of and adherence to a preregistration is evaluated (Stage 2 review) and ensured in the registered report format.

Preregistration is a practice that increases the transparency of analytic decisions. It constrains the effect of biases and allows readers to interpret results in light of the broader analytic context. This, in turn, supports rigorous scientific research and enables more replicable and reproducible work.

Source: Zwaan, Etz, Lucas, and Donnellan (2018) – “Making Replication Mainstream”

Along with promoting the adoption of practices that directly increase transparency, the open science movement has also boosted confidence in the replicability of scientific results. Therefore, our sixth highlighted source discusses replication research, a key mechanism for encouraging the stability and generalisability of psychological phenomena (see Appendix for further resources: Brandt et al., 2014; Makel, Plucker, & Hegarty, 2012; Schmidt, 2009). Replication can increase the generalizability and veracity of findings but also incite controversy. In this integrative summary of replication in psychology, Zwaan and colleagues review definitions and types of replications, commonly expressed concerns about replication, and responses and strategies to resolve these concerns, while also providing options for the statistical evaluation of replication attempts. Despite the intense debate over the intent or need of individual replication attempts, replication is fundamental to the scientific endeavor. In fact, the authors emphasize that “a finding needs to be repeatable to count as a scientific discovery. Second, research needs to be reported in such a manner that others can reproduce the procedures” (p. 2). Without these components, a finding cannot inform scientific thinking.

There are different types of replication with varying goals and contributions. Direct replications seek to duplicate the necessary elements that produced the original finding, whereas conceptual replications purposefully change at least one component of the original procedure, for example, the sample or measure. Direct and conceptual replications serve different purposes; direct replications assess whether similar findings are reproduced in subsequent attempts, whereas conceptual replications assess whether previous findings are reproduced when tested under different conditions. Relatedly, Zwaan et al. suggest that “conceptual replication” is misleading and that labeling this form of replication as “alternative test” (p. 4) more clearly articulates the concept and leads to less conflation with direct replication. Regardless of the type of replication, conducting replications could be greatly facilitated if original research made greater use of open materials as discussed above. Similarly, replications are excellent examples of the type of research that are easily preregistered, as the methods, analytic plan, and expected results are already known.

At 61 pages, this is the longest source covered here. However, the full publication comprises a 13-page article, followed by extensive open peer commentary from independent authors, and a rejoinder from the original authors. The primary article can stand alone, but the open commentaries and rejoinder inform the ongoing replication conversation.

Zwaan et al. consider several commonly expressed concerns about replication and follow each of these sections with brief rebuttals. The specific concerns include: the role of context variability (e.g., does it matter whether the researcher wears a labcoat?), the scientific value of direct replications, the feasibility of direct replications, whether replications distract from more fundamental existing problems (e.g., publication bias), how replications affect the reputation of researchers, and the lack of standardized methods for evaluating replication results. Additionally, there may be misalignment of incentives for what benefits a field and what benefits individual researchers. Such misalignment may inhibit adherence to scientific norms and values, including the value of replication. Nevertheless, Zwaan et al. make the case for why no theoretical or statistical obstacles prevent replications from becoming a mainstream part of psychological research. They argue that in conjunction with other methods, replication helps clarify when confidence in results should be present as well as build theory, both of which help align psychology with the scientific method. From this perspective, producing replicable work and assessing the replicability of findings plays a central role in any credible scientific domain and fundamentally aligns with values of open science. Individual researchers can contribute to greater incorporation of replication into practice by conducting replications themselves and facilitating the replication of their own work through the use of open and reproducible materials, data, and code.

Source: Chopik, Bremner, Defever, and Keller (2018) – “How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science”

Given that one of the main barriers to adopting open practices is a lack of education, our final highlighted source considers how best to integrate and promote open science through teaching (see Appendix for further resources: Frank & Saxe, 2012; Janz, 2016). Chopik et al. begin by asking: why are we not teaching our undergraduate students about the replication crisis and the related methodological reforms to? The authors observed that instructors often have difficulties integrating these topics into their lectures or are unsure where to start. This led to the development of a 1-hour lecture introducing students to causes and impact of psychology’s replication problem, attempts to evaluate the reproducibility and replicability of findings, and proposed solutions. The full lecture slides and script are openly available to provide a tool for instructors to keep themselves and their students up to date with open science (https://osf.io/mh9pe). Chopik et al. assessed the impact of the lecture by comparing students’ evaluation of psychology as a science and psychology reported in the media before and after the lecture. The need for such a lecture was evident; only 31% of their students had heard of the replication crisis prior to the lecture. Because 68% of students were currently or had previously taken a stats class, this result also highlights the need to integrate this information into formal statistics courses. Following the lecture, the majority (above 80%) of students agreed on the importance of methodological considerations (several of which are discussed previously in this paper), including reporting studies that “didn’t work out,” making data publically available, and choosing a sample size a priori. Students trusted psychological research less following the lecture. However, they also viewed psychology more similar to the natural or hard sciences, and this lecture did not reduce intentions to study at a graduate level. Many students agreed that psychology has a problem replicating results and that poor incentive structures can undermine science; however, nearly all students agreed that replication issues were not confined to psychology. Not all undergraduate psychology students will become researchers, which may cause some to question the value of teaching the replication crisis and open science. However, even if the intention is not to train future researchers, we must certainly train future consumers of research. Post-lecture, students were more critical of research presented in the media and more able to identify aspects that contribute to more replicable results. Being able to identify reproducible research and appreciate openness and rigor is an essential skill to be a good consumer of psychological research. A 1 hr lecture has the potential to improve this comprehension.

Teaching open science and the replication crisis is a pedagogical challenge. Thankfully, many teachers and researchers are providing resources to facilitate training students in open science. Hawkins et al. (2018) summarizes 11 graduate student-led pre-registered replications of studies in Psychological Science and offer an insightful overview on how this approach may be adapted for classes of different levels. Several Massive Open Online Courses (MOOCs) and similar online resources have been developed to provide a grounding in open science, including Transparent and Open Social Science Research (https://www.bitss.org/mooc-parent-page/), the Open Science MOOC (https://github.com/OpenScienceMOOC), and resources by the EU project FOSTER, including their Open Science Training Handbook (https://book.fosteropenscience.eu/en/). Finally, a recent initiative to support and recognise the teaching of open science practices is the Framework for Open and Reproducible Research Training (FORRT; https://forrt.netlify.com/). FORRT provides a basis for benchmarking course content covered in relation to open and reproducible research. Building on this, FORRT collates teaching-based resources aiming to help develop open and reproducible courses and further integrate these improved practices, while recognizing and supporting the outstanding contribution of teachers. In line with Chopik et al., these and similar other resources may be adapted and implemented into existing courses, greatly reducing the burden on instructors. Making use of resources like this in teaching can support the next generation of researchers in learning and incorporating open science practices from the very beginning.