Skip to main content
Open Access

Seven Easy Steps to Open Science

An Annotated Reading List

Published Onlinehttps://doi.org/10.1027/2151-2604/a000387

Abstract

Abstract. The open science movement is rapidly changing the scientific landscape. Because exact definitions are often lacking and reforms are constantly evolving, accessible guides to open science are needed. This paper provides an introduction to open science and related reforms in the form of an annotated reading list of seven peer-reviewed articles, following the format of Etz, Gronau, Dablander, Edelsbrunner, and Baribault (2018). Written for researchers and students – particularly in psychological science – it highlights and introduces seven topics: understanding open science; open access; open data, materials, and code; reproducible analyses; preregistration and registered reports; replication research; and teaching open science. For each topic, we provide a detailed summary of one particularly informative and actionable article and suggest several further resources. Supporting a broader understanding of open science issues, this overview should enable researchers to engage with, improve, and implement current open, transparent, reproducible, replicable, and cumulative scientific practices.

Background

“Open science” is an umbrella term used to refer to the concepts of openness, transparency, rigor, reproducibility, replicability, and accumulation of knowledge, all of which are considered fundamental features of the scientific endeavor. In recent years, psychological researchers have begun to adopt reforms to make their work better align with these principles and to address the current “credibility revolution” (Vazire, 2018). For example, the Society for the Improvement of Psychological Science (SIPS; https://improvingpsych.org/mission/) is a membership society founded to further promote improved methods and practices in the psychological research field.

The proposed open science reforms are largely a response to realizations that standard research practices undermine fundamental principles of high-quality and open science (e.g., Ioannidis, 2005; Open Science Collaboration, 2015; Simmons, Nelson, & Simonsohn, 2011). Most scientists agree that there is a reproducibility crisis, at least to some extent (Baker, 2016). However, not all psychological scientists have adopted the best practices recommended by experts to make science more reproducible (Ioannidis, Munafò, Fusar-Poli, Nosek, & David, 2014; O’Boyle, Banks, & Gonzalez-Mulé, 2014). In part, this is because current incentive structures are misaligned with fundamental best practices1 (Bakker, van Dijk, & Wicherts, 2012; Higginson & Munafò, 2016). Furthermore, there is confusion, disagreement, and misinformation about what the best practices are, whether and why they are necessary, and how to implement them (Houtkoop et al., 2018). In response to this, researchers have produced many excellent resources considering each major facet of open science and methodological reforms. These resources provide detailed instruction, context, and relevant empirical evidence. However, they are sometimes technical, are distributed across different journals and domains of psychology, or may be difficult to identify and access by a diverse range of researchers. Students and academics with little background knowledge of open science may not easily find and make use of such resources. Indeed, lack of information about the resources available and the incentives for adopting gold-standard scientific practices have recently been identified as primary reasons for researchers in psychology not using such improved scientific approaches (Washburn et al., 2018). Thus, an accessible and consolidated guide that outlines the best openly accessible resources related to improved practices in (psychological) science is needed.

Choosing the focus of such an overview is difficult because of the constant evolution of what is considered psychological best practice. Furthermore, recommendations differ across research aims and disciplines, and even reasonable researchers disagree about what exactly constitutes “best practices.” This review therefore focuses on seven broad topics chosen so they may be flexibly applied depending on researcher preference and the current context of their academic work. Following Corker’s (2018) framing, we view open science as a set of behaviors, and this paper therefore provides curious beginners with the information they need to start implementing these behaviors in their research.

Objectives

In this article, we provide a comprehensive and concise introduction to open scientific practices and highlight resources that can help students and researchers with no background knowledge begin implementing such best practices. Following the format of an annotated reading list introduced by Etz, Gronau, Dablander, Edelsbrunner, and Baribault (2018), we curate and summarize reviews, tutorials, and metascientific research related to seven topics, which were selected by identifying themes in a public, crowdsourced list of readings on reproducibility and replicability maintained by Dan Simons and Brent Roberts (https://docs.google.com/document/d/14lBD0aZDPij2Z6AOpAharOAtmt6ZBI0EuF3_tu8m66I/edit). The seven topics selected are: open access, open data, preregistration, reproducible analyses, replications, and teaching open science. Our aim was to create an annotated reading list that included all topics commonly considered to be open science practices and that have been described in published guides or meta-scientific papers. For each topic, we provide an accessible summary anchored by one publicly available, published, peer-reviewed article, and suggest additional readings. In doing so, we aim to make open science practices both understandable and actionable to readers.

It is important to note that the transparency and robustness added by many open science practices do not always guarantee increased rigor. We initially also included a section on better practices in statistics and methodology, as both the analytic approach and the methodology of a research project are central determinants of the replicability and reproducibility of its research claims. Although we focus here on the adoption of open science practices, we encourage the reader to also carefully plan data-collection and analysis, be aware of the assumptions of their statistical models, and have a proper understanding of the statistical tools they use. For example, it might be useful to consider possible misconceptions and corresponding clarifications regarding widely used frequentist statistical tools, such as p-values, confidence intervals, and power (Greenland et al., 2016; Kass et al., 2016). It is also possible to adopt alternative approaches, such as the Bayesian framework (Wagenmakers et al., 2018). Either way, by implementing better practices in statistics and methodology, a researcher can increase the scientific value of their work by substantially enhancing the credibility of the inferences made. Furthermore, if a researcher is uncertain whether they have engaged in the best methodological or statistical practices, they may be more reluctant to maximize the transparency of their work (e.g., by publishing data or code). Thus, best practices in methods and statistics underlie many of the other factors discussed in the following sections.

Beginning with a broad introduction to open science, we move on to discuss practices that directly engage in making the process or products of science more reproducible, replicable, and transparent. Finally, we discuss replication research and teaching, two practices that fundamentally align with and support these values.

Open Science

Source: Munafò et al. (2017) – “A Manifesto for Reproducible Science”

Open science does not refer to one set of specific rules; instead, it is a collection of several research practices that variously manifest themselves in different research contexts (Corker, 2018). Open science practices are valuable not only because they are transparent but also because they help improve the quality and accumulation of scientific knowledge. The first source highlighted in this paper provides a skilled overview of open science and related topics concerning reproducibility and replicability (see Appendix for further resources: Corker, 2018; Fecher & Friesike, 2014; Spellman, Gilbert, & Corker, 2017). Munafò et al. define open science as “the process of making the content and process of producing evidence and claims transparent and accessible to others” (p. 5). For example, research transparency and accessibility are essential for evaluating the credibility of both statistical evidence and scientific claims. The credibility of evidence depends in part on its reproducibility; given the same quantitative evidence (i.e., data) and the same statistical analysis, can the same result be obtained? The credibility of scientific claims also depends in part on their replicability; if an experiment is repeated with the same procedures, therefore generating new data, will the same result be obtained (for discussion of these definitions, see Plesser, 2018)? Neither reproducibility nor replicability can be evaluated without both transparency and accessibility of the research process and the evidence generated from it. Thus, open science practices are crucial to the most basic aspects of the scientific process.

Munafò et al. call attention to many factors that undermine reproducibility and replicability. Some are the direct result of non-transparency (e.g., of missing study details or inaccessible data), whereas others stem from suboptimal methodological practices and research design. Such factors may include flexibility in statistical analysis or post hoc hypothesizing, when hypotheses are changed to better fit the results found. Figure 1 provides a powerful visualization of threats to scientific credibility that may occur at every stage of the research process. Although many of these threats have been known for decades (e.g., Cohen, 1965), reforms in psychology have only recently begun to address them more widely. Below, we highlight two instances identified by Munafò et al. where progress is being made: (1) developing and refining new tools to protect researchers against threats to credibility, such as self-deception, and (2) creating new incentive structures that reward researchers for being transparent and making research and related products (e.g., data or code) accessible. These and other tools outlined in Munafò et al. are given full focus later in this annotated reading list.

Figure 1 Threats to reproducible science. An idealized model of the scientific method is shown, with potential threats to the published research and science’s ability to self-correct shown underneath. This is a grayscaled reproduction of Figure 1 from Munafò et al. (2017) under the Creative Commons Attribution 4.0 International License; a copy of this license is available at http://creativecommons.org/licenses/by/4.0/.

Munafò et al. note that publications remain the primary scientific currency and that journals are more likely to publish novel, positive, and straightforward results rather than negative results or replications, ultimately rewarding science susceptible to false-positive results. However, such problematic incentive structures are changing. Since implementing publication badges for papers adopting open scientific practices, such as an Open Data badge, the journal Psychological Science has observed a large increase in articles sharing their data (see Open Data, Materials, and Code section). Funders are also adopting new research transparency requirements, and new funding opportunities are available that specifically focus on promoting reproducible research, meta-science, or replication studies (see Replication Research section). Another key part of the changing incentive structure is university hiring and promotion practices, which can encourage the use of unhelpful metrics, such as the journal impact factor (McKiernan et al., 2019). Increasingly, solicitations for academic appointments include language valuing open science practices (see https://osf.io/7jbnt).

Understanding the concept of open science is key to understanding the broad value of each topic reviewed in this paper. Whether by increasing or supporting rigor and transparency, each of the practices covered in subsequent parts of this paper can help individual researchers and psychological science as a whole engage in more replicable and reproducible research.

Open Access

Source: Tennant et al. (2016) – “The Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review”

A basic and essential part of research transparency is openness regarding the publication and dissemination process (see Appendix for further resources: Chan et al., 2002; COPE, OASPA, DOAJ, & WAME, 2018; Piwowar et al., 2018). Open Access (OA) refers to the unrestricted public availability of research products. Tennant et al., our second highlighted source, provide a thorough discussion of the history, forms, processes, and consequences of OA. The article discusses relevant empirical evidence related to the strengths and weaknesses of OA from the perspective of different stakeholders.

OA aims to remove barriers to accessing and distributing research and its relevant products. Enabled by advances in communication technology and coined by the Budapest Open Access Initiative (Chan et al., 2002), OA is defined as the free, public availability of a research product on the internet for distribution and re-use with acknowledgment. It therefore shares many functional similarities with the Creative Commons attribution licence CC-BY (Tennant et al., 2016). OA is typically used in reference to published journal articles, but any output could be OA, including student works and study materials, code, and data.

Most psychologists use the taxonomy described in Harnad et al. (2008) to define OA. Harnad distinguishes two non-exclusive routes to OA, while other taxonomies consider whether a paper has been made public by a fully OA journal or whether a work is explicitly licensed for reuse (Piwowar et al., 2018). The gold route refers to OA in the formal publication system: works are made publicly available at the point of publication (i.e., by the publishers themselves). The green route refers to authors self-archiving: works are made publicly available by the people who created them. Green OA includes self-archiving of works that have been peer-reviewed, as well as those that have not (e.g., pre-prints). Tennant et al. describe that across academia, only 25% of articles are OA through green or gold routes, despite 79% of journals indexed by SCOPUS endorsing author self-archiving in some format.

The accumulation of scientific knowledge is facilitated by wide access to research products so that anyone can access, build on, and respond to prior work. Tennant et al. discuss two specific effects of OA on academic research. First, OA works are used more: within academia, OA works are cited between 36% and 600% more than non-OA works, although opinions differ about the causal nature of this citation advantage. Outside of academia, OA works are given more coverage by journalists and discussed more in non-scientific settings (e.g., on social media). Second, OA works facilitate meta-research because they enable the use of automated text- and data-mining tools. Such options support meta-researchers in investigating research findings, which in turn helps us better understand what existing research can (and cannot) tell us.

Tennant et al. also discuss the complex economic impacts of OA on publishers, institutions, private research companies, and funders. They discuss the societal impact of OA, exploring the ethical implications of public access to research products, the importance of OA for people in low- and middle-income countries, and deceptive publishing practices related to the OA movement. Finally, they highlight connections between OA and the open science movement, and consider several directions for future research.

Those looking to make their research products OA have several options. Any product of research that authors own can be self-archived for free (green OA). To self-archive, authors should confirm ownership of the to-be-posted material, obtain permission from co-authors, and decide whether and how to licence the work before posting it on a public, stable institutional or subject repository, such as PsyArXiv (for a complete list see http://v2.sherpa.ac.uk/opendoar/). Authors can also make published works available (gold OA) by publishing in an exclusively OA journal (https://doaj.org/) or a hybrid journal that supports gold OA. For information about OA policies at specific journals, see the SHERPA RoMEO database (http://www.sherpa.ac.uk/romeo/index.php).

Open Data, Materials, and Code

Source: Klein et al. (2018) – “A Practical Guide for Transparency in Psychological Science”

Another basic form of research transparency is making a study’s materials, data, and/or analysis code publicly available (see Appendix for further resources: Gilmore, Kennedy, & Adolph, 2018; Levenstein & Lyle, 2018). According to the sixth edition of the APA Publication Manual, researchers must be able and willing to share their raw data with editors during the manuscript review process and with “other qualified professionals,” for 5 years after publication of their paper (APA, 2010). This standard is well grounded in several excellent reasons for sharing data that make work more reproducible and replicable, with a focus on enabling more reproducible and replicable research. For example, sharing data enables: verification, the process of checking results to minimize errors and bias in published work, and analytic reproducibility, the process of checking which data analysis steps were executed for cleaning and analyzing the data at hand.

In practice, data sharing is not common. In a descriptive survey of 600 researchers, Houtkoop et al. (2018) found that only 10% of respondents had ever publicly shared data. The vast majority of researchers fail to share their data either because they do not see it as necessary or important, or because they do not know how. Klein et al. provide practical guidelines to help researchers navigate the process of preparing and sharing their research. The authors make a strong case for why sharing data should be common practice in science. Two central arguments are that (1) science is based on verifiability not trust – one wants to be informed about every detail in an analysis instead of relying solely on the authors word for it, and (2) analytic reproducibility can only be achieved when data are openly available – re-running analyses to identify errors is a key ingredient of a healthy research cycle. Open data, materials, and code thus help increase the credibility of the research process, and boost the efficiency of scientific discovery and verifiability.

Klein et al. move on to tackle the what, when, where, and how of sharing products of research. As a bare minimum, they recommend that the data files the analysis is based on are shared openly. Furthermore, any additional information about an experiment (e.g., its stimuli, tasks, descriptions of instruments) that can be shared is helpful, although researchers should ultimately aim to share detailed empirical protocols and computer code that enables a fully reproducible analysis. Anonymizing data and explicitly reporting why transparent sharing might not be possible (e.g., data collected from sensible populations) are also important steps in this process. Regarding when to share research products, the short answer is better late than never. The longer answer may be connected to the relevant standards of journals (e.g., some journals ask researchers to share data once a paper goes into press) or funding agencies (e.g., some funders mandate that the data should be made available with closure of a project). Researchers should also consider how they share their data. Data should be shared in a FAIR manner – findable, accessible, interoperable, and reusable. Findable and accessible are mainly concerned with where data are uploaded. Considerations include the availability of persistent DOIs, metadata, tracking of data re-use, licensing, access control, and long term availability. One service that provides these features is the Open Science Framework (https://osf.io). Interoperable and reusable highlight the importance of thinking about data format (proprietary, e.g., Microsoft Word, vs. non-proprietary, e.g., text files) and how such formats might change in the future. One important way to ensure that materials, data, and code are reusable is through detailed documentation. This helps future users (including the original researcher, at a later point in time) understand the functioning of the materials, the structure and format of the data, and how the code processes this data to arrive at the statistical results in a paper. Some practical recommendations on data management are provided by Schönbrodt, Gollwitzer, and Abele-Brehm (2017).

Sharing data, materials, and analysis code is a basic open science practice that researchers can engage in to the extent it makes sense for their research context. Doing so, as Klein et al. note, not only makes work more replicable and reproducible but also increases efficiency, reduces errors, and brings benefits to the broader scientific community by increasing collective efficiency and credibility.

Reproducible Analyses

Source: Wilson et al. (2017) – “Good Enough Practices in Scientific Computing”

Another fundamental open science practice is producing reproducible analyses. This entails providing the resources (i.e., open data and open code) that allow others to exactly generate the results reported in the final research product. Code and data must also remain safe, organized, and accessible over time. Our fourth highlighted source therefore focuses on what computational skills are necessary to be a productive and reproducible scientist in the 21st century (see Appendix for further resources: Brown et al., 2014; Goh, Hall, & Rosenthal, 2016; Poldrack et al., 2017; Software Carpentry Workshops). Although most researchers do not consider themselves software developers, the vast majority now use computers to manage and store their data, collaborate with colleagues via online networks, and oftentimes write code to conduct their data analysis. Wilson et al. highlight how – in an age where scientific work is mediated by technology – computing best practices are as integral to robust research practices as fundamental lab techniques. They outline the basic scientific computing skills that are “good enough” to ensure that research is not just reproducible but also efficient, transparent, and accessible in the future.

Wilson et al. cover six different aspects to scientific computing: data management, software, collaboration, project organization, tracking changes, and writing manuscripts. Although every aspect might not apply to every researcher, any academic will likely find many of these aspects applicable to their daily work. For example, the importance of saving raw data separately from any manipulated versions of such data is highlighted. Furthermore, the data themselves should be backed-up in multiple locations (see also Spellman et al., 2017). This practice sounds simple and commonplace, but errors in raw data storage have caused multiple high-profile retractions in recent years (including a study published in Nature where the co-authors could not agree on the nature of their original data file; for retraction, see Brown et al., 2013; for additional discussion, see Oransky, 2013).

Wilson et al. also suggest certain practices regarding project organization. For example, each research project should have its own directory (folder) on a computer. Relevant text documents, raw data, and cleaned data files should then be put in separate sub-directories. Further, they discuss different techniques to keep track of work, either by using version control software or by copying the entire project folder whenever substantial changes are implemented. Although these practices may seem excessive initially, copying entire folders allows work and progress to be backed up and tracked quickly and easily. Wilson et al. stress: Data are cheap, time is expensive.

Many labs use different software for their daily work; Wilson et al. highlight aspects that all researchers should strive to fulfill, regardless of the software they choose to use. All steps of data processing should be recorded, either by coding analyses in open-source programs like R or Python or by “pasting” SPSS analytical steps into a reproducible syntax file. Wilson et al. also highlight the importance of not duplicating code – to decrease the potential for errors and increase usability. When sharing materials and data, researchers should explicitly decide on licensing agreements (St. Laurent, 2004). Such licensing agreements and data sharing practices are especially important when submitting data to “reputable DOI-issuing repositories”, such as Figshare, Dryad, and Zenodo. Doing so will allow researchers to share their data and receive recognition for it when re-used by others. Overall, implementing good coding and data storage practices in daily research is crucial to ensure that analyses are reproducible in the future, not just by others but also by the researchers themselves.

Increasing numbers of funders and institutions are asking for data management plans (e.g., Swiss National Science Foundation, 2018) and supporting the move to more “coding based” data analysis options; however, researchers are often not informed about what is necessary to ensure that all aspects of their research are reproducible. Wilson et al. provide a key starting point for those unsure about how to begin or those wanting to improve. Because nearly every researcher now works with computers on a daily basis, but rarely has formal training to do so responsibly, this paper will be critical reading for many. Making analyses and data as reproducible, clear, sustainable, and accessible as is feasible is an important step toward implementing and encouraging open scientific principles.

Preregistration

Source: Wagenmakers, Wetzels, Borsboom, van der Maas, and Kievit (2012) – “An Agenda for Purely Confirmatory Research”

Preregistration is an open science practice that protects researchers from some of the influence of misaligned incentives, allowing them to be more transparent about their analytic decision-making. Our fifth highlighted source focuses on preregistration (see Appendix for further resources: Chambers, Feredoes, Muthukumaraswamy, & Etchells, 2014; Nosek, Ebersole, DeHaven, & Mellor, 2018; van’t Veer & Giner-Sorolla, 2016). In this paper, Wagenmakers et al. call for a widespread adoption of preregistration, arising from the need to promote purely confirmatory research and transparently demarcate exploratory research. They begin by outlining the main limitations and pressures that lead researchers to overstate the evidentiary value of their results: cognitive biases, particularly confirmation and hindsight bias, and the pressure to publish large quantities of predominantly positive results. They emphasize that biases are inherently human and that succumbing to publication pressures is rational given the academic system. Although this situation is understandable when examining the motivations of any individual researcher, on a field-wide level, it can produce a scientific literature that is uninterpretable, populated by results that are oftentimes inaccurate or less compelling than claimed. “Research findings that do not replicate are worse than fairy tales; with fairy tales the reader is at least aware that the work is fictional” (Wagenmakers et al., 2012, p. 633). Wagenmakers et al. argue that the best way to remedy this situation is to encourage researchers to commit to hypotheses and analysis plans before they interact with their data, thus promoting a scientific system less governed by biases and publication pressures.

To understand the power of preregistration, it is key to examine the distinction between confirmatory and exploratory research (De Groot, 1956/2014). For research to be confirmatory, hypotheses and analyses must be specified before data collection and/or at a time when the researcher has no direct access to the data. As the term “confirmatory research” suggests, this type of research confirms hypotheses. In contrast, exploratory research focuses on generating hypotheses. Both exploratory and confirmatory research are important for science to progress, but they must be correctly and transparently labeled. Wagenmakers et al. argue that “almost no psychological research is conducted in a purely confirmatory fashion” (Wagenmakers et al., 2012, p. 633). They list common researcher degrees of freedom (Simmons et al., 2011), that is, possible points of analytic flexibility, such as cherry-picking which dependent variables or results to report or constructing hypotheses that best fit the data (i.e., HARKing, Kerr, 1998; see Figure 1). The results of such exploration of the data – whether intentional or unintentional – are often falsely labeled as confirmatory, increasing false-positive rates. They argue that for statistical results to be reliable, the practice of presenting exploratory research as confirmatory must be comprehensively prevented. The solution lies in preregistration: researchers committing to the hypotheses, study design, and analyses before the data are accessible. In their paper, Wagenmakers et al. present an exemplary preregistered replication as an illustration of this practice.

Wagenmakers et al. contend that preregistration is a straightforward solution to prevent researchers from presenting exploratory results as confirmatory, thus counteracting cognitive and publication biases. However, preregistration is no panacea; vague preregistrations can leave room for unreported exploration (Veldkamp et al., 2018) and can still be subject to the pressures of publication bias. The concept of “registered reports” (Chambers et al., 2014) might offer a better solution. Here, the publishability of a study is decided based on the preregistered study design and analysis plan (Stage 1 review). Following successful peer-review of the study design and protocol prior to data collection, a study is given in principle acceptance by the journal. This means that the study will be published regardless of the outcome, so long as the process adheres to the peer reviewed preregistered study design and analysis plan. Any further exploration must be clearly stated as such. Thus, both the rigor of and adherence to a preregistration is evaluated (Stage 2 review) and ensured in the registered report format.

Preregistration is a practice that increases the transparency of analytic decisions. It constrains the effect of biases and allows readers to interpret results in light of the broader analytic context. This, in turn, supports rigorous scientific research and enables more replicable and reproducible work.

Replication Research

Source: Zwaan, Etz, Lucas, and Donnellan (2018) – “Making Replication Mainstream”

Along with promoting the adoption of practices that directly increase transparency, the open science movement has also boosted confidence in the replicability of scientific results. Therefore, our sixth highlighted source discusses replication research, a key mechanism for encouraging the stability and generalisability of psychological phenomena (see Appendix for further resources: Brandt et al., 2014; Makel, Plucker, & Hegarty, 2012; Schmidt, 2009). Replication can increase the generalizability and veracity of findings but also incite controversy. In this integrative summary of replication in psychology, Zwaan and colleagues review definitions and types of replications, commonly expressed concerns about replication, and responses and strategies to resolve these concerns, while also providing options for the statistical evaluation of replication attempts. Despite the intense debate over the intent or need of individual replication attempts, replication is fundamental to the scientific endeavor. In fact, the authors emphasize that “a finding needs to be repeatable to count as a scientific discovery. Second, research needs to be reported in such a manner that others can reproduce the procedures” (p. 2). Without these components, a finding cannot inform scientific thinking.

There are different types of replication with varying goals and contributions. Direct replications seek to duplicate the necessary elements that produced the original finding, whereas conceptual replications purposefully change at least one component of the original procedure, for example, the sample or measure. Direct and conceptual replications serve different purposes; direct replications assess whether similar findings are reproduced in subsequent attempts, whereas conceptual replications assess whether previous findings are reproduced when tested under different conditions. Relatedly, Zwaan et al. suggest that “conceptual replication” is misleading and that labeling this form of replication as “alternative test” (p. 4) more clearly articulates the concept and leads to less conflation with direct replication. Regardless of the type of replication, conducting replications could be greatly facilitated if original research made greater use of open materials as discussed above. Similarly, replications are excellent examples of the type of research that are easily preregistered, as the methods, analytic plan, and expected results are already known.

At 61 pages, this is the longest source covered here. However, the full publication comprises a 13-page article, followed by extensive open peer commentary from independent authors, and a rejoinder from the original authors. The primary article can stand alone, but the open commentaries and rejoinder inform the ongoing replication conversation.

Zwaan et al. consider several commonly expressed concerns about replication and follow each of these sections with brief rebuttals. The specific concerns include: the role of context variability (e.g., does it matter whether the researcher wears a labcoat?), the scientific value of direct replications, the feasibility of direct replications, whether replications distract from more fundamental existing problems (e.g., publication bias), how replications affect the reputation of researchers, and the lack of standardized methods for evaluating replication results. Additionally, there may be misalignment of incentives for what benefits a field and what benefits individual researchers. Such misalignment may inhibit adherence to scientific norms and values, including the value of replication. Nevertheless, Zwaan et al. make the case for why no theoretical or statistical obstacles prevent replications from becoming a mainstream part of psychological research. They argue that in conjunction with other methods, replication helps clarify when confidence in results should be present as well as build theory, both of which help align psychology with the scientific method. From this perspective, producing replicable work and assessing the replicability of findings plays a central role in any credible scientific domain and fundamentally aligns with values of open science. Individual researchers can contribute to greater incorporation of replication into practice by conducting replications themselves and facilitating the replication of their own work through the use of open and reproducible materials, data, and code.

Teaching Open Science

Source: Chopik, Bremner, Defever, and Keller (2018) – “How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science”

Given that one of the main barriers to adopting open practices is a lack of education, our final highlighted source considers how best to integrate and promote open science through teaching (see Appendix for further resources: Frank & Saxe, 2012; Janz, 2016). Chopik et al. begin by asking: why are we not teaching our undergraduate students about the replication crisis and the related methodological reforms to? The authors observed that instructors often have difficulties integrating these topics into their lectures or are unsure where to start. This led to the development of a 1-hour lecture introducing students to causes and impact of psychology’s replication problem, attempts to evaluate the reproducibility and replicability of findings, and proposed solutions. The full lecture slides and script are openly available to provide a tool for instructors to keep themselves and their students up to date with open science (https://osf.io/mh9pe). Chopik et al. assessed the impact of the lecture by comparing students’ evaluation of psychology as a science and psychology reported in the media before and after the lecture. The need for such a lecture was evident; only 31% of their students had heard of the replication crisis prior to the lecture. Because 68% of students were currently or had previously taken a stats class, this result also highlights the need to integrate this information into formal statistics courses. Following the lecture, the majority (above 80%) of students agreed on the importance of methodological considerations (several of which are discussed previously in this paper), including reporting studies that “didn’t work out,” making data publically available, and choosing a sample size a priori. Students trusted psychological research less following the lecture. However, they also viewed psychology more similar to the natural or hard sciences, and this lecture did not reduce intentions to study at a graduate level. Many students agreed that psychology has a problem replicating results and that poor incentive structures can undermine science; however, nearly all students agreed that replication issues were not confined to psychology. Not all undergraduate psychology students will become researchers, which may cause some to question the value of teaching the replication crisis and open science. However, even if the intention is not to train future researchers, we must certainly train future consumers of research. Post-lecture, students were more critical of research presented in the media and more able to identify aspects that contribute to more replicable results. Being able to identify reproducible research and appreciate openness and rigor is an essential skill to be a good consumer of psychological research. A 1 hr lecture has the potential to improve this comprehension.

Teaching open science and the replication crisis is a pedagogical challenge. Thankfully, many teachers and researchers are providing resources to facilitate training students in open science. Hawkins et al. (2018) summarizes 11 graduate student-led pre-registered replications of studies in Psychological Science and offer an insightful overview on how this approach may be adapted for classes of different levels. Several Massive Open Online Courses (MOOCs) and similar online resources have been developed to provide a grounding in open science, including Transparent and Open Social Science Research (https://www.bitss.org/mooc-parent-page/), the Open Science MOOC (https://github.com/OpenScienceMOOC), and resources by the EU project FOSTER, including their Open Science Training Handbook (https://book.fosteropenscience.eu/en/). Finally, a recent initiative to support and recognise the teaching of open science practices is the Framework for Open and Reproducible Research Training (FORRT; https://forrt.netlify.com/). FORRT provides a basis for benchmarking course content covered in relation to open and reproducible research. Building on this, FORRT collates teaching-based resources aiming to help develop open and reproducible courses and further integrate these improved practices, while recognizing and supporting the outstanding contribution of teachers. In line with Chopik et al., these and similar other resources may be adapted and implemented into existing courses, greatly reducing the burden on instructors. Making use of resources like this in teaching can support the next generation of researchers in learning and incorporating open science practices from the very beginning.

Conclusions and Implications

Open science practices are a collection of behaviors that improve the quality and value of psychological research and aim to accelerate knowledge acquisition in the sciences. One barrier that prevents psychological scientists from adopting open science practices is a lack of knowledge. In this paper, we aimed to reduce this barrier by providing a summary and overview of seven topics and papers that cover pertinent and important issues and solutions surrounding open science. Starting with a broad review of open science, we also discussed specific open science practices, including open data, materials, and code, preregistration, and teaching open science. Readers of all backgrounds can therefore consult this text to understand the purpose of specific open scientific practices, obtain information about how to implement specific reforms, and find pointers to more detailed resources.

We hope that this paper will provide researchers interested in open science an accessible entry point to the practices most applicable to their needs. For all of the steps presented in this annotated reading list, any time taken by researchers to understand the issues and develop better practices will be rewarded in orders of magnitude. On an individual level, time and effort are ultimately saved, errors are reduced, and one’s own research is improved through a greater adherence to openness and transparency. On a field-wide level, the more researchers invest in adopting these practices, the closer the field will come toward adhering to scientific norms and the values it claims to espouse.

This article grew out of a hackathon at the 2018 meeting of the Society for the Improvement of Psychological Science (https://osf.io/cjwu8/). All authors contributed one or part of a summary and reviewed two other summaries. Sophia Crüwell conceptualized the article and oversaw the organization, managing edits and revisions. Sophia Crüwell, Matthew C. Makel, Hannah Moshontz, Jesse C. Niebaum, Amy Orben, and Sam Parsons contributed substantially to editing and reviewing the article before submission.

1We recognize a substantial debate on what constitutes “best practices” and note that best practices likely vary across research aims and disciplines. Here, we focus on introductory and consensus best practices to implement into a research program.

References

  • American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author. First citation in articleGoogle Scholar

  • Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature News, 533, 452–454. https://doi.org/10.1038/533452a First citation in articleCrossrefGoogle Scholar

  • Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. https://doi.org/10.1177/1745691612459060 First citation in articleCrossrefGoogle Scholar

  • Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., … van’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. https://doi.org/10.1016/j.jesp.2013.10.005 First citation in articleCrossrefGoogle Scholar

  • Brown, S. D., Furrow, D., Hill, D. F., Gable, J. C., Porter, L. P., & Jacobs, W. J. (2014). A duty to describe: Better the devil you know than the devil you don’t. Perspectives on Psychological Science, 9, 626–640. https://doi.org/10.1177/1745691614551749 First citation in articleCrossrefGoogle Scholar

  • Brown, W. M., Cronk, L., Grochow, K., Jacobson, A., Liu, C. K., Popović, Z., & Trivers, R. (2013). Retraction: Dance reveals symmetry especially in young men. Nature, 504, 470. https://doi.org/10.1038/nature12728 First citation in articleCrossrefGoogle Scholar

  • Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. (2014). Instead of “playing the game” it is time to change the rules: Registered reports at AIMS neuroscience and beyond. AIMS Neuroscience, 1, 4–17. https://doi.org/10.3934/Neuroscience2014.1.4 First citation in articleCrossrefGoogle Scholar

  • Chan, L., Cuplinskas, D., Eisen, M., Friend, F., Genova, Y., Guedon, J.-C., … Velterop, J. (2002, February 14). Budapest Open Access Initiative. Retrieved from https://www.budapestopenaccessinitiative.org/read First citation in articleGoogle Scholar

  • Chopik, W. J., Bremner, R. H., Defever, A. M., & Keller, V. N. (2018). How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology, 45, 158–163. https://doi.org/10.1177/0098628318762900 First citation in articleCrossrefGoogle Scholar

  • Cohen, J. (1965). Some statistical issues in psychological research. In B. B. WolmanEd., Handbook of clinical psychology (pp. 95–121). New York, NY: McGraw-Hill. First citation in articleGoogle Scholar

  • Committee on Publication Ethics (COPE) Council, Open Access Scholarly Publishers Association (OASPA), Directory of Open Access Journals (DOAJ), & World Association of Medical Editors (WAME). (2018, January). Principles of transparency and best practice in scholarly publishing. (Version 3). Retrieved from https://publicationethics.org/resources/guidelines-new/principles-transparency-and-best-practice-scholarly-publishing First citation in articleGoogle Scholar

  • Corker, K. (2018, September 12). Open science is a behaviour. [Blog post]. Retrieved from https://cos.io/blog/open-science-is-a-behavior/ First citation in articleGoogle Scholar

  • De Groot, A. D. (1956/2014). The meaning of “significance” for different types of research. Translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas. Acta Psychologica, 148, 188–194. https://doi.org/10.1016/j.actpsy.2014.02.001 First citation in articleCrossrefGoogle Scholar

  • Etz, A., Gronau, Q. F., Dablander, F., Edelsbrunner, P. A., & Baribault, B. (2018). How to become a Bayesian in eight easy steps: An annotated reading list. Psychonomic Bulletin & Review, 25, 219–234. https://doi.org/10.3758/s13423-017-1317-5 First citation in articleCrossrefGoogle Scholar

  • Fecher, B., & Friesike, S. (2014). Open science: One term, five schools of thought. In S. BartlingS. FriesikeEds., Opening science (pp. 17–47). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-00026-8 First citation in articleGoogle Scholar

  • Frank, M. F., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600–604. https://doi.org/10.1177/1745691612460686 First citation in articleCrossrefGoogle Scholar

  • Gilmore, R. O., Kennedy, J. L., & Adolph, K. E. (2018). Practical solutions for sharing data and materials from psychological research. Advances in Methods and Practices in Psychological Science, 1, 121–130. https://doi.org/10.1177/2515245917746500 First citation in articleCrossrefGoogle Scholar

  • Goh, J. X., Hall, J. A., & Rosenthal, R. (2016). Mini meta‐analysis of your own studies: Some arguments on why and a primer on how. Social and Personality Psychology Compass, 10, 535–549. https://doi.org/10.1111/spc3.12267 First citation in articleCrossrefGoogle Scholar

  • Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, p values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31, 337–350. https://doi.org/10.1007/s10654-016-0149-3 First citation in articleCrossrefGoogle Scholar

  • Harnad, S., Brody, T., Vallieres, F., Carr, L., Hitchcock, S., Gingras, Y., … Hilf, E. R. (2008). The access/impact problem and the green and gold roads to open access: An update. Serials Review, 34, 36–40. https://doi.org/10.1016/j.serrev.2007.12.005 First citation in articleCrossrefGoogle Scholar

  • Hawkins, R. X. D., Smith, E. N., Au, C., Arias, J. M., Catapano, R., Hermann, E., … Frank, M. C. (2018). Improving the replicability of psychological science through pedagogy. Advances in Methods and Practices in Psychological Science, 1, 7–18. https://doi.org/10.1177/2515245917740427 First citation in articleCrossrefGoogle Scholar

  • Higginson, A. D., & Munafò, M. R. (2016). Current incentives for scientists lead to underpowered studies with erroneous conclusions. PLoS Biology, 14, e2000995. https://doi.org/10.1371/journal.pbio.2000995 First citation in articleCrossrefGoogle Scholar

  • Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V., Nichols, T. E., & Wagenmakers, E. J. (2018). Data sharing in psychology: A survey on barriers and preconditions. Advances in Methods and Practices in Psychological Science, 1, 70–85. https://doi.org/10.1177/2515245917751886 First citation in articleCrossrefGoogle Scholar

  • Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2, e124. https://doi.org/10.1371/journal.pmed.0020124 First citation in articleCrossrefGoogle Scholar

  • Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention. Trends in Cognitive Sciences, 18, 235–241. https://doi.org/10.1016/j.tics.2014.02.010 First citation in articleCrossrefGoogle Scholar

  • Janz, N. (2016). Bringing the gold standard into the classroom: Replication in university teaching. International Studies Perspectives, 17, 392–407. https://doi.org/10.1111/insp.12104 First citation in articleGoogle Scholar

  • Kass, R. E., Caffo, B. S., Davidian, M., Meng, X. L., Yu, B., & Reid, N. (2016). Ten simple rules for effective statistical practice. PLOS Computational Biology, 12, e1004961. https://doi.org/10.1371/journal.pcbi.1004961 First citation in articleCrossrefGoogle Scholar

  • Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217. https://doi.org/10.1207/s15327957pspr0203_4 First citation in articleCrossrefGoogle Scholar

  • Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., … Frank, M. C. (2018). A practical guide for transparency in psychological science. Collabra: Psychology, 4, 20. https://doi.org/10.1525/collabra.158 First citation in articleCrossrefGoogle Scholar

  • Levenstein, M. C., & Lyle, J. A. (2018). Data: Sharing is caring. Advances in Methods and Practices in Psychological Science, 1, 95–103. https://doi.org/10.1177/2515245918758319 First citation in articleCrossrefGoogle Scholar

  • Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537–542. https://doi.org/10.1177/1745691612460688 First citation in articleCrossrefGoogle Scholar

  • McKiernan, E. C., Schimanski, L. A., Nieves, C. M., Matthias, L., Niles, M. T., & Alperin, J. P. (2019). Use of the journal impact factor in academic review, promotion, and tenure evaluations. ELife, 8, e47338. https://doi.org/10.7554/eLife.47338 First citation in articleCrossrefGoogle Scholar

  • Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., … Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562–016-0021 First citation in articleCrossrefGoogle Scholar

  • Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115, 2600–2606. https://doi.org/10.1073/pnas.1708274114 First citation in articleCrossrefGoogle Scholar

  • O’Boyle, E. H., Banks, C. B., & Gonzalez-Mulé, E. (2014). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 376–399. https://doi.org/10.1177/0149206314527133 First citation in articleCrossrefGoogle Scholar

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. https://doi.org/10.1126/science.aac4716 First citation in articleCrossrefGoogle Scholar

  • Oransky, I. (2013). At long last, disputed dance study retracted from Nature. Retrieved from https://retractionwatch.com/2013/11/27/at-long-last-disputed-dance-study-retracted-from-nature/ First citation in articleGoogle Scholar

  • Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., … Haustein, S. (2018). The state of OA: A large-scale analysis of the prevalence and impact of open access articles. PeerJ, 6, e4375. https://doi.org/10.7717/peerj.4375 First citation in articleCrossrefGoogle Scholar

  • Plesser, H. E. (2018). Reproducibility vs. replicability: A brief history of a confused terminology. Frontiers in Neuroinformatics, 11, 76. https://doi.org/10.3389/fninf.2017.00076 First citation in articleCrossrefGoogle Scholar

  • Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., … Yarkoni, T. (2017). Scanning the horizon: Towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18, 115–126. https://doi.org/10.1038/nrn.2016.167 First citation in articleCrossrefGoogle Scholar

  • Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90–100. https://doi.org/10.1037/a0015108 First citation in articleCrossrefGoogle Scholar

  • Schönbrodt, F. D., Gollwitzer, M., & Abele-Brehm, A. (2017). Der Umgang mit Forschungsdaten im Fach Psychologie: Konkretisierung der DFG-Leitlinien. Im Auftrag des DGPs Vorstands (17. 09. 2016) [Data management in psychological science: Specification of the DFG guidelines]. Psychologische Rundschau, 68, 20–35. https://doi.org/10.1026/0033-3042/a000341 First citation in articleLinkGoogle Scholar

  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. https://doi.org/10.1177/0956797611417632 First citation in articleCrossrefGoogle Scholar

  • Spellman, B., Gilbert, E. A., & Corker, K. S. (2017, September 20). Open science: What, why, and how. https://doi.org/10.31234/osf.io/ak6jr First citation in articleGoogle Scholar

  • St. Laurent, A. (2004). Understanding open source and free software licensing. Sebastopol, CA: O’Reilly Media. Retrieved from http://www.oreilly.com/openbook/osfreesoft/book/ First citation in articleGoogle Scholar

  • Swiss National Science Foundation. (2018). Data Management Plan (DMP) – Guidelines for researchers. Retrieved from http://www.snf.ch/en/theSNSF/research-policies/open_research_data/Pages/data-management-plan-dmp-guidelines-for-researchers.aspx First citation in articleGoogle Scholar

  • Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. (2016). The academic, economic and societal impacts of open access: An evidence-based review. F1000Research, 5, 632. https://doi.org/10.12688/f1000research.8460.3 First citation in articleCrossrefGoogle Scholar

  • Vazire, S. (2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13, 411–417. https://doi.org/10.1177/1745691617751884 First citation in articleCrossrefGoogle Scholar

  • van’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology – A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. https://doi.org/10.1016/j.jesp.2016.03.004 First citation in articleCrossrefGoogle Scholar

  • Veldkamp, C. L. S., Bakker, M., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., … Wicherts, J. M. (2018, September 19). Ensuring the quality and specificity of preregistrations. https://doi.org/10.31234/osf.io/cdgyh First citation in articleGoogle Scholar

  • Washburn, A. N., Hanson, B. E., Motyl, M., Skitka, L. J., Yantis, C., Wong, K. M., … Carsel, T. S. (2018). Why do some psychology researchers resist adopting proposed reforms to research practices? A description of researchers’ rationales. Advances in Methods and Practices in Psychological Science, 1, 166–173. https://doi.org/10.1177/2515245918757427 First citation in articleCrossrefGoogle Scholar

  • Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, A. J., Love, J., … Morey, R. D. (2018). Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. Psychonomic Bulletin & Review, 25, 35–57. https://doi.org/10.3758/s13423-017-1343-3 First citation in articleCrossrefGoogle Scholar

  • Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. https://doi.org/10.1177/1745691612463078 First citation in articleCrossrefGoogle Scholar

  • Wilson, G. (2006). Software carpentry: Getting scientists to write better code by making them more productive. Computing in Science & Engineering, 8, 66–69. https://doi.org/10.1109/MCSE.2006.122 First citation in articleCrossrefGoogle Scholar

  • Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017). Good enough practices in scientific computing. PLoS Computational Biology, 13, e1005510. https://doi.org/10.1371/journal.pcbi.1005510 First citation in articleCrossrefGoogle Scholar

  • Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. https://doi.org/10.1017/S0140525X17001972. Preprint openly available at https://psyarxiv.com/4tg9c/ First citation in articleCrossrefGoogle Scholar

Appendix

Additional Resources

In Table A1, we provide 20 further resources. They are briefly summarized and ranked by focus (theoretical to applied) and difficulty (easy to hard) on scales of 1–10, respectively. Some of these further resources are not peer-reviewed articles but nevertheless very helpful. A spreadsheet providing quick open access links to the resources highlighted in this paper and its appendix is available at https://bit.ly/2OLxUsW

Table A1 Further resources
Sophia Crüwell, Meta-Research Innovation Center Berlin, QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Anna-Louisa-Karsch-Straße 2, 10178 Berlin, Germany,