Skip to main content
Free AccessEditorial

Registered Report 2.0: The PCI RR Initiative

Published Online:https://doi.org/10.1027/1618-3169/a000512

In the year 2014, the journal Experimental Psychology introduced the Registered Report (RR) as a new article type and was then among one of the first adopters of this new publication type (Stahl, 2014). To fully appreciate this important step in the development of the journal, it is helpful to recollect the challenges that psychological science faced at this time. The robustness of psychological findings was seriously questioned within and outside of academia by the discovery of faked, massaged, and unreproducible study findings; reported use of questionable research practices, such as “Hypothesizing after results are known” and p-hacking; the use of editorial practices that incentivized publications of “great results” for an increase in journal citation metrics (“impact factor”) and social media clicks — but at the cost of selective reporting and inflated false positive results.

At the core of many of these problems is a results-driven publication culture that puts QUALITY of published research in competition with SUCCESS of the researcher. For the advancement of science, it is essential to publish high-quality studies regardless of the study outcome. This explicitly includes “negative” results in form of unexpected and/or null findings. For the advancement of the individual scientific career, however, it is essential to publish many “great results” — or she will perish (i.e., will not get promoted, receives no tenure, necessary research funds, etc.). Especially for early-career scientists it is difficult to find a good balance between these conflicting goals. The proposed solution is a change of the publication culture that evaluates the quality of a research study regardless of the outcome of the study. A key instrument for this change is the Registered Report.

Registered Reports require a preregistration of research questions, design and data-analytic choices before research outcomes are known, and often before data collection starts (Chambers & Tzavella, 2020). There exist several ways and forms how researchers could preregister a study plan, but basically one could distinguish between peer-reviewed and non-peer reviewed procedures (Hardwicke & Ioannidis, 2018). Experimental Psychology, and an increasing number of other publication outlets, offer a peer review of study plans using a review system in two stages. At Stage 1, authors submit a research proposal that explains the theoretical background, method, and data analysis for a planned but not yet conducted experiment. After review and revisions, the favourably assessed study plan receives an in-principle acceptance (IPA) and data collection could start (Stage 2). Importantly, study results are then published independently of the direction or “significance” of the outcome, provided that the authors adhered to the approved protocol and interpreted the results in line with the evidence.

This preregistration procedure offers several advantages in respect to quality control and career planning. For example, researchers could use constructive feedback from the reviewers in Stage 1 for revisions of the study plan. Furthermore, commitment to the preregistered study plan, and control of its proper implementation, is ensured by the peer review in Stage 2. Finally, researchers can trust that after IPA, the study will be published regardless of the results, which facilitates planning of a productive research career.

However, the peer-reviewed preregistration procedure, as currently practiced by many scientific journals, also has disadvantages. Firstly, peer review of preregistration documents is time-consuming, which demands a highly efficient workflow. Secondly, and related to the previous point, scientific journals often have different guidelines and procedures for Registered Reports, which complicates assessments by reviewers. Finally, with the submission of a Stage 1 proposal to a specific journal, authors must commit to a potential publication in a specific journal a long time ahead of the actual publication process, which could feel forced if circumstances or preferences have changed.

These problems are partly addressed by a recent initiative which was dubbed the Peer-Community In Registered Reports — or in short: PCI RR. Figure 1 lists key features of this initiative. PCI RR is not a journal but a non-commercial, researcher-run platform that manages peer review of Registered Report preprints from several disciplines including psychology. The use of this platform is free for authors, readers, and supporting journals. Importantly, several journals have agreed to publish Registered Reports that were accepted (or “recommended” in PCI language) after peer review in PCI RR. Experimental Psychology is among these early adopters (for the full list of PCI-friendly journals see https://rr.peercommunityin.org/about/pci_rr_friendly_journals). This means, Experimental Psychology will publish the PCI-recommended article without further peer review. Editors of Experimental Psychology will still check PCI-recommended Registered Reports for a fit to the journal’s scope (experimental study in psychology) and to the journal’s minimum requirements for a Registered Report in terms of bias control (Level 6 as described in the PCI RR taxonomy, see https://rr.peercommunityin.org/help/guide_for_authors); however, the assessment of the quality of the work is delegated to the peer review process at PCI RR. Hence, authors can submit their study proposal to PCI RR for peer review even if they are undecided whether they want a publication in the journal Experimental Psychology.

Figure 1 Key facts about PCI RR. (Figure retrieved from https://tinyurl.com/cf89de73 using CC-BY 4.0).

The review procedure at PCI RR does not deviate much from the two-stage RR review system of Experimental Psychology. Figure 2 shows how it works. At Stage 1, authors submit their study proposal by providing a link to the preregistration documents deposited in a recognized repository (e.g., OSF, GitHub; optionally private and shared through a view-only URL). At this point, authors can either submit a full Stage 1 manuscript or a one-page “snapshot” that provides a brief overview of the project. When submitting a snapshot, PCI RR uses the study information to schedule the review process in advance (e.g., 6 weeks ahead), allowing authors to prepare a full Stage 1 manuscript in the intervening period and thus, eventually, performing in-depth Stage 1 evaluation within days rather than weeks of receiving a full submission (https://rr.peercommunityin.org/help/guide_for_authors).

Figure 2 Workflow in PCI RR. (Figure retrieved from https://tinyurl.com/cf89de73 using CC-BY 4.0).

All submissions must pass an initial triage assessment by the PCI RR Managing Board and the handling recommender before the full Stage 1 manuscript is sent out for in-depth reviews to reviewers. Proposals that are favourably assessed receive IPA, which commits PCI RR to recommending the final article regardless of the outcomes, provided the authors adhere to their approved protocol and interpret the results in line with the evidence. Following IPA, PCI RR then registers the approved Stage 1 manuscript on the Open Science Framework, either publicly or under a temporary embargo. Then, after completing the research, authors post the Stage 2 manuscript on an established preprint server, with the manuscript including the approved protocol plus results and discussion, which may include clearly labelled post hoc analyses in addition to the preregistered outcomes. Where possible, the reviewers from Stage 1 then return to assess the completed Stage 2 manuscript, focusing on compliance with protocol and whether the conclusions are justified by the evidence. Following a positive PCI RR recommendation, the approved preprint remains permanently discoverable and citable on the preprint server, and the reviews and recommendation are published on the PCI RR website. Authors can then opt to submit the accepted Stage 2 manuscript to Experimental Psychology, which will publish the recommended article without further peer review.

We thank the founders of PCI RR for inviting us to join this initiative, and in particular, Prof. Chris Chambers who has joined our own editorial board as a consulting editor. In addition, (outgoing) editor-in-chief Andreas Eder has agreed to serve as a “recommender” at PCI RR (analogous to an action editor) for a continued, mutual exchange between the journal and PCI RR.

The PCI RR website launched officially on April 19, 2021 and is now accepting submissions. You can visit the website https://rr.peercommunityin.org/ and submit your proposal for a Registered Report. Submission is, of course, also possible via our own journal platform (https://www.hogrefe.com/us/journal/experimental-psychology), which has additional advantages (e.g., the participation in our grant program for replication studies using the Registered Report type, https://www.hogrefe.com/us/journals/exppsy/call-for-replication-studies). Whichever platform you will ultimately use — we would be happy if you would choose Experimental Psychology as a publication outlet for your next Registered Report!

We thank Chris Chambers for providing comments.

References