Strengthening the Evidence Base in Suicide Prevention

We have come a long way in our understanding of suicide and how to prevent it, but there is still much that we don't know. Part of the reason for this is that our research is not always as good as it could be. We sometimes use suboptimal methods, we don't always describe what we have done in a transparent and replicable fashion, and we're not always great at synthesizing the evidence from multiple studies in a way that allows us to take the field forward. Of course, there are many, many exceptional pieces of research too, but sometimes these are not given the attention they deserve.

In 2020, Crisis is introducing two new article formats that are explicitly designed to strengthen the evidence base in suicide prevention. These are Systematic Reviews and Registered Reports. Both are described below, with greater detail being provided about registered reports on the assumption that readers and authors may be less familiar with this type of article than systematic reviews. More detail about the specific requirements for both systematic reviews and registered reports can be found by following the Instructions to Authors link on our home page at https://www.hogrefe.com/j/crisis

Systematic Reviews

Systematic reviews are designed to appraise, synthesize, and report on the results of multiple studies that investigate the same phenomenon (Green, 2005). Our aim in creating this new article format and actively calling for high-quality systematic reviews is to help to strengthen the evidence base on suicide and its prevention. Bringing together the findings from a number of studies in a systematic way can teach us much more than we can learn by considering individual studies in isolation.

Systematic reviews tend to examine interventions, allowing relevant stakeholders to consider the totality of available evidence and fund and deliver what can be regarded as best practice in a given field. They needn't be restricted to interventions, however; systematic reviews can also bring together evidence from descriptive and analytical studies that allow us to gauge, for example, the overall prevalence of particular conditions, the risk factors for these conditions, and the mechanisms by which these risk factors may operate. Plugging the search terms suicide and systematic review into Medline reveals that the most highly cited systematic review in our field is one by Mann et al. (2005) which assembled the evidence for universal, selective and indicated suicide prevention interventions. This review was recently updated by Zalsman et al. (2016).

In some systematic reviews, authors formally pool the data from multiple studies in order to arrive at an overall estimate of the phenomenon of interest via a meta-analysis. Again, meta-analyses may be designed to gauge the overall effectiveness of given interventions, as is the case with those conducted through the Cochrane Collaboration (Friedrich, 2013). They may also be conducted to generate reliable estimates of prevalence or risk factors, or to summarize other descriptive statistics across studies. Another quick-and-dirty search of Medline, this time using the terms suicide and meta-analysis, identifies a seminal paper by Harris and Barraclough (1997) as the most highly cited in the area. This meta-analysis set out to determine the risk of suicide for people with common mental disorders, and debate about the relationship between mental illness and suicide has continued since it was published (Too et al., 2019).

Of course, Crisis has published systematic reviews since its inception. Since I took over as Editor-in-Chief at the beginning of 2018, we have published systematic reviews on such topics as the psychological distress and impairment experienced by crisis telephone line workers (Kitchingman, Wilson, Caputi, Wilson, & Woodward, 2018), the effectiveness of suicide prevention training for nurses (Ferguson et al., 2018), the quality of economic evaluations of suicide prevention interventions (Bustamante Madsen, Eddleston, Schultz Hansen, & Konradsen, 2018), factors influencing the likelihood that people will seek professional help if they are struggling with suicidal thoughts and behaviors (Han, Batterham, Calear, & Randall, 2018), the range of experiences of bereaved individuals (Andriessen, Krysinska, Castelli Dransart, Dargis, & Mishara, 2019; Andriessen, Krysinska, Draper, Dudley, & Mitchell, 2018), the nature and trajectory of bereavement and postvention research (Maple et al., 2018), the inclusion and exclusion of people with psychosis in trials of suicide prevention interventions (Villa, Ehret, & Depp, 2019), and the classification of suicidal behaviors (Goodfellow, Kolves, & De Leo, 2019).

I know, however, that Crisis could have published more systematic reviews but that contributors have sometimes been constrained by the existing range of article formats. To date, most systematic reviews have been published as Research Trends articles, which have a strict word limit of 4,500 words, including references. Systematic reviews of large numbers of studies do not fit readily within this format, hence our decision to create the new article type. The word limit for the new Systematic Reviews category is 6,000, and there is also scope for including Electronic Supplementary Material if necessary.

We are distinguishing systematic reviews from other kinds of reviews, and will be favoring those that are explicit about the search terms and databases used, the inclusion and exclusion criteria, how the studies were assessed for quality and so on. At a minimum, we will be asking for reviews to adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher, Liberati, Tetzlaff, Altman, & Group, 2009). This is not to say that we will be expecting all systematic reviews to involve meta-analyses, or even to draw on quantitative data, but we will be aiming to ensure that the studies included in a given review have been identified in a comprehensive and defensible manner, that steps have been taken to minimize bias, and that anyone reading the review could replicate it if they wished to.

Registered Reports

The Registered Report is a relatively new form of article which explicitly fosters reproducibility and transparency in scientific endeavours. Registered Reports were first introduced in the journal Cortex in 2013 to address questionable research practices like hypothesizing after results are known (HARKing) and conducting multiple analyses and reporting only those that reach a certain level of statistical significance (p-hacking) (Chambers, 2013). Traditionally, journals have inadvertently created incentives for these sorts of practices by a bias towards articles that present positive findings. The reward systems of academic institutions and granting bodies have also unwittingly helped to shape these behaviors; academics' track records are based on articles that report novel or ground-breaking results.

We're all for studies that make an impact, but this impact should not be achieved by cherry-picking results and telling the story with the benefit of hindsight. Registered Reports address this by giving priority to the rationale for the study, as well as its hypotheses, study design, and methods. Analyses are predefined, and relevant power calculations are declared at the outset.

The way this will work in Crisis, as it does in other journals that have adopted Registered Reports, is that there will be a two-stage peer review process. Authors will submit a Stage 1 manuscript which contains an introduction and rationale and describes the methods and proposed analyses in sufficient detail that they could be replicated by someone else. Any pilot data may also be presented at this point, particularly if it underpins the rationale, speaks to the feasibility of conducting the study, or provides support for any element of the analysis.

At this point, the Stage 1 manuscript will be subject to the normal peer review process. It will undergo editorial triage, and, assuming that it is not rejected at this point, will then be sent to reviewers. Reviewers may recommend immediate rejection or in-principle acceptance, or that the authors make some revisions. If the latter is the case, authors will be invited to respond in the usual way. Depending on the extent of the suggested changes, the reviewers may be invited to offer a second critique. Ultimately, the decision will rest with the editorial team, with the goal for the authors being in-principle acceptance of their Stage 1 manuscript.

Once the Stage 1 manuscript receives this in-principle acceptance, the author must preregister it in a repository. Importantly, it is not until this point that the authors can begin their study. They must adhere strictly to the methods they have articulated in their Stage 1 manuscript; only minor deviations are acceptable, and these must be discussed with the editorial team. In many ways, the Stage 1 manuscript is like an application to an ethics committee or institutional review board, but it has the additional advantage of formally committing the authors to conducting their study in a particular way.

Once the data collection and analysis have been completed, the authors will write up their final paper, often called the Stage 2 manuscript. This essentially involves adding the results and discussion sections to the Stage 1 submission. The authors submit this and it too undergoes peer review; this time the reviewers basically check that what was proposed in the Stage 1 manuscript has actually been carried out. Importantly, because the Stage 1 manuscript will have already received in-principle acceptance, the Stage 2 manuscript is automatically accepted, irrespective of the directionality of the results, unless the study has diverged from what was originally proposed (and notwithstanding any minor suggestions from the reviewers about clarity etc). This means that null findings will be given as much weight by the journal as positive findings. The word limit for Stage 1 manuscripts is 3,500 and the word limit for Stage 2 manuscripts is 4,500.

Registered Reports may not suit all types of research. In particular, they are better geared for confirmatory, hypothesis-driven research than they are for exploratory research (Chambers, Forstmann, & Pruszynski, 2017). By way of example, an evaluation of a training program designed to improve gatekeepers' confidence in dealing with someone who might be at risk of suicide would be eminently amenable to a Registered Report. Its hypothesis can be clearly stated (i.e., that gatekeepers' confidence will be increased), and number and nature of its participants and the process by which they will be recruited are readily identifiable, as are the data collection instruments and the analysis strategy. By contrast, a qualitative study examining the responses from family and friends that suicidal individuals found helpful and unhelpful may not fit the Registered Report format, particularly if it does not have an a priori hypothesis but lets themes emerge from the data, and if it uses purposive sampling that concludes when saturation is reached.

Registered Reports have many benefits. They reward researchers for being exceptional at their craft, rather than for what their studies may or may not find. They enable authors to receive independent input into their studies at the time when it can be most useful, which is also likely to be more satisfying for reviewers. And, because they are agnostic about the directionality of findings, they have the potential to overcome issues relating to publication bias. They can shine a light on hypotheses that are not supported by well-designed, well-powered studies. This is crucial in a field like suicide prevention where it is just as important to know what doesn't work as what does work. It is worth noting here though that it's not all about giving journal space to negative results; in fact, Registered Reports incentivise high-risk studies which may produce game-changing findings (Ansari & Gervain, 2018).

Some concerns have been expressed about Registered Reports too, although most of these have been countered. Firstly, there is the worry that the precise pre-specification of methods will limit procedural flexibility and analytic creativity. In fact, the format allows for this by allowing for minor methodological deviations, providing they have editorial imprimatur (Chambers et al., 2017). There is also nothing to stop authors conducting post-hoc analyses, providing they are clearly differentiated from the originally planned analyses in the final text (Chambers et al., 2017). In addition, authors are at liberty to withdraw their Stage 1 manuscripts if they choose to modify their approach significantly.

A second concern is that the process will create delays, particularly because researchers have to wait until their Stage 1 manuscript receives in-principle acceptance before they can begin their empirical work. Although this is true, there are likely to be time savings later in the process because the final acceptance will be expedited (Chambers et al., 2017). As researchers become more familiar with the Registered Reports process, they are likely to create their own efficiencies by, for example, working up Registered Reports at the same time as grant applications and ethics applications. There are precedents for this, with one journal (Nicotine and Tobacco Research) joining forces with a funder (Cancer Research UK) to pilot a model in which successful grant applications are then converted into Registered Reports (Munafo, 2017).

A third issue is whether the two-stage review process will result in a greater impost on reviewers (Robertson, 2017). Currently, we often ask reviewers to take a second look at a given article, particularly if it has undergone major revisions. This might happen with the Stage 1 manuscript, so the additional effort would come when they were asked to review the Stage 2 manuscript. The benefit here would be that they would already be familiar with the study and the process would typically be quick because the Stage 1 manuscript would have already received in-principle acceptance. Again, efficiencies may become apparent over time; in the above-mentioned collaboration between Nicotine and Tobacco Control and Cancer Research UK, reviewers assess both the grant applications and the Registered Reports (Munafo, 2017). In addition, as noted above, reviewers may find the Registered Reports model more satisfying than the traditional model because they can have genuine input into a given study's design.

A final issue which has been discussed by some commentators is the risk that authors may be "scooped," either by unscrupulous reviewers or by other academics who access their Stage 1 manuscript in the repository. In our view, this risk is low, not least because the date of receipt of the Stage 1 manuscript will be evident, but also because many repositories allow Stage 1 manuscripts to be embargoed.

The number of journals that have adopted the registered report format has grown steadily since 2013 and, as of January 1, 2020, is sitting at 217 (https://cos.io/rr/). This has more than doubled since February 22, 2018 when Hardwicke and Ioannidis (2018) conducted a review of registered reports and found the number was 91. At Crisis, we can learn from the experiences of other journals that have pioneered this format. Hardwicke and Ioannidis (2018) found that many journals have kept the publication of Stage 1 manuscript with in-principle acceptance in-house, which has meant that they are not publicly available. This protocol transparency is crucial for registered reports to achieve their full potential (Chambers & Mellor, 2018). In addition, Hardwicke and Ioannidis (2018) were only able to identify 91 final articles from 16 journals, with most participating journals not yet having published any final articles under the format. At Crisis, we will ensure that Stage 1 manuscripts with in-principle acceptance are readily accessible and will monitor the progress of the studies that are related to them. Our aim is to ensure that the new format leads to greater transparency and more high-quality studies.

A Final Comment

We're excited to be introducing our two new article formats this year, not only for Crisis but also for the suicide prevention community. In different ways, systematic reviews and registered reports have an important role to play in strengthening the evidence base and the research that underpins it. As a final comment, it is worth noting that both formats may be particularly beneficial for emerging researchers – particularly PhD students – who are entering the field of suicide prevention. Indeed, it was the Early Career Group of the International Association for Suicide Prevention that suggested that Crisis should publish Registered Reports. All PhD students prepare literature reviews, and our new Systematic Review format may give them an avenue for publishing them. PhD students also put their hearts and souls into designing studies that may or may not have positive outcomes, and the opportunity to publish these as Registered Reports that favor methodological rigor over ultimate findings may also be valuable for them. These emerging researchers represent the future of suicide prevention, and they are leading the way in promoting scientific advances in the field (Daly, Morch, & Kirtley, 2018). We hope that the new Systematic Reviews and Registered Reports formats will help them – and those of us who have been around for much longer than them – to achieve these goals.