How Communicating Vaccine Benefits and Harms in Fact Boxes Affects Risk Perceptions
Two Randomized Trials
Abstract
Abstract:Background: A fact box is a promising tool for benefit-risk communication. This compact table summarizes the benefits and harms of a health-related intervention and compares the intervention to a control group. Although previous research has demonstrated that fact boxes are well understood, little is known about how they affect risk perceptions. Risk perception is particularly relevant to vaccination behavior. Aims: Two experiments investigated how different profiles of vaccine benefits and harms influence risk perceptions and the intention to vaccinate. Method: In Experiment 1, 430 participants were included in a 4 (benefits [vaccine effectiveness]) × 2 (harms [probability of vaccine adverse events]) between-subjects design. In Experiment 2, 541 participants were included in a 2 (benefits) × 2 (harms) × 2 (comprehension test conducted before or after assessment of risk perceptions) between-subjects design. Measures: Perceived risk of vaccination, intention to vaccinate, comprehension, and, in Experiment 2 only, perceived risk of non-vaccination. Results: Greater benefits decreased the perceived risk of vaccination and increased the intention to vaccinate. More harms increased the perceived risk of vaccination and, in Experiment 2 only, decreased the intention to vaccinate. More benefits increased the perceived risk of non-vaccination. Exploratory analyses showed that the comprehension and position of a comprehension test influenced risk perceptions. Limitations: The experiments used MTurk convenience samples and a fictitious disease. Conclusion: Communicating vaccine profiles in fact boxes affects risk perceptions. Additional measures may cue a deeper elaboration of the vaccine profile.
When making medical decisions (e.g., for or against vaccination), individuals should ideally know the relevant benefits and harms of the behavioral options before making an individual decision. A promising benefit-risk communication tool is the fact box (Way et al., 2016), which provides essential information on treatment (e.g., antibiotics), screening (e.g., prostate cancer), or preventive behavior (e.g., vaccination) in a simple, comprehensible, and compact table (also called a drug facts box; see Figure 1 for the fact box used in this work). It summarizes the best available clinical evidence about an intervention’s most important benefits and harms, comparing a treatment to a control group (e.g., a vaccine vs. placebo vaccine; Harding Center for Risk Literacy, 2019). Fact boxes are well understood, enable knowledge gain and an easier comparative assessment of two behavioral options than text-based summaries (Brick et al., 2020; McDowell et al., 2019; Rebitschek et al., 2022; Schwartz et al., 2007, 2009). However, the impact of fact boxes on people’s risk perceptions deserves more attention. Risk judgment is a “subjective and value-laden view” (Chauvin, 2018, p. 37). It is an integral component of many health behavior models in terms of the “possibility of loss” (van der Pligt, 1996, p. 34). A study of the risk perception in fact boxes, with a focus on the benefit aspect of vaccination (i.e., preventing the disease), showed that providing a fact box can decrease the estimated probability of COVID-19 but did not affect the fear of contraction or the perceived severity of the infection (Rebitschek et al., 2022). Since adequate risk perceptions are crucial for health decision-making (Sheeran et al., 2014), the present study aimed to provide more insights into this aspect.
Even if people read evidence-based information (e.g., a fact box), they might not derive the normatively expected inferences from it. When integrating information, people draw on, for example, prior knowledge and affective associations with the subject at hand. In the case of low prior knowledge, complex situations, and restricted opportunities for an in-depth evaluation, people tend to utilize simple rules (heuristics or mental shortcuts) to arrive at a judgment (Raue & Scholl, 2018). According to the risk-as-feeling framework, affects function as cues and heuristics in risk assessment (Tompkins et al., 2018). Despite being useful in many decision situations, mental shortcuts can lead to suboptimal, biased risk judgments. Thus, offering fact boxes could still have unintended consequences on risk perception – for example, they can decrease the perceived severity of drug adverse events in cases where serious and minor adverse events are presented simultaneously (argument dilution effect; Sivanathan & Kakkar, 2017). In this study, we consider two phenomena that use affective valence as an information cue: the inverse benefit-risk relationship, as described by the affect heuristic, and the negativity bias.
The effect heuristic postulates that the overall affective impression about an event or object systematically influences the evaluation of benefits and risks in people’s minds (Alhakami & Slovic, 1994; Finucane et al., 2000; Skagerlund et al., 2020; Slovic, Finucane, et al., 2007). Thus, judgments of benefits and risks are often inversely related, even if they are objectively independent or positively correlated (Alhakami & Slovic, 1994; Slovic, Finucane, et al., 2007). This phenomenon has also been observed in judging perceived vaccine benefits and risks, in which the high effectiveness of a vaccine was associated with lower perceived risks (Alhakami & Slovic, 1994; European Medicines Agency, 2012; Mostafapour et al., 2019), even though vaccine benefits and risks are distinct and independent events. According to the affect heuristic, providing information about benefits should change the overall affective evaluation and, thereby, the perceived risks (for experimental support of this assumption in vaccine information without evidence-based numbers, see Mostafapour et al., 2019). Thus, providing evidence-based numbers about vaccine benefits in a fact box could affect the perceived vaccination risk, despite being unrelated to the vaccine risks.
The negativity bias postulates asymmetrical weighting of events of different valences, with negative events exerting a greater influence (Baumeister et al., 2001; Rozin & Royzman, 2001). In general, individuals are more sensitive to losses than to gains (Baumeister et al., 2001; Kahneman & Tversky, 1984; Rozin & Royzman, 2001). Thus, it is not surprising that information about the existence of risk influences the perception of risk more strongly than information about its non-existence (Siegrist & Cvetkovich, 2001). The greater impact of reports about the existence of risks on perceived risk compared to the absence of risk has been demonstrated for information about vaccine adverse events (Betsch et al., 2013, 2015). Since fact boxes display both positive (vaccine benefits) and negative events (vaccine harms), the latter could receive more weight in the vaccine risk assessment.
In two experiments, we studied how communicating different benefit-risk profiles of a fictitious vaccine via fact boxes can influence the risk perception of vaccination and intention to vaccinate. In doing so, we provide more insights into how presenting information in fact boxes affects risk perception and examine whether the described mental shortcuts (inverse benefit-risk relationship and negativity bias) can be observed.
Hypotheses
Our hypotheses concentrate on vaccination versus non-vaccination based on the behavioral options presented in the fact box. Participants received different benefit-risk profiles of a fictitious vaccine and judged the risk of vaccination and their intention to vaccinate. Previous research has observed consistent relationships between risk perception of vaccination and intention to vaccinate (Betsch et al., 2013; Haase et al., 2015). In line with the inverse relationship of perceived benefits and risks described by the affect heuristic, the benefits hypothesis expects that higher benefits lead to lower risk perceptions of vaccination and higher intentions to vaccinate.
While benefits are inherently positive, adverse events have a negative expected utility for general health outcomes. Thus, the harms hypothesis assumes that the more likely harms (i.e., vaccine adverse events) are to occur, the higher the perceived risk of vaccination and the lower the intention to vaccinate.
Concerning negativity bias, we expect that the probability of harms will moderate the main effect specified in the benefits hypothesis. A high probability of vaccine adverse events should diminish the positive effect of the benefits on the perceived risk of vaccination and the intention to vaccinate (harms limit benefits hypothesis).
However, how do individuals perceive the benefits when they only read about the harms? Rothman and Salovey (1997) postulated that laypersons generally expect protective measures to provide desirable outcomes with relative certainty. In line with this, US citizens judged vaccines as relatively high in benefit and low in risk (Fischhoff et al., 1978; Slovic, Peters, et al., 2007), and 90% (tend to) agree that vaccines are effective (Larson et al., 2016). Thus, it appears reasonable to anticipate that the participants expect a high level of protection when no information regarding vaccine effectiveness is available. The perfect benefits hypothesis expects that a fact box presenting a vaccine offering perfect protection and a fact box without such information elicit the same levels of the perceived risk of vaccination and intention to vaccinate.
Experiment 1
Method
Participants and Design
The preregistered experiment implemented a 4 (benefits: no information, low, high, or perfect vaccine effectiveness) × 2 (harms: low or high probability of vaccine adverse events) between-subjects design. US citizens aged 18 years and older were recruited via Amazon Mechanical Turk (MTurk) during January and February 2018 and rewarded with US $1.00 for participation. In total, 602 Mturk workers completed the entire study without leaving the survey page. We excluded 23 workers due to repeated participation. Of the remaining 579 participants, 430 passed the comprehension test and were included in data analysis (generally positive vaccination attitude [M = 4.14, SD = 0.86]; for sociodemographic characteristics, see the supplementary material in OSF, Table E2).
Materials and Measures
Fact Box
Participants received one out of eight fact boxes about a vaccine against the fictitious disease “dysomeria,” caused by the “DS-virus” (see Figure 1 and the supplementary materials, Material Construction section, for information on the fact box design). Using a fictitious disease allowed for varying the benefits and risks independently while avoiding the ethical challenges inherent in providing false information about existing vaccines. All numbers in the fact box refer to the case of people “exposed to the DS-virus.” The fact box displayed probabilities of benefits and harms given a placebo vaccination versus the dysomeria vaccination. The upper-left table area in Figure 1 informs about the risk of infection and secondary diseases, given exposure to the pathogen while unvaccinated against the disease. The upper-right table area illustrates the vaccine benefits by displaying the reduced probability of contracting the disease. The probabilities of adverse events in the case of a placebo vaccination (lower-left table area) serve as a reference point to evaluate the harms in the case of the dysomeria vaccination (lower-right table area).
Benefits
The fact box informed the participants that dysomeria vaccine can prevent severe disease outcomes. The larger the difference in infection and secondary disease probabilities between the placebo and the dysomeria vaccine, the higher the benefits. With placebo vaccination, infection with dysomeria will occur in 48.0% of cases, given exposure to the pathogen; with dysomeria vaccination, the infection probabilities are 38.4% (low [effectiveness]), 9.1% (high), or 0.0% (perfect), depending on the condition. Thus, the total relative risk reduction in the conditions with low, high, and perfect benefits represent vaccine effectiveness rates of 20.0%, 81.0%, and 100.0%, respectively. Individuals in the placebo group will contract the secondary diseases of meningitis in 2.4% (with dysomeria vaccination: 1.9% vs. 0.5% vs. 0.0%) and impairment of motor and sensory functions in 0.5% (with dysomeria vaccination: 0.4% vs. 0.1% vs. 0.0%) of cases. In conditions with no information about benefits, the whole table section of benefits was missing, and participants read a text about the placebo probabilities.
Harms
The fact box stated that the mild vaccine adverse events were fever (low = 16.8% vs. high = 33.6%), headache (16.0% vs. 32.0%), and rash (14.6% vs. 29.2%). In each condition, the participants also received information about the adverse events of the placebo vaccine (fever 4.0%, headache 7.3%, rash 5.5%).
Dependent Variables
Participants expressed their perceived risk of vaccination (“How risky do you judge the vaccination against dysomeria to be?”) (Haase et al., 2015) on a visual analog slider from 0 = not risky at all to 100 = very risky and indicated their intention to vaccinate (“If you had the possibility to get vaccinated against dysomeria in the next week, what would you do?”) (Betsch et al., 2015) on a visual analog slider from 0 = definitely not get vaccinated to 100 = definitely get vaccinated.
Comprehension Test
The comprehension items tested whether the participants could find, compare, and perform calculations using the numbers presented in a fact box. It included four single-choice and six fill-in-the-gap questions. Numerical questions probed for the correct handling of numerical statistics by asking for specific numbers or differences between dysomeria and placebo vaccinations. Gist questions dealt with the essential meaning of the content presented (with reference to Tait et al., 2010). Participants in the two conditions with no information on vaccine effectiveness had to answer one of each question type for numerical handling and two gist questions. They passed the comprehension test if they correctly answered at least one numerical and one gist question. Participants in the six conditions with a complete fact box received six numerical and four gist questions. According to the preregistration, they should have passed the comprehension test if they had correctly answered four numerical and three gist questions. However, we had to exclude one gist question due to ambiguity in the correct answers in the two conditions with low vaccine benefits (the correct answer to “If a person received the vaccination against dysomeria, which of the following is most likely?” [answer options: “infection with dysomeria,” “fever,” and “rash”] was counterintuitive since, in this case, “infection with dysomeria” was most likely). Thus, the cut-off for the gist questions for conditions with a complete fact box was two correct answers out of the three gist questions.
Further Variables
For further variables, see the supplementary materials, section “Questionnaire Experiment 1: Further Variables.”
Procedure
Data collection and payment took place via MTurk. After the participants gave their informed consent, they reported their sociodemographic data. They then received the scenario about the fictitious disease with an incidence rate of 1 in 100,000 people per year and one of eight versions of the fact box about vaccination, depending on the assigned condition. They were to imagine that their doctor had provided them with the fact box. The disease information of the scenario and the fact box were continuously displayed throughout the experiment. Participants judged the perceived risk of vaccinating against dysomeria and indicated their intention to vaccinate. After the comprehension questions, general attitudes toward vaccination and the 5C were collected. The participants were debriefed, thanked, and instructed to refer to MTurk for payment.
Data Analysis
In contrast to previous fact box studies (e.g., Sullivan et al., 2015; Woloshin & Schwartz, 2011), we excluded participants who failed the comprehension test (preregistered). Otherwise, potential differences in the dependent variables could be attributed to the lack of understanding of the fact box content instead of the varying benefit-risk profiles. Additionally, participants who took a break during survey completion (preregistered) or took part twice (not preregistered) were eliminated. SPSS (IBM Corp, 2016) was used for the preregistered analyses of variance (ANOVAs). ANOVAs were conducted with benefits (low, high, perfect) and harms (low, high) as factors and risk perception of vaccination and intention to vaccinate as dependent variables. To test the perfect benefits hypothesis for the same dependent variables, the factor benefits included “no information” and “perfect” as factor levels. Bootstrapped standard errors and 95% confidence intervals of means were calculated (applying 1,000 iterations and the bias‐corrected and accelerated interval method) as the dependent variables were skewed. Post hoc tests were Bonferroni-corrected.
The preregistered exclusion of participants based on the posttreatment variable comprehension could cause an “imbalance [in] the sample with respect to observed or unobserved confounders” (Montgomery et al., 2018, p. 766). To explore the presence of such potential post-treatment bias, all hypothesis tests were rerun with the entire sample, including participants who failed the comprehension test. In the results section, we will only highlight deviating results; all results are reported in the supplementary materials, Tables E10–E12.
Results
Comprehension Test
Overall, 74.3% of the participants passed the gist and numerical comprehension tests. There was no statistically significant difference in passing rates depending on the conditions (each χ2-test p > .088).
Perceived Risk of Vaccination
As expected in the benefits hypothesis, the perceived risk of vaccination was higher in the condition with low benefits than in conditions with high or perfect benefits (F[2, 318] = 14.332, p < .001, ηp2 = .083; p < .001 for both post hoc tests; see the supplementary materials, Figure E2A and Table E5 for means). There was no mean difference in the perceived risk of vaccination between high and perfect benefits (p = 1.00 for the post hoc test).
The harms hypothesis predicts that the more likely it is for harms to occur, the higher the perceived risk of vaccination. This main effect just satisfied the criterion of statistical significance (F[1, 318] = 3.887, p = .050, ηp2 = .012). When using the entire sample to test for potential post-treatment bias, the main effect failed to satisfy the criterion of statistical significance (supplementary materials, Table E9). The harms limit benefits hypothesis, which expects an interaction effect, did not hold (F < 1).
In the perfect benefits hypothesis, we expected that when information on the vaccine benefits is absent, participants would assume 100% effectiveness. This was not the case (see the supplementary materials, Figure E2B and Table E6 for means). The result of the 2 (benefit: no info, perfect) × 2 (harms: low, high) analysis of variance showed that the perceived risk of vaccination was higher in the no information condition compared to the condition with 100% effectiveness (F[1, 209] = 18.787, p < .001, ηp2 = .082). There was no main effect for harms (F[1, 209] = 1.790, p = .182, ηp2 = .008) and no interaction effect (F[1, 209] = 1.277, p = .26, ηp2 = .006). The perfect benefits hypothesis was thus rejected.
Intention to Vaccinate
The same analyses were repeated with the intention to vaccinate as the dependent variable. Means are displayed in the supplementary materials, Figure E2C and Table E7. We found a significant main effect for benefits (F[2, 318] = 18.042, p < .001, ηp2 = .102), indicating that more beneficial vaccines increased the intention to vaccinate (p < .001 for all post hoc tests except benefits high vs. perfect). There was no significant main effect for harms (F[1, 318] = 2.406, p = .122, ηp2 = .008) and no significant interaction effect (F[2, 318] = 1.481, p = .23, ηp2 = .009); thus, the harms hypothesis and harms limit benefits hypothesis was rejected.
The test of the perfect benefits hypothesis (see the supplementary materials, Figure E2D and Table E8 for means) revealed that the intention to vaccinate was lower when there was no information than when there was information that the vaccine was 100% effective (F[1, 209] = 10.287, p = .002, ηp2 = .047). Again, there was no significant main effect of harms and no significant interaction effect (all Fs < 1).
Experiment 2
Experiment 2 pursued three aims. First, it attempted to bolster confidence in the findings of Experiment 1 by replicating them with slightly adapted materials and measurements (omitting the perfect benefits hypothesis). Second, it explored the potential confounding effects of the experimental setup on forming risk perceptions. Previous studies have included comprehension tests before assessing risk perception as one of many outcomes (Schwartz et al., 2009; Sullivan et al., 2015; Woloshin & Schwartz, 2011). This procedure could lead to more deliberation and thus influence risk perception. To explore a potential order effect, we varied the position of the comprehension test in Experiment 2.
Third, we explored the effect of communicating the benefit-risk profile on the perceived risk of non-vaccination as the counterpart to the risk of vaccination. Some theories of health-protective behavior explicitly assume that individuals compare the risk of maintaining their current behavior (e.g., non-vaccination) to the risk of adopting protective behavior (e.g., vaccination; Weinstein, 1993). The fact box about vaccination provides explicit information about both health behaviors. In Experiment 1, we focused on the risk of adopting preventive behavior. The adverse events caused by the vaccine (the fact box section “harms”) are a source of risk for the behavior “vaccination.” The probability of the infectious disease is lower for vaccinated individuals and thus the fact box section “benefits” counts as benefits in terms of the behavior “vaccination.” From the perspective of the behavior “non-vaccination,” the probability of the disease harms is the source of risks (the fact box section “benefits”), while vaccine adverse events can be avoided altogether. A rational actor should avoid the behavior of “non-vaccination” because the probability and severity of disease harms are higher than the probability and severity of vaccine harms. To form the intention to vaccinate, both health behavior options should be considered. Thus, in Experiment 2, we explored the impact of different vaccine risk profiles on the perceived risk of non-vaccination.
Method
Participants and Design
Participants were randomly assigned to a preregistered 2 (benefits: low or high vaccine effectiveness) × 2 (harms: low or high probability of vaccine adverse events) × 2 (position of comprehension test: after or before dependent variables) between-subjects design. Recruited MTurk workers were 18 years and older, had an approval rate of at least 90%, and had participated in neither the first experiment nor a pretest of the comprehension test. Data were collected in April 2019, and participation was rewarded with US $1.00. In total, 793 MTurk workers completed the entire study without leaving the survey page, and 541 passed the comprehension test. They generally had a rather positive vaccination attitude (M = 4.25, SD = 0.86; for sociodemographic characteristics, see the supplementary materials, Table E13).
Materials and Measures
Fact Box
Participants received fact boxes similar to those in Experiment 1. The design was slightly changed for a better orientation. The reference group size was 100 people (instead of 1,000 as in Experiment 1), and the outcome probabilities were displayed without decimals. Compared to Experiment 1, there was a higher risk for secondary diseases caused by the fictitious disease dysomeria (without vaccination: meningitis risk in Experiment 2 = 10% vs. Experiment 1 = 7.3%; impairment risk = 5% vs. 0.5%) in order to state absolute numbers > 0 for each outcome. Likewise, the probabilities for secondary diseases after dysomeria vaccination increased (meningitis risk: 8% for low vaccine effectiveness vs. 2% for high vaccine effectiveness; impairment risk: 4% vs. 1%).
Dependent Variables
Participants expressed their perceived risk of vaccination (“In your opinion, how risky is the dysomeria vaccination?”) and their perceived risk of non-vaccination (“In your opinion, how risky is it to omit the dysomeria vaccination?”) using a visual analog slider from 0 = not risky at all to 100 = very risky. For the intention to vaccinate the same item as in Experiment 1 was used.
Comprehension Test
As in Experiment 1, participants had to demonstrate a sufficient understanding of the fact box content. To account for the observed ambiguity in Experiment 1, a new comprehension test was constructed with reference to Woloshin and Schwartz (2011) and pretested. It consisted of eight true/false questions, one fill-in-the-gap question, and one single-choice question. To answer the questions, participants had to extract the gist, find the correct numbers in the table, and perform calculations. Although the ratio of items that required only finding numbers, comparing them, or performing a calculation with them was the same as in Experiment 1, there were fewer questions in which participants had to fill in the gaps by themselves. The participants passed the comprehension test when they had at least 7 correct answers out of 10 comprehension questions (Woloshin & Schwartz, 2011).
Attention Check
Two attention checks were included and consisted of an instruction and a question about the favorite color or political interest. The participants had to select one particular answer as described in the instruction text instead of stating their real preference to pass the attention checks (Berinsky et al., 2014). As preregistered, results were explored when participants who failed both attention checks were excluded.
Further Variables
For further variables, see the supplementary materials, section “Questionnaire Experiment 2: Further Variables.”
Procedure
The procedure was the same as in Experiment 1, except that we experimentally varied the position of the comprehension test (before or after the dependent variables).
Data Analysis
As preregistered, analyzed data only included complete participation with comprehension scores of at least 70%. SPSS (IBM Corp, 2016) was used for the 2 × 2 × 2 analyses of variance with bootstrapped standard errors and 95% confidence intervals of means, applying 1,000 iterations and the bias‐corrected and accelerated interval method. Excluding participants who failed both attention checks resulted in the same results for the hypotheses tests and exploratory analyses (see the supplementary materials, Tables E18–E21). As in Experiment 1, we also analyzed the hypotheses using the entire sample to explore the presence of a potential post-treatment bias (see the supplementary materials, Tables E22–E24); we report only deviating results in the Results section.
Results
Comprehension Test
Overall, 68.2% of the participants passed the comprehension tests. There was no statistically significant difference between the eight conditions (χ2-test p = .75).
Perceived Risk of Vaccination
As expected in the benefits hypothesis, the 2 × 2 × 2 analysis of variance revealed that a higher vaccine benefit led to a lower perceived risk of vaccination (F[1, 533] = 30.490, p < .001, ηp2 = .054; see supplementary materials, Figure E4A and Table E14 for means).
Also, the main effect of harms on the perceived risk of vaccination was significant (F[1, 533] = 6.595, p = .010, ηp2 = .012; providing evidence for the harms hypothesis). The more likely it was for vaccine adverse events to occur, the higher the perceived risk of vaccination.
We found no significant main effect for the position of the comprehension test and no significant interaction effects (all Fs < 2), with the exception of the interaction between position and benefits (F[1, 533] = 5.073, p = .025, ηp2 = .009). When vaccine benefits were high, the perceived risk of vaccination was similar regardless of the position of the comprehension test. However, when benefits were low, completing the comprehension test first increased the perceived risk of vaccination compared to testing comprehension after risk assessment.
Intention to Vaccinate
A 2 × 2 × 2 analysis of variance with the intention to vaccinate as a dependent variable revealed two main effects and no statistically significant interaction effect (all Fs < 2; see the supplementary materials, Figure E4B and Table E15 for means). As expected in the benefits hypothesis, higher benefits increased the intention to vaccinate (F[1, 533] = 53.102, p < .001, ηp2 = .091). Additionally, higher harms decreased the intention to vaccinate (F[1, 533] = 4.931, p = .027, ηp2 = .009; harms hypothesis). Using the entire sample failed to show a significant main effect, as described in the harms hypothesis (see the supplementary materials, Table E23). In summary, variation in benefits and (only given a sufficient comprehension score) harms affected the intention to vaccinate, whereas the position of the comprehension test in the survey did not.
Exploration
Perceived Risk of Non-Vaccination
We explored whether the manipulated factors also affected the perceived risk of non-vaccination. Greater vaccine benefits indeed increased the perceived risk of non-vaccination (F[1, 533] = 55.530, p < .001, ηp2 = .094; see the supplementary materials, Figure E4C and Table E16 for means). Neither harms (F < 1) nor the comprehension test (F[1, 533] = 3.421, p = .065, ηp2 = .006) affected the perceived risk of non-vaccination. However, there was a significant interaction between harms and the comprehension test (F[1, 533] = 4.787, p = 0.029, ηp2 = .009): only in the case of low harms did complete the comprehension test first lead to a higher perceived risk of non-vaccination than completing it after judging the risk perception. This interaction was not significant when the entire sample is considered (see the supplementary materials, Table E24). All other interactions were nonsignificant (all Fs < 1).
Non-Comprehension
We conducted t-tests for independent samples to test for the influence of comprehension on the dependent variables. The perceived risk of vaccination (t[570.68] = 8.695, p < .001, d = 0.663) and the perceived risk of non-vaccination (t[657.44] = 5.816, p < .001, d = 0.444) were higher for participants who failed the comprehension test. This was also true for the intention to vaccinate (t[706.91] = 4.265, p < .001, d = 0.325) (see the supplementary materials, Table E20 for means).
General Discussion
The two experiments show that individuals incorporate the presented vaccine fact box content into their risk perception and vaccination intention. In doing so, it became apparent that they use a mental shortcut, as described by the inverse benefit-risk relationship (benefits hypothesis). Furthermore, this inverse relationship and the absence of the negativity bias (harms limit benefits hypothesis) highlight the importance of communicating the vaccine benefits. Overall, the finding that presenting benefit-risk profiles positively affects the intention to get vaccinated matches that of Kaplan and Milstein (2021) in the context of COVID-19.
Previous research has shown that the magnitude of a drug’s effectiveness influences the perceived risk of treatment – the more benefits that are present, the fewer risks are perceived (O’Donoghue et al., 2014; Sullivan et al., 2015). We demonstrated the same relationship in the field of prevention, suggesting that the mechanisms of the affect heuristic are at work. The presented fact box should elicit a deliberative weighing of behavioral options. However, anticipated regret could be a part of these deliberations. We cannot judge whether the effect is the underlying mechanism shaping the inverse relationship since we did not measure affective outcomes or processes. Further research should thus assess emotional reactions and examine their potential effects on risk perceptions.
In addition to the inverse benefit-risk relationship, our results for omitting the benefits information and for the risk of non-vaccination further underline the significance of communicating vaccine benefits. When the vaccine benefits were not presented, the perceived risk of vaccination was similar to a vaccine with low effectiveness. Also, the intention to vaccinate was lower in the case of no benefit information compared to the condition of a perfectly effective vaccine (see the supplementary materials, Tables E5–E8 for means). In reality, the clinical efficacy of many vaccines is above 80% (Hamborsky et al., 2015). Thus, communicating the vaccine benefits should also lower the perceived risk of vaccination with real vaccines. Additionally, higher benefits also led to a higher perceived risk of non-vaccination in Experiment 2. Both risk assessments are important for health-protective behavior, which can be modeled as a trade-off between the risk of maintaining the current behavior and the risk of adopting protective behavior (Weinstein, 1993). Taken together, communicating vaccine benefits could lead to more accurate risk perception and greater intention to vaccinate for most vaccines.
The finding that harms can influence the perceived risk of health behaviors has been shown before in the context of medication (Sullivan et al., 2015). It is reasonable to avoid adverse events because they threaten our health. The more interesting question could be the threshold of the accepted probability and severity of adverse events that are acceptable for preventing an uncertain, potentially lethal, or quite severe infection. Contrary to our expectations, we did not observe an interaction between benefits and harms. To create realistic benefit-risk profiles for the fictitious vaccine, it claimed to prevent more severe outcomes with a greater probability than it caused less severe adverse events. The resulting risk of disease could have outweighed the risk of vaccine adverse events – especially if participants did not pay sufficient attention to the conditional character of the disease probability (i.e., the table-caption “exposed to DS-virus”).
The thorough comprehension of the fact box content was relevant for risk assessment in two ways. First, the comprehension test may serve as a proxy for numeracy or table literacy. Previous studies did not assess the effect of comprehension on perceived risk. Nevertheless, less numerate people are assumed to draw less precise and less significant meanings from numbers (Peters et al., 2006; Reyna, 2004). Our explorative results showed that risk perception could be distorted if the fact box content was not sufficiently understood. Second, the position of the comprehension test influenced the perceived risk when the more thoroughly elaborated aspect was not central for risk assessment (benefits in the case of the perceived risk of vaccination and harms in the case of the perceived risk of non-vaccination; the latter result is limited to the subsample of participants sufficiently comprehending the fact box). The comprehension test seems to stimulate an examination of the fact box content and, thus, could influence risk assessment. This insight has two implications: (1) Previous studies that assessed risk perceptions after comprehension questions should be interpreted with caution; (2) Asking comprehension questions could be used to deepen elaboration.
Overall, the observed effects were small in cases where the participants comprehended the presented information – suggesting that even thoroughly crafted evidence-based information plays a secondary role in forming risk perception and vaccination intention. Evidence-based information can be rather abstract and lack relevance for the everyday life decisions and experiences of people. Further research could test whether combining the fact box with storytelling generates more interest in the information, can enhance comprehension, and help to elaborate key messages (Dahlstrom, 2014) without introducing unintended effects (e.g., persuasion).
However, vaccination is a complex measure of protection. Information on the disease’s incidence, contagiousness, and current vaccination rates was not communicated in the present fact boxes – yet, these indicators could be important for the decision as well (e.g., perceiving low disease risk as a barrier to influenza vaccine uptake; Schmid et al., 2017). Importantly, most vaccines have indirect effects in addition to the direct effects of preventing disease (i.e., they reduce the transmission of pathogens and thus have a social benefit, too). Presenting this particular benefit can be challenging in the current versions of the fact boxes. Demonstrating the positive benefit for others and the individual may be a useful and effective further development of fact boxes – and depict a more comprehensive picture of the benefits of vaccination.
Limitations
The participants were recruited via MTurk. Using MTurk as a participant pool is widely practiced; however, it has also been repeatedly criticized. It is recommended to take measures such as attention checks to improve data quality when recruiting via this platform (Buhrmester et al., 2018). Generally, we restricted the main analyses to participants who passed the comprehension tests. Included participants paid sufficient attention and used adequate care to pass the test. Thus, we assume that they applied the same carefulness in completing the rest of the questionnaire. In Experiment 2, we restricted participation by applying an approval rate of at least 90% and adopted attention checks. MTurk samples are more heterogeneous than the standard student sample and satisfy psychometric publication standards (Buhrmester et al., 2011; Mason & Suri, 2012; Paolacci et al., 2010).
To examine risk perceptions given the fact box content is well understood, we preregistered to exclude the participants after the treatment. Exploratory analyses forgoing this exclusion criterion showed that some of the weak effects were not found when using the full sample, hinting at the presence of confounders. Thus, the results must be strictly interpreted for cases in which the fact box is sufficiently comprehended.
We used conditional probabilities for vaccine benefits because publicly available vaccine fact boxes also use them. It remains unclear how much attention participants paid to the conditional probability in the form of the reference group “exposed to DS-virus” when expressing their risk. Further research could address this point by comparing conditional and unconditional probabilities in this communication format to explore if aspects such as very small probabilities and class references (e.g., with confirmed exposure vs. in general) influence the risk perception in the context of vaccination.
The fact box compared vaccinated people to those who received a placebo. The placebo was explained to the participants as receiving a saline injection instead of the dysomeria vaccine. Some of the participants might not have understood these technical terms and thus drew less meaningful interpretations of the information.
We used a fictitious disease to configure the benefit-risk profile in an unconstrained way. Perceptions and decisions regarding the fictitious scenario have no real-life impact on participants and might thus have restricted external validity. Further research could test the effect of communicating vaccine benefits and harms on the risk perception of an actual vaccine-preventable disease or use a research paradigm where hypothetical decisions have consequences for the amount of monetary incentive (Böhm et al., 2016; Hertwig & Ortmann, 2001).
Conclusion
Since vaccine safety is publicly discussed for several vaccines (Larson et al., 2011), it is important to offer evidence-based information about the safety profile of recommended vaccines. Communicating a vaccine’s benefits – that is, its power to effectively prevent disease and severe conditions – via fact boxes can increase the perceived risk of omitting vaccination and decrease the perceived risk of vaccination. However, fact boxes about vaccinations do not convey one single and straightforward message. The recipients need to elaborate on the content to assess and weigh the risk of two health behaviors. Unintended effects on risk perceptions may occur when readers do not comprehend the content. Readers could benefit from measures that cue deeper processing of the fact box content and can facilitate the comprehension of how the vaccine omission or uptake affects a person without relying on numbers. Further research could test whether combining the fact box with those measures can facilitate the assessment of vaccination outcomes without introducing further unintended effects.
The authors thank Lars Korn for his valuable comments on earlier versions of this manuscript.
References
(1994). A psychological study of the inverse relationship between perceived risk and perceived benefit. Risk Analysis, 14(6), 1085–1096. https://doi.org/10.1111/j.1539-6924.1994.tb00080.x
(2001). Bad is stronger than good. Review of General Psychology, 5(4), 323–370. https://doi.org/10.1037/1089-2680.5.4.323
(2014). Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58(3), 739–753. https://doi.org/10.1111/ajps.12081
(2015). The narrative bias revisited: What drives the biasing influence of narrative information on risk perceptions? Judgment and Decision Making, 10(3), 241–264. https://doi.org/10.1017/S1930297500004654
(2013). Effect of narrative reports about vaccine adverse events and bias-awareness disclaimers on vaccine decisions: A simulation of an online patient social network. Medical Decision Making, 33(1), 14–25. https://doi.org/10.1177/0272989X12452342
(2016). Selfish-rational non-vaccination: Experimental evidence from an interactive vaccination game. Journal of Economic Behavior & Organization, 131, 183–195. https://doi.org/10.1016/j.jebo.2015.11.008
(2020). Risk communication in tables versus text: A registered report randomized trial on “fact boxes”. Royal Society Open Science, 7(3), Article
190876 . https://doi.org/10.1098/rsos.190876(2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. https://doi.org/10.1177/1745691610393980
(2018). An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science, 13(2), 149–154. https://doi.org/10.1177/1745691617706516
(2018).
Individual differences in the judgment of risks: Sociodemographic characteristics, cultural orientation, and level of expertise . In M. RaueE. LermerB. StreicherEds., Psychological perspectives on risk and risk analysis (pp. 37–61). Springer International Publishing. https://doi.org/10.1007/978-3-319-92478-6_2(2014). Using narratives and storytelling to communicate science with nonexpert audiences. Proceedings of the National Academy of Sciences, 111(supplement_4), 13614–13620. https://doi.org/10.1073/pnas.1320645111
. (2012). Benefit-risk methodology project – Report on risk perception study module, EMA. https://www.ema.europa.eu/en/documents/report/benefit-risk-methodology-project-report-risk-perception-study-module_en.pdf
(2023, May 30). Data and materials for “How communicating vaccine benefits and harms in fact boxes affects risk perceptions: Two randomized trials”. https://doi.org/10.17605/OSF.IO/KZ2BQ
(2000). The affect heuristic in judgments of risks and benefits. Journal of Behavioral Decision Making, 13(1), 1–17. https://doi.org/10.1002/(SICI)1099-0771(200001/03)13:1<1::AID-BDM333>3.0.CO;2-S
(1978). How safe is safe enough? A psychometric study of attitudes towards technological risks and benefits. Policy Sciences, 9(2), 127–152. https://doi.org/10.1007/BF00143739
(2015). Source credibility and the biasing effect of narrative information on the perception of vaccination risks. Journal of Health Communication, 20(8), 920–929. https://doi.org/10.1080/10810730.2015.1018605
(2015). Epidemiology and prevention of vaccine-preventable diseases (13th ed.). Public Health Foundation. https://www.cdc.gov/vaccines/pubs/pinkbook/index.html
. (2019, May 22). Faktenboxen
[Fact boxes] . https://www.harding-center.mpg.de/de/faktenboxen(2001). Experimental practices in economics: A methodological challenge for psychologists? The Behavioral and Brain Sciences, 24(3), 383–403. discussion 403–451. https://doi.org/10.1037/e683322011-032
. (2016). IBM SPSS statistics for Windows (24.0). https://www.ibm.com/support/pages/downloading-ibm-spss-statistics-24
(1984). Choices, values, and frames. American Psychologist, 39(4), 341–350. https://doi.org/10.1037/0003-066X.39.4.341
(2021). Influence of a COVID-19 vaccine’s effectiveness and safety profile on vaccination acceptance. Proceedings of the National Academy of Sciences, 118(10), Article
e2021726118 . https://doi.org/10.1073/pnas.2021726118(2011). Addressing the vaccine confidence gap. The Lancet, 378(9790), 526–535. https://doi.org/10.1016/S0140-6736(11)60678-8
(2016). The state of vaccine confidence 2016: Global insights through a 67-country survey. EBioMedicine, 12, 295–301. https://doi.org/10.1016/j.ebiom.2016.08.042
(2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23. https://doi.org/10.3758/s13428-011-0124-6
(2019). Effect of tabular and icon fact box formats on comprehension of benefits and harms of prostate cancer screening: A randomized trial. Medical Decision Making, 39(1), 41–56. https://doi.org/10.1177/0272989X18818166
(2018). How conditioning on posttreatment variables can ruin your experiment and what to do about it: Stop conditioning on posttreatment variables in experiments. American Journal of Political Science, 62(3), 760–775. https://doi.org/10.1111/ajps.12357
(2019). Exploring the effect of risk and benefit information provision on vaccination decision-making. Vaccine, 37(44), 6750–6759. https://doi.org/10.1016/j.vaccine.2019.08.083
(2014). Randomized study of placebo and framing information in direct-to-consumer print advertisements for prescription drugs. Annals of Behavioral Medicine, 48(3), 311–322. https://doi.org/10.1007/s12160-014-9603-1
(2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419. https://doi.org/10.1017/S1930297500002205
(2006). Numeracy and decision making. Psychological Science, 17(5), 407–413. https://doi.org/10.1111/j.1467-9280.2006.01720.x
(2018).
The use of heuristics in decision making under risk and uncertainty . In M. RaueE. LermerB. StreicherEds., Psychological Perspectives on Risk and Risk Analysis (pp. 153–179). Springer International Publishing. https://doi.org/10.1007/978-3-319-92478-6_7(2022). Fact boxes that inform individual decisions may contribute to a more positive evaluation of COVID-19 vaccinations at the population level. PLoS One, 17(9), Article
e0274186 . https://doi.org/10.1371/journal.pone.0274186(2004). How people make decisions that involve risk: A dual-processes approach. Current Directions in Psychological Science, 13(2), 60–66. https://doi.org/10.1111/j.0963-7214.2004.00275.x
(1997). Shaping perceptions to motivate healthy behavior: The role of message framing. Psychological Bulletin, 121(1), 3–19. https://doi.org/10.1037/0033-2909.121.1.3
(2001). Negativity bias, negativity dominance, and contagion. Personality and Social Psychology Review, 5(4), 296–320. https://doi.org/10.1207/S15327957PSPR0504_2
(2017). Barriers of influenza vaccination intention and behavior – A systematic review of influenza vaccine hesitancy, 2005–2016. PLoS One, 12(1), Article
e0170550 . https://doi.org/10.1371/journal.pone.0170550(2007). The drug facts box: Providing consumers with simple tabular data on drug benefit and harm. Medical Decision Making, 27(5), 655–662. https://doi.org/10.1177/0272989X07306786
(2009). Using a drug facts box to communicate drug benefits and harms: Two randomized trials. Annals of Internal Medicine, 150(8), 516–527. https://doi.org/10.7326/0003-4819-150-8-200904210-00106
(2014). Does heightening risk appraisals change people’s intentions and behavior? A meta-analysis of experimental studies. Psychological Bulletin, 140(2), 511–543. https://doi.org/10.1037/a0033065
(2001). Better negative than positive? Evidence of a bias for negative information about possible health dangers. Risk Analysis, 21(1), 199–206. https://doi.org/10.1111/0272-4332.211102
(2017). The unintended consequences of argument dilution in direct-to-consumer drug advertisements. Nature Human Behaviour, 1(11), 797–802. https://doi.org/10.1038/s41562-017-0223-1
(2020). The affect heuristic and risk perception – Stability across elicitation methods and individual cognitive abilities. Frontiers in Psychology, 11, Article
970 . https://doi.org/10.3389/fpsyg.2020.00970(2007). The affect heuristic. European Journal of Operational Research, 177(3), 1333–1352. https://doi.org/10.1016/j.ejor.2005.04.006
(2007). Risk perception of prescription drugs: Results of a national survey. Drug Information Journal, 41(1), 81–100. https://doi.org/10.1177/009286150704100110
(2015). Communicating benefit and risk information in direct-to-consumer print advertisements: A randomized study. Therapeutic Innovation & Regulatory Science, 49(4), 493–502. https://doi.org/10.1177/2168479015572370
(2010). Effect of various risk/benefit trade-offs on parents’ understanding of a pediatric research study. Pediatrics, 125(6), e1475–e1482. https://doi.org/10.1542/peds.2009-1796
(2018).
Emotional aspects of risk perceptions . In M. RaueE. LermerB. StreicherEds., Psychological perspectives on risk and risk analysis (pp. 109–130). Springer International Publishing. https://doi.org/10.1007/978-3-319-92478-6_5(1996). Risk perception and self-protective behavior. European Psychologist, 1(1), 34–43. https://doi.org/10.1027/1016-9040.1.1.34
(2016). Pharmaceutical benefit–risk communication tools: A review of the literature. Drug Safety, 40(1), 15–36. https://doi.org/10.1007/s40264-016-0466-1
(1993). Testing four competing theories of health-protective behavior. Health Psychology: Official Journal of the Division of Health Psychology, American Psychological Association, 12(4), 324–333. https://doi.org/10.1037/0278-6133.12.4.324
(2011). Communicating data about the benefits and harms of treatment: A randomized trial. Annals of Internal Medicine, 155(2), 87–96. https://doi.org/10.7326/0003-4819-155-2-201107190-00004
