The Challenge of Inferring Unconscious Mental Processes

Research on the nature and scope of unconscious or implicit mental processes has long been a core concern of cognitive psychologists, and interest in the topic shows no signs of abating (e.g., Blake, 2021; LeDoux et al., 2020). Reasons for this interest are not hard to find. For example, our perceptual experience is strikingly impoverished (Cohen et al., 2021), raising the question of how much impact events have on us when they are not consciously perceived. Similarly, we appear to be able to acquire complex skills such as language with little awareness of the underlying rules (Chomsky, 1980).

When researchers explore unconscious influences in learning, memory, action, perception, decision making, cognitive control, emotion, and other domains, they commonly employ experimental tasks in which two measurements are taken: One is the primary performance index and the other is a measurement of the participant's awareness of whatever critical stimulus feature or relationship is assumed to drive performance in the primary task. For instance, speed of responding to briefly presented angry or neutral faces might be measured together with perceptual awareness (e.g., discrimination) of facial expression (Hedger et al., 2016), or learning of an advantageous decision strategy might be related to verbal reports of the strategy (Bechara et al., 1997). Although there are other general approaches for studying unconscious processes – for example, research on the Unconscious Thought Effect and the Implicit Association Test (IAT) takes different approaches (see Abadie and Waroquier, 2019, and Greenwald and Banaji, 2017) – many hundreds of studies conducted over the past several decades can be conceptualized within this framework of bivariate measurement.

Given how common the practice is of collecting bivariate performance/awareness data, one might imagine that there has been rich discussion about the general strategies available to researchers for analyzing and interpreting such data. After all, inferring unobservable mental processes from patterns of observable behavior is one of the most fundamental and challenging problems facing psychological science (Kellen et al., 2021). Yet with rather few exceptions (e.g., Dienes, 2015; Greenwald et al., 1995; Sand & Nilsson, 2016; Stephens et al., 2019), very little attention has been devoted to evaluating the adequacy of different methods for inferring unconscious mental processes from bivariate data or to examining the assumptions on which they rest.

We undertake such an assessment in the current article. By and large, we take a relatively abstract viewpoint, describing particular experimental results only when they help to illustrate a general point. This article is not intended as a review of research on unconscious mental processes, which can be found elsewhere (e.g., Greenwald & Banaji, 2017; Hassin, 2013; Hedger et al., 2016; Lovibond & Shanks, 2002; Mertens & Engelhard, 2020; Newell & Shanks, 2014; Nobre et al., 2020; Phillips, 2021; Weiskrantz, 1997). Instead, we critically examine the adequacy of various inferential approaches that are almost taken for granted by many researchers in this field. We show that the inferential challenge is far more profound than is generally acknowledged and offer suggestions for how future research on this fundamental topic might surmount this challenge.

There has of course been extensive debate about what constitutes an acceptable method for measuring awareness, evaluating the relative merits of verbal reports, visibility ratings, discrimination at the subjective and objective thresholds, confidence reports, and so on (Blake, 2021; Cheesman & Merikle, 1984; Dienes & Scott, 2005; Ericsson & Simon, 1984; Merikle & Reingold, 1992; Peirce & Jastrow, 1884; Sandberg et al., 2010; Shanks & St. John, 1994). Here, we ignore these deep issues and take it as read that data have been collected employing a suitable awareness measure.

Theoretical and Conceptual Context

Construct validation and formal models play crucial roles in the enterprise of inferring mental processes (Cronbach & Meehl, 1955; Grahek et al., 2021). The general logic has two aspects. The first is to articulate theoretical constructs, employ psychometric methods to measure them reliably, and establish their validity. The second is to place them within a nomological framework, that is, to formulate a model embodying the constructs and their putative interrelationships, although invariably simplifying the domain in which it is to be applied. This model can be purely statistical (e.g., regression, factor analysis, and structural equation modeling) or generative (e.g., a computational model is constructed that generates simulated data, which can then be compared against the actual data collected empirically). To the extent that model and data align quantitatively and qualitatively, the model gains some support, especially if it outperforms alternative models, although of course it can never be proven that the model is “true.” The more complex the data set, in terms of the number of manipulated independent variables and outcome measures, the harder the task for the modeler, but the more meaningful is a good fit to the data.

From the perspective of implicit or unconscious mental processes, research on the IAT provides an example of the application of this general approach. The theory that the IAT provides a measure of implicit attitudes that are dissociable from explicit attitudes can be tested by collecting data on participants' performance on the IAT and related tests as well as their responses on standard explicit attitude questionnaires. The psychometric properties of these measures are assessed. Analytic techniques such as structural equation modeling can then be employed to test different models, particularly to compare a model that has a single latent attitude that determines responses on both implicit and explicit tests versus one that has distinct latent attitudes, one controlling explicit attitudes and the other implicit ones. Manipulations can be introduced which are intended to have selective influence on one type of attitude or the other. Although there is much controversy about the resulting data patterns and their interpretation (Kurdi et al., 2019; Nosek & Smyth, 2007; Schimmack, 2021), the general methodology is well-understood and relatively uncontroversial. For examples from different domains of unconscious processing, see West et al. (2021) for language development and Kaufman et al. (2010) for implicit learning.

When we look more broadly at the methods researchers use to establish the existence or otherwise of unconscious mental processes, we soon find that the above example is the exception rather than the rule. Instead of complex data sets involving multiple independent variables and outcome measures, psychometrically analyzed and validated, and contrasted against the predictions of competing theoretical models, researchers collect very simple bivariate data and approach these data from a straightforward analytic perspective. For example, in one of the most influential studies on unconscious processes, Bechara et al. (1997) obtained data across participants on a measure of performance (selecting advantageous cards in risky choice in the Iowa Gambling Task) and a measure of awareness (responses to probe insight questions) and concluded that advantageous selections were made when participants lacked awareness of the choice payoffs. Similarly, in another hugely influential study, Warrington and Weiskrantz (1968) measured performance on a priming measure (identifying degraded pictures or words) and a conscious memory measure across individuals with amnesia and reported priming in the absence of detectable conscious memory. As a final example, Dehaene et al. (1998) reported that briefly presented and masked (“subliminal”) words affected semantic judgments even though participants performed at chance on an assessment of awareness of the masked words. Of course these studies were much more sophisticated than this very brief overview suggests, but from the perspective of their inferential logic, they all relied on interpretation of patterns observed in bivariate data. Similar data form the basis of inference in numerous other studies published over the past 50 years.

In adopting such a minimalist form of data, these studies draw heavily for their inferential justification on the dissociation logic that had its heyday in the 1980s and 1990s (Dunn & Kirsner, 1988; Shallice, 1988). It was the hope of this research program that mental structure could be revealed by patterns of selective influence. For example, if damage to a particular brain structure led to deficits in performing task A but not task B (or even better, if damage to brain structure 1 led to deficits in performing task A but not task B, while damage to brain structure 2 led to deficits in performing task B but not task A), then functionally distinct mental and neural systems could be identified. Among many lines of evidence speaking against this strong form of inference (e.g., Kinder & Shanks, 2003; Munakata, 2001; Plaut, 1995), developments in state-trace analysis (Newell & Dunn, 2008; Stephens et al., 2019; Yeates et al., 2015) showed that dissociation patterns are rarely inconsistent with explanations based on a single latent construct that has monotonic but distinct mappings onto the measured dependent variables. For this reason, as well as the finding that many classic examples in the implicit/explicit domain have proven hard to replicate (e.g., Berry et al., 2014; Berry et al., 2017; Le Pelley et al., 2019; Ward et al., 2020; Zaki et al., 2003), dissociation logic has largely been abandoned in recent years. We return to this issue later.

Bivariate Data: The Basic Problem

What is the problem with using bivariate performance/awareness data as the basis for inferences about unconscious processes? At first glance, the logic seems straightforward. Consider the artificial data set shown in Figure 1A. Each datapoint represents a single participant, with their mean performance (such as response facilitation) across a set of trials on the y-axis plotted against a mean measure of awareness of a critical stimulus feature or relationship on the x-axis. The scale values are unimportant, but we plot performance on a scale going up to hundreds (as might be appropriate for a millisecond measure based on response times [RTs]) and awareness on a scale that might be relevant for a discriminability measure such as d' from signal detection theory. In both cases, we assume that zero reflects the baseline or chance level of that measure. Figure 1A shows a “holy grail” pattern in which the vast majority of a large sample of participants score above chance on the performance measure while scoring at chance on the awareness measure. Indeed, these data are generated from a model in which performance is truly unrelated to awareness, and awareness is completely absent.

Specifically, for a given participant, we assume that performance P = a · S_P + e_P where S_P is a normally distributed variable representing whatever latent mental content or representation determines the individual's level of performance strength, a is a constant scaling factor, and e_P is a random error term (see Table 1). This error represents all potential sources of noise, both intrinsic to the individual such as trial-by-trial variability in attention and extrinsic such as measurement error. At the same time, we calculate the participant's level of awareness from the equation A = b · S_A + e_A where S_A represents the latent mental content that determines the individual's measured level of awareness, b is another constant scaling factor, and e_A is another source of random error. These equations may seem unnecessarily complex as the S and e terms in each formula could trivially be replaced by a single variable, but we keep them distinct to emphasize that S represents an unobservable latent construct, the very entity that our analysis is attempting to identify, and that this construct varies across individuals – some perform better and some worse in the experimental task. Table 1 gives details of the means and SDs of the normal distributions from which various parameters are sampled and the values of a and b. None of the points made in this article depend particularly on these values – they are chosen to yield patterns that bear some approximate similarity to typical experimental results. Note that the error terms always have a mean of zero. This model is designated Model 2 to emphasize the fact that it includes 2 latent processes, S_P and S_A.

Table 1 Models used to generate the data in Figure 1

Model	Performance	Awareness	Parameters
Note. Model names indicate whether 1 or 2 latent processes are included. The normal distribution N () is parameterized by its M and SD.
Model 2	P = a · SP + eP	A = b · SA + eA	SP ∼ N (1.0, 0.2) a = 100 eP ∼ N (0, 80) SA ∼ N (0, 0.1) b = 1.0 eA ∼ N (0, 0.1)
Model 1a	P = a · S + eP	A = b · S + eA	S ∼ N (1.0, 0.2) a = 100 eP ∼ N (0, 80) b = 1.0 eA ∼ N (0, 1.0)
Model 1b			S ∼ N (1.0, 1.0) a = 100 eP ∼ N (0, 80) b = 0.3 eA ∼ N (0, 1.0)

Table 1 Models used to generate the data in Figure 1

View as image HTML

What exactly do the S parameters represent? They are intended to be very general theoretical constructs that simply reflect whatever unobservable mental state determines a particular measurable behavior, be it on a performance or an awareness scale. In visual perception, S could represent the amount of information picked up by the senses about an external object. In the case of learning and memory, it could represent the state of acquired/retained knowledge extracted from events, akin to how familiarity is conceptualized in signal detection theory. In skill acquisition, it could represent the degree of acquired domain expertise. We simply assume that these latent constructs have normally distributed values across participants. Although the error terms in the model affect performance and awareness, they do so in a way that is completely independent of the individual's strength of latent knowledge.

It should be clear from these model formulas that the latent entities S_P and S_A are independent. Moreover, since the true mean of S_A is zero at the group level, this model embodies complete independence between performance and awareness as well as a complete absence of awareness and hence licenses quite a strong inference that whatever is controlling performance (i.e., S_P) is unrelated to awareness. Therefore, the data depicted in Figure 1A would form the basis of a compelling argument for the existence of an unconscious process influencing performance. Of course this is not a deductive inference, and it would be entirely open to an opponent of unconscious processes to formulate a model that generates a close fit to the data without the need to distinguish S_A and S_P. Although this would be challenging, it would not be impossible, especially if different functions are incorporated to map S onto the two observable measures (Kellen et al., 2021; Stephens et al., 2019). So what is the problem?

Even casual acquaintance with the literature on unconscious or implicit processes makes it clear that patterns like that shown in Figure 1A are as elusive as unicorns. Indeed, the frequency with which advocates of unconscious mental processes employ the alternative methods we describe later suggests that they regard this pattern as hard to demonstrate. We will not undertake a detailed literature review here, but it is safe to say that there exist vanishingly few examples of studies which meet the relevant requirements, namely that (1) performance P is robustly above chance, (2) it is beyond dispute that awareness A is at chance, and (3) these first two requirements are regularly and convincingly corroborated in replication studies. The requirement that it is beyond dispute that A is at chance could be achieved within a frequentist framework either by a hypothesis test with a small fixed type II error rate showing a nonsignificant result (Neyman & Pearson, 1933) or by equivalence tests showing that A is smaller than some minimal theoretically meaningful level (Lakens et al., 2018), or within a Bayesian framework by strong evidence for the null hypothesis (Sand & Nilsson, 2016).

Some patterns that appear to resemble Figure 1A disappear in replications where it turns out that although awareness is truly at chance, performance is also at chance (e.g., Heycke et al., 2017; Röseler et al., 2021; Stein et al., 2020). Many behavior priming effects – subtle influences on behavior assumed to lie outside awareness – have disappeared in replication studies (e.g., Doyen et al., 2012; Klein et al., 2014; O'Donnell et al., 2018; Shanks et al., 2015). Meta-analyses also suggest that some implicit effects are inflated by publication bias, with low-performance levels being under-represented (e.g., Lodder et al., 2019; Mertens & Engelhard, 2020; Nobre et al., 2020; Shanks et al., 2015; Vadillo, Hardwicke, & Shanks, 2016).

Putting aside the replicability of performance effects in individual studies, another problem inherent in data patterns like that shown in Figure 1A is that they only license an interpretation in terms of unconscious processes if one is willing to assert that the null hypothesis of the awareness test (awareness = 0) is true. If statistical power is low, because a small sample of participants have responded to a small number of awareness trials, then failing to reject a false null hypothesis is a highly likely outcome (Sand & Nilsson, 2016). This is a methodological rather than theoretical concern that can be ameliorated by testing large samples or combining data in a meta-analysis.

When such measures are taken to boost power, it is common to find that awareness is robustly better than chance. Well-powered replications and meta-analyses reveal that awareness is actually above chance in many experiments supposedly studying unconscious processes, conforming to a pattern more like that shown in Figure 1B. For example, studies following Bechara et al.'s (1997) research on the Iowa Gambling Task found that participants often have appreciably high levels of awareness undetected by the methods employed in the original experiments (Konstantinidis & Shanks, 2014; Maia & McClelland, 2004), while a replication of Dehaene et al.'s (1998) subliminal semantic priming effects found a similar result (Meyen et al., 2021). In meta-analyses, we have found that awareness is reliably above chance when data are aggregated from contextual cuing (Vadillo, Konstantinidis, & Shanks, 2016) and probability cuing (Vadillo et al., 2020; see also Jiang et al., 2018) experiments almost all of which individually claim the opposite, and Nobre et al. (2020) found a similar pattern in studies on implicit effects in inattentional blindness.

Occasionally, researchers have endeavored to engineer a pattern like that in Figure 1A by deliberately manipulating key aspects of the task to reduce awareness to chance (e.g., Stein et al., 2021; Wildegger et al., 2015). For example, subliminal perception studies (e.g., Beauny et al., 2020; Wildegger et al., 2015) sometimes use a “staircase” procedure in which the presentation duration of a stimulus is increased or decreased across trials until identification accuracy is at chance in every participant. It is unfortunate that this procedure is not employed more often. Even when it is adopted, the results are often inconclusive (we discuss Wildegger et al.'s results in more detail below).

As might be expected from the combination of both above-chance performance and awareness, the data in Figure 1B were generated from a model (Model 1a in Table 1) with a single latent process simultaneously determining both outcome measures1. Specifically, performance is based on P = a · S + e_P and awareness on A = b · S + e_A, where S no longer has a subscript because it is common to both outcomes. As before, S varies across participants, reflecting the fact that the level of latent task knowledge is systematically greater in some than in others. The noise parameters e_P and e_A are again independent of S and have been chosen to reflect the fact that awareness scores are often found to be much more dispersed than the idealized pattern in Figure 1A.

It should be clear from these illustrations that falsifying a model that only includes a single latent process is not a trivial task. The first two recommendations in Box 1 would alleviate some of the concerns expressed in this section.

Table 2

	Recommendation
1	We recommend the wider use of frequentist-statistical approaches with fixed error rates and Bayesian statistical approaches to avoid slippage from failure to reject the null hypothesis (awareness = 0) to asserting that the null is true. Bayes factors permit evaluation of the strength of evidence in support of the null.
2	Sample sizes should be planned a priori to allow for either high-powered tests (in the frequentist approach) or clear support for one of the competing hypotheses (in the Bayesian approach) on (a) the average measure of awareness and (b) the performance–awareness correlation.
3	Researchers should report the reliabilities of their dependent measures, and especially of their awareness measure, and attempt to increase these reliabilities when they are inadequate. When these measures are based on multiple trials, a split-half correlation can be calculated without any additional data needing to be collected. When awareness is based on a single response, a retest should be included in the experimental design.
4	We advise extreme caution regarding the use of the post hoc selection and regression intercept methods, which rest on assumptions that are often unwarranted (e.g., negligible influence of measurement error).
5	Eliciting trial-by-trial awareness reports (e.g., “visible”/“invisible”) and analyzing a performance measure just across the invisible trials – a form of post hoc selection – should only be undertaken when the awareness test can be demonstrated to have high reliability.

View as image HTML

Post Hoc Selection

As a reaction to the difficulty of obtaining a precise group-level null awareness effect, researchers often adopt what seems at first sight to be an entirely reasonable analytic approach, namely to eliminate from the analysis all participants who score above some more or less strict cutoff on the awareness assessment. Then, the main analysis computes the average score on the primary performance measure for the remaining participants, who, seemingly, must lack awareness. If this average performance score exceeds chance, then it appears that unconscious processes must be playing a role. Figure 1C shows such a pattern where only participants scoring at or below chance on the awareness measure are retained.

For example, Wildegger et al. (2015, Exp. 1b) measured the effect of a briefly presented prime (an oriented line) on speed of reporting the orientation of a target line. The prime's orientation was either congruent or incongruent with that of the target. In a separate prime awareness test, participants made forced-choice prime identification responses and their overall accuracy was determined. Wildegger et al. (2015) found that 5 out of 20 participants performed better than chance in this awareness test and hence eliminated them from the primary analysis, which nevertheless obtained a 6-msec priming effect. Wildegger et al. (2015) therefore interpreted their findings as evidence of an unconscious influence of the prime.

Shanks (2017) showed that, counterintuitively, this analytic method is flawed and can lead to wholly false conclusions about unconscious mental processes. Box 2 describes a statistical problem to illustrate the underlying issue. Readers are invited to reflect on it before continuing.

Table 3

You measure the toxicity of all 20 additives, but your equipment cannot measure toxicity perfectly. Your measurements are inevitably noisy and include some random measurement error. This error is independent of true toxicity.

Having selected the five least toxic ones, you average their measured toxicities and report this figure to the supermarket.

Is there any problem with this procedure?

View as image HTML

Is there any problem with the apparently intuitive procedure described in Box 2? Indeed there is. To illustrate this concretely, suppose the true toxicities are sampled from a normal distribution with mean of 50 units and a SD of 10 units and that your equipment adds measurement error with a mean of zero and SD = 5. Your estimate of the toxicity of the five least toxic additives will be about 36.4, but the true toxicity of these additives is in fact appreciably higher, M = 39.1.²

This mismeasurement is a consequence of regression to the mean. Although error in the measurement of toxicity is statistically independent of true toxicity, these errors are not independent of the true toxicity of the least toxic additives. This is simply because random underestimation of an additive's toxicity increases the chances that that additive is selected for inclusion and, conversely, random overestimation decreases an additive's chances of inclusion. If one were to order the additives from highest to lowest by measured toxicity, the errors would tend to be positive in the most toxic additives and negative in the least toxic ones. Crucially, if you remeasure the toxicity of the five chosen additives, their toxicities will increase. On this second, independent measurement, errors will now be genuinely independent of the true toxicities (as well as being independent of the errors in the first measurement), and so the measured toxicities will regress to the mean.

The relevance to studies of unconscious mental processes is straightforward. It could be the case that the measured awareness (and performance) in the hypothetical participants in Figure 1C is precise and accurate. Maybe the awareness test is exceptionally reliable and only a very small amount of error (e_A) is present. But it could equally be the case that error is nontrivial. In this situation, by selecting participants who fall below some awareness cutoff, the researcher is inevitably (on average) selecting ones for whom true awareness is greater than measured awareness, often by a large amount. Suppose that a participant's true latent knowledge is greater than chance, even by only a small amount (that is, S > 0). If the error term e_A for this individual is negative, then measured awareness ( = b · S + e_A) may be less than or equal to chance (0) and the participant will be included in the analysis. Because participants are selected post hoc in this method according to their measured awareness, the researcher has no control over the magnitude of error that is inherent to the measured awareness of the participants retained in the analysis. Thus, we cannot straightforwardly know whether these participants are truly unaware (e_A is negligible) or truly aware (e_A is negative).

These two alternatives make divergent predictions regarding an easily tested outcome, namely test–retest reliability. If e_A is genuinely small and measured awareness in the selected participants is an accurate reflection of true awareness, then the results should look very similar on a second, independent test of awareness (high reliability). However, if e_A is negative, we will expect regression to the mean to occur on an independent awareness test. On such a test, the values of e_A will have a mean of zero, causing measured awareness ( = b · S + e_A) to increase from its negative value and regress toward the mean level of awareness of the entire unselected group. The net effect will be low test–retest reliability. In case any readers believe that awareness tests are usually sufficiently reliable to make any such regression effects negligible, we will later describe a published study in which the reliability of the awareness test was only slightly above zero.

The consequences of regression to the mean are well-known (Campbell & Kenny, 1999; Morton & Torgerson, 2005). The following analogy may be helpful. Imagine that a school administers a test to all its students, perhaps a test of executive skills, and on the basis of their normed test scores, a group of low-scoring students is assessed for some remedial intervention. After the intervention, their scores are found to have improved appreciably. Readers will immediately recognize that attributing the improvement to the intervention is unwarranted because it may alternatively be the result of regression to the mean. Some students, purely by chance, will have underperformed on the initial test. Perhaps they were tired or distracted. Thus, their scores, which combine a true score plus error, will regress to the mean on a second test in which the random errors are drawn independently from the error distribution. In the context of research on unconscious processes, the mere possibility of such a state of affairs is sufficient to entirely undermine many studies that have based their conclusions of post hoc selection without assessing the reliability of their awareness tests (Shanks, 2017).

It might be assumed that any such regression effect will be small, perhaps too small to matter. We will comment later on the actual level of reliability that is seen in typical tests of awareness, but models can once again be instructive. In fact, as perceptive readers will have noticed, the points in Figure 1C are identical to those falling to the left of A = 0 in Figure 1B – they were generated from a model with only a single latent process (Model 1A). So we can ask, how much regression would we see in these datapoints? Would it be enough to change our inference that awareness is not greater than zero in these simulated participants? The purple triangle in Figure 1C shows the true mean awareness (and observed mean performance) in the included datapoints. Not only is it shifted to the right, but it is far beyond A = 0! In other words, if we were able to strip away the error term that contributes to each datapoint in the figure to leave just the term b · S, we would find it to be much less negative – indeed, most of the resulting points would be positive. In fact, the mean true awareness is virtually identical to the whole group's observed mean (Figure 1B). Of course this is an extreme case arising from parameters chosen to make a point. With relatively less error in the awareness measure, the degree of regression would be smaller, though always yielding true mean awareness greater than 0 (because mean S is greater than zero).

This simple simulation yields a profoundly counterintuitive conclusion. A subset of (simulated) participants can be selected post hoc on the basis that their awareness is not above chance, and the mean performance of these participants will be appreciably above chance, indeed close to the overall group mean, apparently revealing an unconscious mental process. Yet the apparent unawareness in these participants is an illusion, caused directly by the post hoc selection method. These participants only appear to be unaware. In truth, the latent process (S) that determines awareness scores is robustly positive. But it has by chance been combined with strongly negative error scores and then selected by the researcher. If we were to independently remeasure awareness in these participants, regression to the mean would play its inevitable role and reveal their true awareness to be robustly above chance.

To this point, our conceptualization of bivariate data patterns relevant to unconscious processing has assumed that the measure of awareness is continuous, and this is also reflected in the models in Table 1 where A is a continuous measure. Researchers (e.g., Beauny et al., 2020; Biderman et al., 2020) frequently however collect discrete (often binary) awareness reports, for example, asking participants whether a masked stimulus was “seen” or “unseen,” or employ scales with more than two categories such as the 4-point Perceptual Awareness Scale (Ramsøy & Overgaard, 2004). These represent special cases of bivariate data in which measures on the awareness scale are discrete and the analysis averages across trials rather than participants. Importantly, as Shanks (2017) and Schmidt (2015) have pointed out, treating invisibility reports as true indices of unconscious processing will often lead to incorrect inferences. Eriksen's (1960) famous critical appraisal of unconscious discrimination published over 60 years ago has been described as a

Schmidt's (2015) argument is that interpreting invisibility reports at face value pays no heed to the individual's response bias, and decades of development of signal detection theory have demonstrated that discrimination and bias are distinct entities. Responding “invisible” may not mean that the internal representation of the stimulus is truly unconscious, it may instead mean that its strength is below the threshold the individual has set for reporting “visible.” Imagine a hypothetical situation in which a participant is tested twice under otherwise identical conditions, except that a standard manipulation of response bias (such as instructions or payoffs) induces a conservative bias in the first test (very few stimuli are reported as visible) and a liberal bias in the second test (many stimuli reported as visible). Consequently, some events will be reported as invisible in the first test but visible in the second. Would it not be highly implausible to infer that the manipulation affects the participant's actual subjective experience? Much more likely is that the underlying state of experience is the same and that it is the reporting criterion that has changed. This response bias alternative is familiar as an explanation of “blindsight” (Phillips, 2021).

Shanks' (2017) point is distinct from but complementary to this. Reported awareness depends on both the underlying true representation (S) and random error. Stimuli that evoke “invisible” responses will sometimes do so because S is close to zero, but on other occasions will do so because a positive value of S (that is, an internal representation that has nonzero strength) is combined with a negative value of the error term. By selecting trials post hoc in which the participant says “invisible,” the experimenter relinquishes control over the items included in the analysis, in which this bias may therefore intrude. If it were possible to obtain a second independent awareness report for these items, regression to the mean in the error term would pull these reports upward and turn many of them into “visible” reports, as in the continuous case illustrated in Figure 1C. The onus must be on the researcher to eliminate this possibility, for example, by showing that her visibility scale has very high reliability.

As an illustration of how this issue undermines research findings, consider a recent neuroimaging study by Sheikh et al. (2019) whose research question was whether the brain extracts semantic information from unconscious (masked) words. On each trial, a word was briefly flashed and participants made two responses: First, they judged whether it was animate or inanimate, and second, they reported their conscious experience of the word on a 3-point scale from “I didn't see anything” (i.e., invisible) to “I think I saw the word clearly or almost clearly.” Sheikh et al.'s (2019) main finding was that activity in several brain regions distinguished animate from inanimate words even when they were judged invisible. This inference rests, however, on the unlikely assumption that the visibility reports are perfectly reliable and free of random error (it also pays no heed to the flexibility with which the participant sets her response criterion; Schmidt, 2015). Any tendency for error to contaminate these reports must inevitably mean that among the trials classified as invisible are some (perhaps many) where the true latent state driving these reports is greater than zero: They are, in effect, misclassified.³Shanks (2017) provides simulations of this type of situation where the awareness report is discrete. There is simply no getting away from the ineluctable effects of regression to the mean.

Sheikh et al. (2019) provided a further reason to believe that awareness was truly absent on “invisible” trials: Objective awareness, as measured by the accuracy of animate/inanimate judgments, was close to and not significantly greater than chance (50%) on these trials. But resonating with our earlier discussion, the upper limit of the 95% CI of this estimate was ∼70%. Hence, the data look much more like the pattern shown in Figure 1B than in Figure 1A. This study therefore illustrates not only the unwarranted use of trial-based post hoc selection but also the dangers of slipping from failure to reject the null to acceptance of the null.

Loken and Gelman (2017) pointed out that studies (particularly ones with small samples) affected by measurement error may erroneously yield statistically significant effects. The magnitude of these effects will inevitably shrink in replications in which these errors regress. Applied to find evidence of unconscious processes, the post hoc selection method represents the flip side of this coin, namely where measurement error leads to erroneous null results (the absence of awareness). It is a method that cannot fulfill the hope of researchers who use it in the expectation that it can reveal true unconscious processes.

Correlation and Regression Slope

If the analytic strategies discussed above fall short of providing the evidence needed to infer unconscious processing, then perhaps stronger inferences can be drawn from the pattern of correlation observed between performance and awareness? Looking at Model 1, it should be clear from the equations in Table 1 that they can be rearranged to yield the simple linear equation

It is clear that many researchers implicitly accept the above reasoning because correlations have been regularly (and perhaps increasingly) used to justify claims about unconscious processes. Malejka et al. (2021, Table 1) provide examples of the use of correlation analyses to infer unconscious processes in response inhibition, contextual cuing, evaluative and classical conditioning, artificial grammar learning, language and category learning, and other domains. For example, Salvador et al. (2018) recently reported a pair of experiments on memory suppression. The experimental details are not important here, except to note that they measured awareness in a discrimination test for briefly flashed shapes previously presented as cues in a memory suppression task. Although discrimination was significantly better than chance (i.e., conforming to the pattern shown in Figure 1B), awareness was uncorrelated (r = −.09, combined across the two experiments) with performance (a memory suppression dependent measure).

Like the other approaches we have discussed, this method raises a number of issues. To begin with, just because a correlation is low and not significantly different from zero does not entail that the true relationship is zero – a conclusion that cannot easily be drawn if one is employing Null Hypothesis Significance Testing (NHST). In Salvador et al.'s (2018) data, the very low power of their experiments meant that the 95% CI on their correlation estimate was (−.31, .15). This wide interval is compatible with a nontrivial positive correlation. This problem could be addressed by using substantially larger samples, allowing the CI to be more narrowly estimated, but another problem would remain. It goes without saying that the measures Salvador et al. (2018) obtained, like all measures in behavioral research, were subject to measurement error. In the presence of such error, a correlation coefficient will invariably underestimate the true relationship. This regression attenuation or regression dilution phenomenon (Spearman, 1910) has been much studied in areas such as biomedical statistics (see Hutcheon et al., 2010). In economics, the underestimation of relationships in regression models when the predictors contain error variance is so firmly established that it has been termed an “iron law” (Hausman, 2001).

To estimate the degree of this attenuation in Salvador et al.'s (2018) data, Malejka et al. (2021) computed the reliability of their performance and awareness measures via odd–even split-half correlations. Remarkably, the resulting value for the awareness test was a mere .15. It was much higher (.65) for the performance measure. [As an aside, while both reliabilities matter in correlation analyses, the reliability of the predictor variable – awareness in this case – is much more important when performance is being regressed onto awareness, for reasons explained by Hutcheon et al. (2010)]. The reliability of Salvador et al.'s awareness measure is far below the minimum level expected of a psychometrically sound measure. Collapsing data across both experiments, the Pearson correlation coefficient between performance and awareness when disattenuated for measurement error was −.28 with an extremely wide 95% CI of [–1.00, 0.47]. Clearly, based on the measures they used, Salvador et al. were only able to rule out correlations larger than about .5. Stated differently, their finding of a nonsignificant performance–awareness correlation only challenges single-process models that predict fairly high correlations.

One of the key points of the current article (see Box 1) is that future research in this field must report the reliabilities of the measures obtained (particularly measures of awareness). Readers might be skeptical of this need, perhaps thinking that Salvador et al.'s (2018) reliabilities are atypical. In fact, there is an accumulating body of evidence that they are far from unusual, and indeed, we recommend a default assumption that standard measures used in this research field have medium-to-low reliabilities (LeBel & Paunonen, 2011). Studies that have reported reliabilities have usually revealed disappointingly low values (Geyer et al., 2020; Kalra et al., 2019; Vadillo et al., 2020; West et al., 2018).

Comprehensive data illustrating this point come from very large samples analyzed by Vadillo et al. (2021). These data are from contextual cuing experiments using a popular visual search task (see Goujon et al., 2015). On each of many visual search trials, participants locate a target (often an inverted T) among distractors (Ls) as rapidly as possible. Some of the distractor patterns repeat during the experiment, always with the target in the same location, and RTs confirm that participants learn about these repeating displays as their responses become progressively faster, relative to novel patterns. After many blocks of search trials, participants' awareness of the repeating patterns is probed. One common assessment of awareness comprises a recognition test in which they are explicitly instructed to judge whether displays are repeated or novel. In another type of test, displays are shown in which the target has been replaced by another distractor, and participants are instructed to point to the location of the hidden target. Vadillo et al. collected data from 505 participants and also analyzed data from three experiments by Colagiuri and Livesey (2016) with sample sizes of 63, 84, and 766. The methods, which are standard across dozens of contextual cuing studies, were quite similar across these experiments.

Vadillo et al. (2021) found that the split-half reliability of the performance measure in these studies was very consistently slightly less than .5 (with the Spearman–Brown correction). The reliability of the awareness measures was around .5 for the experiments conducted by Vadillo et al., but closer to .3 for Colagiuri and Livesey's experiments. While these figures are not quite as starkly low as those in Salvador et al.'s experiments noted above, there are good reasons to suspect that they overestimate the reliabilities of these measures in similar studies, which usually include much briefer (and therefore noisier) tests of awareness. In any case, these figures confirm the considerable magnitude of measurement error in these scales as well as the impossibility of obtaining high performance–awareness correlations: If a measure does not correlate well with itself – which is what reliability measures – it is unlikely to correlate any better with another measure.⁴

With this very large sample, a meta-analysis yielded a mean performance–awareness correlation of .039, [−.014, .092]. One might look at this as providing quite strong support for independence between the measures and as suggesting that the true correlation must be close to zero. But a further meta-analysis that took account of the reliabilities yielded a substantially larger disattenuated mean correlation of r = .104 as well as a much wider CI [−.016, .225] – an example of the “iron law” in action. Hence, even with data from over 1,400 participants and a nonsignificant correlation, the hypothesis that the correlation is as large as .225 cannot be rejected. To put this in perspective, it is larger than the average effect size obtained across all psychological research! In an analysis of 200 meta-analyses, Stanley et al. (2018) estimated the mean effect size at r = .19.

In the face of these problems, what can researchers do? A powerful approach is to switch from NHST and instead conduct a Bayesian analysis that permits us to quantify the degree of support for the null compared to the alternative (r > 0) hypothesis. Malejka et al. (2021) described three such Bayesian models. The reader is referred to that article for details. A Bayesian approach requires a specification of the prior distribution of correlations, which could, for example, be a uniform distribution (all correlations between 0 and 1 are equally plausible a priori), or, more plausibly, one in which small positive correlations are more likely. For present purposes, Malejka et al.'s key finding in their reanalysis of Salvador et al.'s data was that the Bayes factor (BF) in favor of the null hypothesis was only in the region of 1.8 when a prior was used in which small positive correlations are assumed to be more likely. Thus, on a plausible assumption that the true correlation is positive but small, Salvador et al.'s data only marginally favored the null over the alternative hypothesis.

More constructively, Malejka et al. used these Bayesian models as tools for deriving sample size predictions which researchers can use to guide their sample size planning. Given pilot work to enable the reliability of the experimental measures to be estimated, together with a specification of the minimum true correlation one wishes to be able to rule out, Malejka et al.'s simulations specify the sample sizes needed to achieve a given level of evidence (e.g., BF > 10 in favor of the null). For instance, if the true correlation is 0 and reliability is above .65 for both measures, then a commonly achievable sample size of 300 is adequate to find median BFs > 10 in favor of the null (in comparison, there were 44 and 30 participants in Salvador et al.'s experiments).

Regression Intercept

The above discussion highlights the serious interpretive consequences of measurement error in correlational analyses. Such an error leads to systematic underestimation of the true correlation, and we have seen evidence to suggest that in some popular tasks for studying unconscious mental processes, this error is sufficiently large (reliability is low) for the effect on measured correlation to be nontrivial. A second and quite common type of analytic strategy that researchers sometimes apply to performance/awareness correlational data proves to be equally problematic, albeit for a different reason. This is the practice of focusing attention on the regression intercept. Why is the intercept an interesting measure?

Since the two error terms in Equation 1 both have means of 0, it seems natural to infer that the predicted value of P when A = 0 is also zero, which seems intuitive given that this model only includes a single latent variable. Hence, it is very tempting to assume that any single-process model must predict an intercept of zero: When performance is regressed onto awareness, the former must be zero when the latter is.

This logic is widely employed. For instance, in a recent study by Berkovitch and Dehaene (2019), participants judged whether a series of target words were nouns or verbs, each of which was preceded by a masked subliminal prime (also a noun or a verb). The results showed a typical compatibility effect, with faster responses to target words preceded by primes of the same grammatical category. Prime awareness was based on participants' ability to discriminate the primes as nouns or verbs in a visibility task conducted at the end of the experiment. Berkovitch and Dehaene noted that “the intercept of this regression was significant […], indicating that priming remained significant even at null d'” (p. 31). Greenwald et al. (1995) were the first to propose this analytic method, claiming that “the finding of a significant regression intercept effect (i.e., a nonzero indirect effect associated with a zero direct effect) indicates an unconscious contribution to the indirect measure” (p. 26).

But once again this is a fallacious conclusion (Miller, 2000; Sand & Nilsson, 2016). Regression dilution not only flattens the slope of a regression line, but it also systematically biases the intercept upward. This can be seen in Figure 1D. Model 1B, which differs from Model 1A simply in the parameter values, was used to generate the data in this figure. Thus, once again the performance and awareness measures are linked to a common latent construct (S). From Equation 1, we can calculate that the true slope at the latent level is c = a/b = 100/0.3 = 333, shown by the black line in Figure 1D, whereas the empirical regression line (in pink in Figure 1D) has a slope far shallower than this (an increase in awareness of 1 unit is associated with an increase in performance of about 30 units). This is the dilution effect discussed above. But what we can also see is that the observed intercept is not zero, it is much higher than this. Hence, measurement error leads to systematic bias in the estimation of the regression intercept. Error-free measurement of the predictor variable, an assumption of standard regression, is unlikely to be achievable in any behavioral research. Interested readers can readily run this simulation from the openly available R code to see that it is error in the awareness measure (e_A) that is crucial to this effect. When this error value is set to 0, the true intercept (0) is estimated. This is not the case when error (e_P) in the outcome variable (performance) is set to zero (Hutcheon et al., 2010).

Why does this effect occur? For exactly the same reason as for the post hoc selection case, namely regression to the mean. From the model, we can see that because S has a M and SD of 1.0, for most participants, its value is greater than zero. This implies that when a given participant's measured awareness is at or below 0, it will often be because, by chance, e_A is negative. When we calculate this participant's performance, it will be based on the same value of S but combined with e_P which is statistically independent of e_A. The former has an expected value of 0, resulting in a performance score that regresses toward the group mean. Measured performance is closer to the group mean performance level than measured awareness is to group mean awareness.

To amplify, if we take all the simulated participants in Figure 1D whose awareness is close to zero (let us say between −0.1 and 0.1 on the awareness scale), we can ask what the true level of awareness is in these participants. The answer is that it is very close to 0.3, the mean awareness of the group as a whole (because b = 0.3 in this simulation). This is very similar to the situation in Figure 1C, discussed previously. For these participants, the awareness error term e_A is on average about −0.3, thus canceling out the positive true awareness.⁵ The positive regression intercept is an inevitable consequence: Because S is positive rather than zero in these participants, then when combined with independent error to determine their performance level, the latter is robustly positive. Figure 1D depicts a pattern in which average awareness seems barely above chance, performance is clearly above chance overall, the correlation between performance and awareness is very low, and the level of performance when awareness is at chance is also above chance. Yet all these features emerge from a model in which a single latent process determines both performance and awareness.

We would see exactly the same but in reverse if instead of regressing P onto A we regressed A onto P. We would now find that participants with a mean performance level of zero have above-chance awareness! Those scoring at zero on the performance measure will usually only do so because e_P is large and negative. By the same argument as above but turned around, their level of awareness will be close to the group mean awareness because e_A, being independent of e_P, will be close to zero. In both cases, the effect is not indicative of some separable latent process, but it is a statistical artifact caused by regression to the mean.

To be fair, Greenwald et al. were sufficiently aware of this concern that they sought to address it directly. They proposed a “bias correction” method intended to provide an accurate estimate of the true regression intercept, taking the effects of measurement error into account (Klauer et al., 1998). In one recent application, Greenwald and De Houwer (2017) estimated the relationship between awareness and conditioning. In several experiments, they observed intercepts greater than zero (as well as conditioning–awareness correlations not significantly greater than zero) and reported that they remained positive even when corrected.

Although it is rarely used, the development of this correction method is an important contribution. However, it is unclear whether it can solve the underlying problem because it is relatively easy to construct situations in which it undercorrects (Miller, 2000). We (Malejka et al., 2019) generated data from a single-system model similar to Model 1 and then corrected the intercept estimates using the Klauer et al. (1998) method. Despite the fact that the true intercept was known to be zero, the method falsely concluded that it was positive in a high proportion of cases, depending on the precise parameters of the simulation. For instance, the undercorrection was particularly acute when the underlying data included only a small proportion of truly unaware participants. Until a more robust correction method is available (see Hutcheon et al., 2010, for references to the statistical literature on this problem), we urge researchers to be extremely cautious in using the regression intercept method to draw inferences about unconscious processes (Box 1, Recommendation 4).

Single Dissociations

Many reported performance/awareness dissociation patterns are essentially claims about correlation, and hence, the major observations in the previous section apply to them. For example, several studies have included explicit instructions to orient participants toward the key stimulus feature or regularity, confirmed that these instructions boost awareness, and then shown that the performance measure is unaffected by these instructions (e.g., Jiménez et al., 1996; Westerberg et al., 2011). In essence, this is the claim that there is no correlation (across groups) between performance and awareness under the instructional manipulation. The famous claim that hippocampal brain damage affects explicit memory while leaving implicit memory unaffected (e.g., Hamann & Squire, 1997) is again a claim about a null correlation.

Because single-process models, as shown above, can predict correlations that are so low they are hard to distinguish from zero, these patterns of single dissociation are generally not diagnostic of distinct latent processes for performance and awareness (e.g., see Berry et al., 2008; Jamieson et al., 2010; Nosofsky & Zaki, 1998; Shanks & Berry, 2012). It would only be in the case that the measures are highly reliable (Box 1, Recommendation 3) that such patterns would pose a serious challenge to models incorporating a single latent process.

Conclusions

Inferring unconscious mental processes from patterns of bivariate performance/awareness data is fraught with underappreciated problems. In this article, we have based our discussion of these problems around some extremely simple models (Table 1), but it is our strong belief that reflecting on the predictions of such models can illuminate the valid inferences that can be drawn from bivariate data patterns. Just to give one example, our informal probing (via problems like that in Box 2) reveals that many researchers believe that random error is equally distributed across the range of a measured variable. This is false – at the positive extreme of the measured variable, errors will on average be positive and at the negative extreme they will be negative, and this is why regression dilution by measurement error occurs. The implications of this simple statistical fact for the interpretation of bivariate performance/awareness data are profound, as explained in our discussion of post hoc selection and the interpretation of correlational data. Other strong but false intuitions (for instance, that a model with a single latent process necessarily predicts a zero intercept) can equally lead to incorrect conclusions about unconscious mental processes.

These considerations lead to the Recommendations listed in Box 1. Within the domain of bivariate data, we advise less use of some common practices (e.g., relying on regression intercepts) and greater use of others (e.g., calculating CIs or BFs). What else can researchers do? Studies that include experimental manipulations of awareness and performance are important. Any demonstration, for example, that an independent variable increases performance at the same time as reducing awareness (i.e., a double dissociation) would be extremely hard to explain with reference to only a single latent construct. Further efforts to induce strong dissociations of this sort are likely to be illuminating. For example, there have been demonstrations of manipulations that increase priming while decreasing prime visibility (e.g., Biafora & Schmidt, 2020) and of other manipulations that affect conditioned responses and conscious expectancies in opposite direction (Perruchet, 2015).

However, we also urge researchers to extend beyond this domain of methods and analytic approaches. Ultimately, bivariate patterns are not particularly constraining in relation to possible latent factors. In the Introduction, we pointed out that decades of developments in psychometrics have yielded a considerably broader set of methods and tools for inferring mental processes. Research on the IAT (Kurdi et al., 2019; Nosek & Smyth, 2007) provides on exemplar of this very different strategy. Wider adoption of these approaches – which would require employing multiple awareness and performance tests and comparison of single-process versus multiple-process models – might considerably strengthen the inferences we can draw about unconscious processes.