The Effect of a Verbal Concurrent Task on Visual Precision in Working Memory

Dual task paradigms, in which participants complete two tasks individually and then concurrently, have been used since the inception of working memory research (Baddeley & Hitch, 1974). Such paradigms have likely endured due to the simplicity of their logic: If two tasks draw on the same parts of the cognitive system, then people should be worse at doing them together than carrying out the single tasks alone. This logic has been pivotal in establishing multiple component accounts of working memory (Baddeley, 2012). Such accounts vary but share the view that storage in working memory (WM) is served by distinct stores for phonological versus visuospatial information, together with an executive resource that co-ordinates the domain-specific stores. In addition, Baddeley (2000) proposed an amodal store, the episodic buffer, to store integrated items. Others have suggested that binding is served by communication between domain-specific resources without the need for the concept of an executive resource or an episodic buffer (Logie, 2016) and that there may be multiple “executive” resources, each of which supports a specific function, including task switching, updating, and inhibition (e.g., Miyake et al., 2000), as well as communication between domain-specific stores, and implementation of mnemonic strategies.

Classically, the observation that dual task interference is limited when each task involves different modalities has been taken as evidence for modality-specific storage capacities. For example, Cocchini, Logie, Della Sala, MacPherson, and Baddeley (2002) asked participants to retain a sequence of digits with sequence length set at the span of each participant. During a 15-second retention interval, participants either saw a blank screen or were shown a series of random square matrix patterns in which half the squares were black and half white. Following each pattern, they were shown a blank matrix and were asked to recall which squares had previously been shown in black. After the blank or filled retention interval, participants were asked to recall the digit sequence. The matrix recall task was also performed without the verbal memory preload. Recall of the digits was unaffected by the matrix recall task during the retention interval, and recall of the matrix patterns was unaffected by having a digit memory preload. In contrast, when a single matrix pattern was used as a memory preload, and the 15-second retention interval was filled by a perceptuo-motor tracking task, there was significant disruption of recall of the matrix pattern, relative to the condition with a blank retention interval. Digit recall was unaffected when the retention interval was filled with the tracking task.

This picture has been complicated recently by the observation of asymmetric dual task costs between verbal and visual domains, where the effect of a verbal load on visual working memory was found to be larger than the effect of a visual load on verbal memory (e.g., Morey, Morey, van der Reijden, & Holweg, 2013; see Morey, 2018 for a review). This contrasts with previous studies that have shown the domain-specific verbal and visual dual task costs to be symmetric (Farmer, Berman, & Fletcher, 1986; Logie, 1986; Logie, Zucco, & Baddeley, 1990). The manipulation of the cognitive load of a secondary task (e.g., Doherty & Logie, 2016; Logie, Cocchini, Della Sala, & Baddeley, 2004) has also been important for multiple component models. The load can be set so that it is below, at, or above each participant’s capacity to store information in service of a single task. Here, the concern is not the “code” in which information is stored but the role of executive resources in a task. Any detrimental effects on a concurrent task of increasing cognitive load above the capacity of a passive store might imply a role for executive resources in that task, for example, to implement mnemonic strategies.

In contrast to the multiple component accounts of working memory, the embedded processes account does not make use of modality-specific stores (Cowan, 2005). Rather, as items are presented their features are automatically activated in long-term memory (LTM). Embedded within this activated memory, a small number of integrated items can be represented in a domain-general focus of attention. Executive functions are not part of the model as such but play a role in influencing what enters the focus of attention. In contrast to the Cocchini et al. (2002) study, work informed by the embedded processes account has demonstrated substantial dual task interference between modalities (Morey & Cowan, 2004, 2005). The memoranda in such cases are thought to share the limited focus of attention, resulting in the observed drop in performance. Proponents of embedded processes do not deny the possibility of passive storage contributing to performance in working memory (e.g., Morey & Cowan, 2005), drawing on item features that are currently activated in LTM but are outside the focus of attention. The question is whether all storage in WM can be accounted for by activated LTM, or if there are, in addition, domain-specific stores that provide the primary hosts for temporary memory within a multiple component working memory (Logie, 2016; see also Norris, 2017).

Here, we aimed to contrast the predictions of Logie’s (2011, 2016) multiple component account and a general attentional resource account using a dual task paradigm, within the context of a continuous response task. Following Gorgoraptis, Catalao, Bays, and Husain (2011, Experiment 2), our visual task required participants to view an array of colored bars, each shown in a different orientation, and subsequently to recall the angle of orientation of a target bar by circular, analog adjustment. By requiring analog recall of the angle of orientation of a target stimulus, we obtained a fine-grained measure of the quality of representations in WM (Zokaei, Burnett Heyes, Gorgoraptis, Budhdeo, & Husain, 2015). Such a measure may be more sensitive to potential dual task interference than measures of item recall that are widely used in dual task studies. In addition, we required participants to judge which of two colors was present in a test array. This allowed us to explore binary recall of a feature (color) alongside analog recall of another feature for the same item (e.g., Pertzov, Heider, Liang, & Husain, 2015). As well as performing this task on its own, participants performed it while maintaining lists of letters in memory. Crucially, here we titrated the verbal load such that the list length was set to each participant’s span (e.g., Cocchini et al., 2002; Logie et al., 2004). Participants completed the letter recall task with and without the interleaved visual memory task. Given that span is assumed to reflect the maximum capacity for immediate memory, a general attentional resource theory (e.g., Cowan, 2005) would assume that storing a letter sequence at span should use most, if not all, of the focus of attention. Therefore, combining the verbal and visual tasks should result in a substantial cost to performance on one or both memory tasks.

The multiple component account of working memory assumes that there are domain-specific immediate memory systems, respectively, for temporary visual storage (the visual cache, Logie, 1995) and for phonologically based temporary verbal storage (the phonological loop, Baddeley, 1992). A strict interpretation of this theoretical framework would predict that there would be no reduction in performance under dual task conditions, if we assume that span for any given task is a pure measure of the capacity of one domain-specific system that supports performance on that task. However, in the multiple component framework (Logie, 2016), it is assumed that when one component reaches its capacity limit, other components of the system are recruited to support performance. Span provides a measure of the capacity of the cognitive system to perform the task, and this may reflect the use of more than one working memory component to support performance. For example, if a set of letters is presented visually for serial ordered recall, then it is well established that participants typically rely on a phonologically based representation of the letters (e.g., Conrad, 1964), assumed to involve the phonological loop. However, several studies have demonstrated that participants may also retain a representation of the visual appearance of the letters (e.g., Logie, Della Sala, Wynn, & Baddeley, 2000; Logie, Saito, Morita, Varma, & Norris, 2016). That is, span for visually presented letters may involve both the phonological loop and at least some of the capacity of the visual cache. If memory for the letters is then combined with another task that involves visual memory or visual processing, not all of the capacity of the visual component would be available. So, there would be a small overall reduction in dual task compared with single task performance. This effect could be mitigated by using auditory presentation of the verbal memoranda, coupled with a visually presented, nonverbal task. This is the approach used in the experiment that we report here. However, a task designed to test visual working memory might gain support from some verbal storage or processing (e.g., names of shapes or colors, or spatial orientations), even if the main load is on a specific visual component of working memory, so there may still be a small dual task cost even when using different input modalities. Previous studies providing evidence for domain-specific components of working memory almost invariably show such a cost, interpreted as above. However, most striking is that the dual task cost, even if statistically robust, tends to be small compared to the residual levels of performance on each task when they are performed concurrently (e.g., Cocchini et al., 2002; Duff & Logie, 2001; Logie et al., 1990, 2004). Very much larger dual task costs are observed when both tasks are chosen to rely primarily on the same component of working memory (e.g., Logie et al., 1990).

Based on these assumptions, the current study was designed to test distinct (preregistered) predictions concerning dual task costs, derived from different theoretical approaches to working memory, with the multi-component model predicting a small effect size associated with dual task costs, and the embedded processes model a medium to large effect size.

Method

This study was preregistered on the Open Science Framework. The preregistration form and a time-stamped archive of the task and analysis scripts can be found at https://osf.io/e5bkg. The data, materials, and all the analysis scripts can be found at https://osf.io/59c4g/.

Materials

All tasks were written in PsychoPy 1.84 (Peirce, 2007). The code for all the tasks is available at https://osf.io/59c4g/.

Letter Span

Each participant’s verbal letter span was determined using a staircase procedure. Letters were presented over headphones at a rate of one per second followed by a 8-second retention interval. Participants then orally recalled the letters in order, with the experimenter typing responses on a second hidden screen. Participants began with a pair of trials with lists of five letters. If 80% or more of the items were correctly recalled (in the correct list position), then the list length was increased by one, otherwise list length was decreased by one. Participants continued this procedure for eight pairs of trials. If a participant achieved over 80% of items correct on their final pair of trials, and it was the highest list length they had reached, then additional trials were presented until less than 80% of items were correctly recalled. A participant’s span was the longest list length at which 80% or more of the items were correctly recalled. Letters were randomly selected from a pool of 18 letters that excluded vowels and “y” (b, c, d, f, g, h, j, k, l, m, n, p, q, r, s, t, v, and x) (Figure 1).

Letter Recall Task

For the letter recall task, participants completed 20 trials in which lists of letters were presented aurally for spoken recall. The list length on all trials was set at the previously measured span for each individual participant. The timings were identical to the letter span task.

Orientation Recall Task

Participants were presented with 3-item arrays of colored bars measuring 2 × 0.3° of visual angle at different orientations (see Figure 2). The colors of the three items were randomly selected from a set of eight easily distinguishable colors (red, orange, yellow, green, cyan, blue, pink, and purple). The orientations were randomly selected such that no two bars in an array were within 10° of one another. The items were presented at a random subset of eight possible locations equidistant on an invisible circle around fixation with a radius of 6° of visual angle. The study array was presented for 1 s followed by a 1-second retention interval. Following the retention interval, two colored bars were presented and participants had to indicate which of the colors was present in the first display. The target was always one of the two colors presented in this recognition phase. After participants made their response, and following a further 0.5 s delay, a bar of the same color that the participant selected was displayed in the center of the screen. Participants were required to recall the orientation of the bar of that color in the 3-item array using the “left” and “right” arrow keys on a keyboard to adjust the orientation. Each participant completed 60 trials.

Dual Task

For the dual task, participants were presented with an at-span list of letters and completed a trial of the orientation recall task in place of the 8-second retention interval. The letters were then recalled orally. Participants completed 60 such trials.

Results

Confirmatory Analysis

Analysis was carried out using the Bayes Factor package (Morey & Rouder, 2015) in R (R Core Team, 2016). For each outcome measure, the posterior estimates and 95% Bayesian credible interval for the mean difference between the single and dual task conditions are reported. The 95% credible interval excluding zero could be interpreted as suggesting a difference between the two conditions (Kruschke, 2014). Secondly, Bayes Factors are used to determine how likely the data are under a model assuming a small dual task interference effect versus a model assuming a medium to large effect. A small effect was defined as the interval from 0 to 0.3 for a standardized effect size. A medium to large effect is defined as the interval from 0.5 to infinity (e.g., R. D. Morey, 2014). Figure 3 shows how the prior density was distributed over each of these effect size intervals in the analysis. It is clear from Figure 3 that while the large effect interval was from 0.5 to infinity, the prior density approaches zero as effect sizes approach infinity. For the Bayes Factor analysis, values greater than 1 indicate support for a small effect over a large effect. A Bayes Factor of 5 in favor of either model was selected a priori to indicate substantive support from the data. We acknowledge that Bayes Factors should primarily be used to inform relative plausibility of competing models but suggest their use as the basis for decision criteria can be instructive (e.g., Jeffreys, 1961). Finally, posterior estimates of the effect size for the difference between the single and dual task condition for each outcome measure are reported (see also Figure 4).

The median posterior estimate for the mean difference between precision for the single (M = 0.61, SD = 0.3) and dual task (M = 0.54, SD = 0.32) conditions was 0.07 (95% credible interval [−0.0064, 0.15]). The data were 6.29 times more likely under a model assuming a small versus a large effect. The median posterior estimate for the effect size of the change in precision between the single and dual task conditions was 0.32 (95% credible interval [−0.027, 0.68]).

For the probability of recalling the target orientation, the median estimate of the mean difference between the single (M = 0.83, SD = 0.17) and dual task (M = 0.71, SD = 0.27) conditions was 0.11 (95% credible interval [0.015, 0.2]). The Bayes Factor in support of a small versus a large effect was 1.93. Finally, the median estimate for the effect size was 0.41 (95% credible interval [0.054, 0.79]).

The median estimate for the difference in color judgment accuracy between the single (M = 0.91, SD = 0.072) and dual task (M = 0.9, SD = 0.06) conditions was 0.014 (95% credible interval [−0.011, 0.039]). The Bayes Factor in support of a small versus a large effect was 24.09. The median estimate for the effect size was 0.2 (95% credible interval [−0.16, 0.57]).

The median estimate for the difference in the proportion of letters correctly recalled between the single (M = 0.79, SD = 0.12) and dual task (M = 0.76, SD = 0.13) conditions was 0.031 (95% credible interval [0.0071, 0.056]). The Bayes Factor in support of a small versus a large effect was 1.02. The median estimate for the effect size was 0.46 (95% credible interval [0.1, 0.84]). The average span for participants was 6.23 (SD = 1.05; range = 5–9).

Overall, the Bayes Factors and effect size estimates are generally in alignment, indicating some support for small rather than medium-large dual task effects, and effect size estimates of between 0.2 and 0.5.

Exploratory Analysis

Given the reduction in the probability of recalling the orientation of the target bar between the single and dual task conditions, we explored whether this accompanied an increase in the probability of recalling the orientation of the other items in the array (non-targets) or guessing. The median estimate for the difference in the probability of recalling a non-target orientation between the single (M = 0.0000088, SD = 0.000018) and dual task (M = 0.000059, SD = 0.00015) conditions was −0.000045 (95% credible interval [−0.0001, 0.0000077]). The median estimate for the difference in the probability of a uniform response distribution, that is, guessing, between the single (M = 0.17, SD = 0.17) and dual task (M = 0.29, SD = 0.27) conditions was −0.11 (95% credible interval [−0.2, −0.014]).

While we feel the intervals chosen for the multiple component and embedded processes models are justified, we acknowledge others may disagree and prefer alternate intervals. To facilitate this disagreement, Table 1 replicates the Bayes Factor analysis used for the confirmatory analysis by varying the interval chosen to represent the two models. As the upper limit of the multiple component model interval is reduced, the evidence in favor of that model reduces. We have also created a Shiny web application (Chang, Cheng, Allaire, Xie, & McPherson, 2018) where readers are able to select their own intervals for the two models available at https://edjberry.shinyapps.io/BF_intervals/.

Table 1 The Bayes Factors (BF) in support of a small versus large dual task effect for different effect size intervals. Values lower than 1 indicate support for the embedded processes rather than the multiple component model effect size interval. Bold values indicate cases where either interval is supported by a BF of ≥ 5

Multiple component interval	Embedded processes interval	Color judgment	Precision	Probability target	Verbal accuracy
Note. Inf = Infinity.
(0, 0.1)	(0.3, Inf)	4.27	1.15	0.37	0.19
(0, 0.2)	(0.3, Inf)	5.09	1.71	0.66	0.38
(0, 0.3)	(0.3, Inf)	5.50	2.28	1.04	0.66
(0, 0.1)	(0.5, Inf)	18.71	3.17	0.69	0.30
(0, 0.2)	(0.5, Inf)	22.29	4.74	1.22	0.58
(0, 0.3)	(0.5, Inf)	24.09	6.29	1.93	1.02
(0, 0.1)	(0.8, Inf)	1,002.19	85.12	8.28	2.37
(0, 0.2)	(0.8, Inf)	1,194.33	127.15	14.64	4.60
(0, 0.3)	(0.8, Inf)	1,290.44	168.92	23.17	8.03

Table 1 The Bayes Factors (BF) in support of a small versus large dual task effect for different effect size intervals. Values lower than 1 indicate support for the embedded processes rather than the multiple component model effect size interval. Bold values indicate cases where either interval is supported by a BF of ≥ 5

View as image HTML

Finally, the analysis was rerun using only those trials where verbal accuracy was 100%. This criterion is commonly used when evaluating dual task effects (e.g., Morey & Cowan, 2004). The mixture model could not be used for this subset as there were insufficient trials for the model to converge. Nevertheless, the analyses of precision and color recall accuracy could still be carried out. For precision, the Bayes Factor in support of a small versus a large effect was 323.9. The median estimate for the dual task effect size was 0.04 (95% credible interval [−0.30, 0.38]). For color recall accuracy, the small effect model was supported by a Bayes Factor of 1,994.5 (median effect size estimate: −0.13; 95% credible interval [−0.49, 0.22]).

Discussion

This study investigated the magnitude of the cross-modal interference effect for concurrently remembering verbal (letter sequences) and visual information (orientation of colored bars). Bayes Factor analyses supported the prediction that there is a small, rather than a large, reduction in the precision with which items are represented in visual working memory when required to concurrently maintain verbal information. This analysis also supported the prediction that there would be a small reduction in the accuracy of recalling categorical color information for visual items. For the probability of recalling the target orientation, the data were more likely under a small effect model but failed to meet our a priori cut-off. The results for the letter recall were also indeterminate with respect to our two predictions. Thus, while not all our measures reached the preregistered cut-off for providing evidence to differentiate between the two models, on balance our results provide more support for multiple component accounts of WM and little clear support for a domain-general resource account of storage in WM.

The exploratory analyses showed that the reduction in the probability of recalling the target orientation in the dual task condition resulted in an increased probability of making a uniform response (i.e., guessing), rather than participants being more likely to recall a non-target orientation. Thus, a verbal load does not appear to increase the probability of mis-binding errors due to item features interfering in visual WM.

When discussing dual task effects, it is useful to distinguish between perceptual-motor and cognitive processes (e.g., Thalmann & Oberauer, 2016). Dual task costs are typically largest, both within and between modalities, when the interfering tasks involve overlapping perceptual-motor processing (Thalmann & Oberauer, 2016). For example, one would expect interference to be greater if both tasks require participants to make verbal responses. In contrast, dual task interference is generally smaller when the two tasks share only cognitive processes. This distinction fits nicely with our results as we ensured there was no overlap in perceptual-motor processes for our tasks. With our verbal task, stimuli were presented aurally and responses were spoken. For the visual task, stimuli were presented visually with manual responses. Therefore, our results are not contaminated by within-modality perceptual-motor interference inflating supposed cross-modal interference effects. This distinction is important where we want to isolate our inquiry to cognitive processes distinct from the attentional bottlenecks at input or during response output.

One possible limitation of this work could be that 3-item visual arrays are insufficient to capture most or all of a domain-general storage capacity in working memory. This would leave additional capacity to maintain verbal items, resulting in the small dual task costs we observe. However, the difficulty of the verbal task was set individually at each participant’s measured span precisely to address this concern. It was assumed that participants would use any domain-general storage capacity, in addition to passive storage, to maximize performance on the letter span task. This domain-general capacity would then not be available to the same extent in the dual task condition, resulting in a large deterioration in performance. While there was a reduction in performance, this did not meet our preregistered decision rule. This highlights the need for the field to focus on effect size predictions rather than simply whether effects are observed or not. We did not titrate the visual task given the difficulty of doing so with a continuous (analog) response task: performance can only be evaluated from a large number of trials meaning that any staircase procedure would be prohibitively long to complete.

The exploratory analysis reported in Table 1 illustrates how the strength of evidence can shift when the small and large effect size interval are varied. The evidence for a small dual task effect ranged from “decisive” to “barely worth mentioning” (Jeffreys, 1961), depending on the particular pair of intervals that were selected. However, on balance, the preregistered intervals indicate some support for small dual task effects and no evidence for large effects. The Shiny app we have created allows readers to choose their own pair of intervals and see how the results are affected (see Table 1).

Although the outcomes of this study somewhat favor a multiple component approach, they do not decisively decide between this and a general resource account of WM. Challenges remain in identifying effective methodological tools to cleanly distinguish between theoretical accounts. For example, the overall pattern of relatively small dual task costs observed in the present study might be captured by appealing to alternative distinctions between different forms of storage (namely the focus of attention and activated LTM) described within embedded processes accounts. Thus, the visual task could be accomplished via the focus of attention, while the letter stimuli in the verbal serial recall task are held in activated LTM. However, this speculative explanation is unlikely, not least given the recent persuasive arguments by Norris (2017) that LTM does not provide a plausible way of retaining serial order information. More broadly, it is notable that Cowan, Saults, & Blume (2014) modified the original Cowan (2005) theoretical framework by arguing for a peripheral component of working memory that functions like the phonological loop in the multiple component models and is separate from the focus of attention. This suggests that the embedded processes and the multi-component accounts might be starting to resolve their differences (see Baddeley, 2012; Gray et al., 2017; Hu, Hitch, Baddeley, Zhang, & Allen, 2014).

The two accounts could still be contrasted by increasing the number of to-be-remembered items in the visual task. At least, one multiple component account (Duff & Logie, 2001; Logie, 2011; Logie et al., 2004) explains dual task interference by suggesting that a fixed amount of general processing resource is required when completing two tasks simultaneously. This means the magnitude of the interference effect should remain constant under an increase in the overall demand of the two tasks. Some evidence for this was reported by Logie et al. (2004) but with small sample sizes in a study focused on contrasting healthy aging with Alzheimer’s disease. Crucially, in that study, the visual processing task involved following a moving target around a computer screen, so engaged perceptuo-motor processing load rather than memory for visual items. As noted earlier, memory for visual material may be supplemented by using verbal codes. For example, in the current experiment, approximate orientations might be coded as points on a compass (north, northeast, southwest, etc.). This might help explain why some of our measures (namely probability of target orientation recall and letter recall) did not reach the preregistered cut-off for small dual task effects. Future studies could adopt different methodologies, for example, using difficult-to-name colors or different shades of the same color as the visual memoranda, which could further minimize the potential contribution of verbal coding. If the storage of verbal and visual items is dissociable, then increasing the number of visual items should simply result in more visual information being forgotten in both the single and dual task conditions. This would not be affected by the imposition of verbal load over and above the cost associated with performing the two tasks that we observe here. On the other hand, if a domain-general storage capacity supports storage in WM, then the dual task cost should be larger when the number of items for the visual task is increased. If a larger total number of items draws on a shared storage capacity, the reduction in performance under dual task load should be more pronounced, supporting a domain-general account of WM. Future work could also investigate whether other factors that affect dual task interference, such as verbal rehearsal of the concurrent task (Morey & Cowan, 2005), generalize to precision measures.

This study represents an attempt to quantify the magnitude of dual task costs that emerge between verbal and (continuous and categorical) visual memory and compare these against preregistered predictions derived from multi-component and embedded processes accounts of working memory. Overall, three of our measures produced evidence for small dual task costs as predicted by the multi-component approach, while the remaining two were equivocal and did not reach our preregistered criteria for either small or medium to large effects. The outcomes of this study, as well as the opportunity to contrast how the adoption of differing effect size intervals provide shifting evidence for different models, should connect to and inform the ongoing movement to more robustly and transparently test theoretical accounts of working memory.