Personality Assessment To-Go

Formal Aspects of Participation in Unproctored Internet Testing

To obtain a broad view of the current state of research on the topic of formal aspects of UIT, we decided to focus on the aforementioned variables, namely, type of device, time of day, continuation, and total participation time. Concerns regarding possible device effects were already voiced decades ago (Bartram & Bayliss, 1984) with very little research since. Discussions about device effects must consider that a mere dichotomous separation into digital and non-digital does not represent the individual parameters within the large spectrum of digital devices, as it could reference stationary personal computers and mobile smartphones or tablets. These digital devices differ in many respects, such as user interface (e.g., touchscreen or keyboard) and mobility (e.g., stationary PCs or mobile smartphones).

Arthur and colleagues (2018) and Morelli and colleagues (2014) focused on interface and screen size and found no mode-based differences in results. However, Arthur, Keiser, Hagen, et al. investigated mobile versus nonmobile devices only in a proctored lab setting. Therefore, the study mainly focused on cognitive load stemming only from the device due to its structural characteristics (e.g., larger screen sizes, degree of permissibility). Similarly, Morelli et al. assessed only the work-related abilities and skills relevant to customer service advisors, so their results are hardly generalizable or applicable to personality assessment.

Apart from screen size and keyboard, another fundamental way that stationary and mobile devices differ is in the aspect of mobility. When using a predominantly stationary device, such as a PC or a laptop, one typically sits down and does not move the device (or oneself) very often. By contrast, purely mobile devices, such as smartphones or tablets, accompany many people in practically all everyday situations with almost unlimited mobility and can thus be used (e.g., when participating in UIT assessments), for instance, while riding on public transportation, waiting in line, or taking a walk. These differences in mobility may be associated with the attention that people allocate to the device or in their willingness to focus on the contents on the screen. As a result, data quality may be expected to differ between these device types. Interestingly, this fundamental difference between stationary and mobile devices has received practically no attention in (personality) research so far (one of the few examples is the study by Stachl et al., 2020). This is a severe oversight, considering the steep increase in online studies and the fact that participants are free to choose the device they want to use to participate in such studies.

For the remaining three formal aspects, research has also been sparse. Participation continuity was a topic of interest in Miller and colleagues (2002), where participants took breaks as a proxy for general disturbances due to uncontrolled conditions (e.g., fatigue or time pressure). These results were compared with a traditional paper-and-pencil condition and a web-based condition without interruptions. No differences in results were found, but the study assessed alcohol use, and thus, the findings might not apply to personality assessment (e.g., for a study on the weak associations between the Big Five and alcohol consumption, see Lackner et al., 2013).

Effects of time of day and total participation time have also been understudied in the literature. Lawrence et al. (2009) found no difference when looking at time of participation in a web-based personality assessment, but they used unspecified measures of constructs such as “Achievement Orientation,” “Positive Affectivity,” and “Time Management.” With that said, effects of time of day have long been studied in the realm of traditional proctored personality research, especially for the Big Five (Deyoung et al., 2007). The Big Five is a well-established and widely accepted model of personality (McCrae & Allik, 2002). Not only is it used frequently, but it also has solid empirical backing, especially concerning its longitudinal stability and associated life outcomes (Soldz & Vaillant, 1999; Soto, 2019). A relevant concern is that limiting the time of day to certain time slots could bias the data on personality as there are well-studied preferential times for certain personality configurations. Exemplarily, Randler (2008) found significant correlations between time of day and the Big Five (e.g., a positive relationship between agreeableness and generally being a “morning person”). Therefore, limiting the time slot could result in a biased sample due to diurnal preferences or a lower quality data set when certain participants complete the study during an undesired time of day. Unproctored UIT studies that do not limit the time of day could hence be seen as advantageous in this regard.

In light of the increasing number of studies conducted online, it seems advisable to study the consequences of UIT in personality assessment. Several studies have already begun analyzing the effects of the formal aspects of participation, which are commonly unstandardized in UIT. However, both comprehensive evidence in general and evidence in the context of personality assessment are lacking. For this reason, there is a need for research on how formal aspects of UIT influence both assessment quality and results.

The Present Study

In the present study, we investigated the four formal aspects (1) type of device, (2) time of day, (3) continuation, and (4) total participation time in relation to content- and quality-related aspects of the Big Five. We used the type of device to operationalize mobility and dichotomized it into stationary and purely mobile devices. Stationary devices, such as PCs and laptops, allow the user only a limited degree of physical mobility and usually require the user to sit down in front of the device. This requirement may be accompanied by a higher level of attention allocated toward the device. By contrast, purely mobile devices, such as smartphones, allow practically unlimited flexibility and are often used accordingly, for instance, on public transportation or when waiting in line. Such mobility may be associated with lowered attention and greater responsiveness to potentially distracting stimuli.

Time of day represents the participants’ choice of when to begin participating in the study, which can be any time of the day. Continuation represents the different variations of study completion, including completion in one sitting, completion but with an intermission, and participating without completion. Total participation time represents the time used to complete the study and applies only to the group of participants who completed the study in one sitting.

We operationalized content effects by testing for Big Five mean differences between the levels of the formal aspects (i.e., testing whether the levels of the aspects explained variance in the Big Five). Quality was operationalized with the use of internal consistency and response styles. Internal consistency (Cronbach’s α and McDonald’s ω) is a frequently used indicator or proxy of reliability or the interrelatedness of the items used to measure a latent construct (Tavakol & Dennick, 2011). In this study, the analysis of internal consistency scores allowed insights into differences in the amount of measurement error present in the Big Five factors at the different levels of the formal aspects. Response styles provide extensions of these quality evaluations. To our knowledge, response styles have been studied in various contexts (e.g., in cross-cultural comparisons; Johnson et al., 2005) but not in the context of UIT. In this study, we utilized participants’ total numbers of extreme responses and the standard deviations across the Big Five items as potential sources of measurement error (Cronbach, 1946). Taken together, these content and quality aspects may offer additional insights into participants’ response behavior.

Generally, we expected to find differences in all four formal aspects (i.e., type of device, time of day, continuation, and total participation time) about the quality- and content-related aspects of the Big Five. Specifically, we expected that the use of stationary devices would increase quality (Arthur, Keiser, & Doverspike, 2018; Arthur, Keiser, Hagen, et al., 2018). Furthermore, we expected that the internal consistencies would be highest in the morning due to increased attentiveness in comparison with nighttime or even afternoon completion, although this point again had to be analyzed exploratorily to some extent due to the lack of previous studies. Regarding total participation time, we expected total participation times farther from the median of the distribution to be associated with lower internal consistency. Few or no intermissions and hence continuous study participation should also be linked to higher data quality. In addition, we expected certain personality dimensions to have preferential participation times, such as agreeable people preferring to participate in the morning versus at noon (based on the time-preference studies, see Randler, 2008).

In general, with the lack of research on differences in the formal aspects of participation in UIT, by and large, and particularly concerning personality assessment, these hypotheses should be characterized as rather speculative and the analyses as exploratory.

Sample and Procedure

The sample was recruited from different universities in Germany using social media network sites and apps (e.g., Facebook, Instagram, WhatsApp), email forwarding, and newsletters. The advertisement informed potential participants that the study involved personality assessment. It included a weblink to the online assessment tool “Questback” also known as “Unipark,” used to collect the data. Before accessing the study, participants gave informed consent for their data to be used for scientific purposes. They could also have their data entirely deleted after their (partial) participation, but no participant opted for this.

Participants were free to choose the time of day to participate and how long they could take to complete each item. They were also free to stop participating at any time without any consequences. All assessments were in accordance with the local and national ethical guidelines. Participation was voluntary and not monetarily rewarded.

The primary sample of participants who completed the assessment consisted of 441 German participants (343 women). Their mean age was 25.0 years (SD = 6.3), ranging from 18 to 64. As their highest level of education, 7.7% reported that they had attended school for 8–10 years, 66.4% reported 12–13 years, and 25.9% reported holding a university degree; 62.1% were students, 35.1% were employed, and the rest were unemployed or retired.

A secondary sample of 527 participants began participating in the study but stopped before they reached the end of the assessment. Most of them (396) did not provide any data and aborted their participation just after accessing the initial “welcome” page, which described the study. Such participants are common in web-based studies and represent a relevant subsample on their own, but they are rarely included in the analyses.

Measures

After assessing personal details (gender, age, education, and employment), the Big Five personality factors were assessed using the German version of the NEO Five-Factor Inventory (NEO-FFI; Borkenau & Ostendorf, 2008). The NEO-FFI contains 60 items for assessing Agreeableness (e.g., “I try to be courteous to everyone I meet”), Conscientiousness (e.g., “I keep my belongings clean and neat”), Extraversion (e.g., “I like to have a lot of people around me”), Neuroticism (e.g., “I often feel tense and jittery”), and Openness (e.g., “I often try new and foreign foods”). Participants rated each item on a 5-point Likert scale measuring agreement. Cronbach’s αs for the five factors ranged from .75 to .88. Further details are provided in the Results section.

In addition to these data, we also recorded meta-information about each time the participants accessed the study. These included the date and time when the participants first accessed the study, total participation time, and the type of device used to participate. Such information can be assessed by all online assessment tools and are commonly provided to the researchers by most of them.

Calculation of Study Variables

Using the measures and meta-information about participation described above, we created variables about formal aspects of participation (i.e., time of day, device, continuation, and total participation time), content (Big Five mean scores), and quality (internal consistency [α and ω] measures of the Big Five as well as response styles). With respect to the formal aspects of participation, we used the meta-information about participation described above to create the four categorical variables time of day, device, continuation, and total participation time. Time of day represents the time of day of participation. It was created using the time when the study was first accessed and was clustered into the four categories night (23:00–04:59 o’clock), a.m. (05:00–12:59 o’clock), noon (13:00–15:59 o’clock), and p.m. (16:00–22:59 o’clock). The device represented the type of device used to participate and was clustered into two categories: primarily stationary devices (e.g., laptop and desktop computer) and primarily mobile devices (e.g., smartphone; cf. Arthur, Keiser, Hagen, et al., 2018). Continuation represents whether participants completed the assessment without an intermission, with an intermission, or aborted it entirely. Total participation time represents the time the participants took to complete the assessment, measured in seconds. For this study, the originally continuous distribution of total participation time was clustered into three categories: participants with the lowest 25%, the middle 50%, and the highest 25%. Means of total participation time of these three groups were 742 s (SD = 109) for the lowest 25%, 1,102 s (SD = 140) for the middle 50%, and 1,886 s (SD = 527) for the highest 25%. Table 1 presents the total numbers of participants at each level of each of the four formal aspects.

Table 1 Number of participants per formal aspect

Aspects and levels	n	%
Time of day
Night	34	3.5
a.m.	223	23.0
Noon	344	35.5
p.m.	367	37.9
Device
Stationary	182	18.8
Mobile	786	81.2
Continuation
Aborted	527	54.4
Intermitted	36	3.7
Completed	405	41.8
Total participation time
Lower 25%	102	23.1
Middle 50%	203	46.0
Upper 25%	100	22.7

Table 1 Number of participants per formal aspect

View as image HTML

With respect to the content- and quality-related aspects of participation, we calculated the mean Big Five scores, their internal consistencies, and two response styles. Internal consistency was calculated as Cronbach’s α and McDonald’s ω using the R package “coefficient alpha” (Zhang & Yuan, 2016) with robust estimates and 1,000 samples for bootstrapping 95% confidence intervals. Additionally, we used the NEO-FFI items to calculate two response styles: frequency of extreme responses (i.e., responses at either end of the continuum: “1” or “5” on the scales ranging from “1” to “5”) and the standard deviation for each participant across all NEO-FFI items (inverse items not reversed). The former indicates a willingness to agree strongly or disagree with statements (but may be susceptible to acquiescence), whereas the latter indicates the general variability of the responses. All subsequent calculations using the Big Five mean scores or the two response styles were controlled for participants’ age and gender (except for the calculation of internal consistencies).

Contingency Between the Formal Aspects

To test whether the four formal aspects were independent of each other, we used χ² to estimate the bivariate contingencies. Table 2 summarizes these results.

Table 2 Contingencies between the four formal aspects based on their marginal distributions

	n	χ2	df	p	φ
Time of day – device	968	14.75	3	.002	.12
Time of day – continuation	968	28.08	6	< .001	.17
Time of day – total participation time	405	6.25	6	.396	.12
Device – continuation	968	12.16	2	.002	.11
Device – total participation time	405	6.71	2	.035	.13

Table 2 Contingencies between the four formal aspects based on their marginal distributions

View as image HTML

We found contingencies between time of day and device, time of day and continuation, device and total participation time, and device and continuation (see Table 2). The dominant effects are summarized in the following. For noon (as one level of the time-of-day variable), more participants than expected used stationary devices (84 vs. 64.7 expected) and, accordingly, fewer than expected used mobile devices (260 vs. 279.3 expected). The opposite was true for p.m. (49 vs. 69.0 expected, and 318 vs. 298 expected, respectively). For noon, fewer participants than expected aborted their participation (153 vs. 187.3 expected) and, accordingly, more than expected completed the study without an intermission (177 vs. 143.9 expected). With respect to total participation time, more stationary-device users than expected were among those with the 25% lowest total participation times (34 vs. 24.4 expected) and, accordingly, fewer mobile-device users than expected (68 vs. 77.6 expected). This indicates that using a stationary device tended to be associated with shorter participation times. With respect to continuation, more mobile-device users than expected were among those who aborted their participation (448 vs. 427.9 expected) and, accordingly, more stationary-device users than expected were among those who completed the assessment without an intermission (97 vs. 76.1 expected). Overall, the effect sizes for all contingencies were comparable and small (see Table 2).

Formal Aspects in Relation to the Quality-Related Aspects of Participation

To analyze whether the formal aspects were associated with higher or lower quality of the assessed data, we used two aspects of data quality: internal consistency and response styles (as detailed above). For internal consistency, we calculated both Cronbach’s αs and McDonald’s ωs with confidence intervals (as described above) for each of the levels of the four formal aspects. Figure 1 provides a graphical overview of the results. Contrary to our expectations, we found only small differences.

To test these differences for statistical significance, we used the R package “cocron” (Diedenhofen & Musch, 2016), which implements the tests to compare Cronbach α scores as suggested by Feldt et al. (1987). Multiple comparisons for the formal variables time of day, device, and total participation time yielded significant differences only for time of day with respect to Agreeableness (χ² = 9.09, df = 3, p = .028), Conscientiousness (χ² = 36.79, df = 3, p < .001), Extraversion (χ² = 26.33, df = 3, p < .001), and Neuroticism (χ² = 24.50, df = 3, p < .001). Pairwise comparisons showed several significant differences, which are indicated by brackets in Figure 1. It should be noted that most of these significant comparisons were with the level night (of the time-of-day variable), which only included a small number of participants. The level night also had larger confidence intervals for some of the Big Five (see Figure 1). Post hoc inspection of these Big Five dimensions revealed a few items that had surprisingly low or even negative correlations with the respective total score (e.g., Item 9 from Agreeableness, which is “I often find myself arguing with my family and colleagues,” had a surprisingly corrected item-total correlation of −.53).

With respect to the two response styles (frequency of extreme responses and individual standard deviation), we calculated five analysis of variance (ANOVA) models for each dependent variable: univariate ANOVAs (Models 1–3), a multivariate additive model with main effects (Model 4), and a complete multivariate model including all interaction terms (Model 5), as presented in Table 3. The frequency of extreme responses differed only within the device levels (see Model 5), F(1, 383) = 8.95, p < .05, and only with a very small effect of η² = .02 (a higher frequency of extreme responses for participants who used a mobile device). Contrary to our expectations, the individual standard deviation did not depend on the formal aspects (see Table 3).

Table 3 ANOVA models of the two response styles calculated across all NEO-FFI items

		Frequency of extreme responses		Individual standard deviation
	df	F	η2	F	η2
Note. We controlled for age and gender in the dependent variables. *p < .05; **p < .01.
Model 1
Time of Day	3, 437	1.41	.01	0.99	.01
Model 2
Device	1, 439	7.43**	.02	0.44	.00
Model 3
Total Participation Time	2, 402	0.92	.01	0.54	.00
Model 4
Time of day	3, 398	0.86	.01	1.17	.01
Device	1, 398	5.73*	.01	0.32	.00
Total Participation Time	2, 398	1.36	.01	0.77	.00
Model 5
Time of Day	3, 383	0.21	.00	0.97	.01
Device	1, 383	8.95**	.02	2.61	.01
Total Participation Time	2, 383	0.39	.00	0.20	.00
Time of Day × Device	3, 383	1.98	.02	1.27	.01
Time of Day × Total Participation Time	6, 383	1.35	.02	1.25	.02
Device × Total Participation Time	2, 383	1.59	.01	1.21	.01
Time of Day × Device × Total Participation Time	4, 383	0.54	.01	0.13	.00

Table 3 ANOVA models of the two response styles calculated across all NEO-FFI items

View as image HTML

Formal Aspects in Relation to the Content-Related Aspects (Big Five)

We used the same structure of ANOVA models again to analyze the dependencies between the formal aspects and the Big Five (see Table 4). Contrary to our expectations, we did not find significant differences in the formal aspects with respect to Conscientiousness or Neuroticism. In line with our expectations, we found several differences with respect to Agreeableness, Extraversion, and Openness as follows.

Table 4 ANOVA models of the Big Five mean scores

		Agreeableness		Conscientiousness		Extraversion		Neuroticism		Openness
	df	F	η2	F	η2	F	η2	F	η2	F	η2
Note. We controlled for age and gender in the dependent variables. *p < .05; **p < .01.
Model 1
Time of Day	3, 437	3.68*	.03	0.22	.00	3.94**	.03	1.02	.01	2.72*	.02
Model 2
Device	1, 439	3.21	.01	0.60	.00	6.08*	.01	2.97	.01	6.53*	.02
Model 3
Total Participation Time	2, 402	1.20	.01	1.60	.01	2.69	.01	0.63	.00	1.44	.01
Model
Time of Day	3, 398	3.50*	.03	0.23	.00	3.57*	.03	0.88	.01	2.07	.02
Device	1, 398	6.06*	.02	0.14	.00	2.39	.01	1.60	.00	7.18**	.02
Total Participation Time	2, 398	1.16	.01	1.69	.01	1.50	.01	0.33	.00	2.42	.01
Model 5
Time of day	3, 383	2.95*	.02	0.13	.00	4.28**	.03	0.97	.01	0.91	.01
Device	1, 383	1.72	.00	0.10	.00	0.20	.00	0.21	.00	1.53	.00
Total Participation Time	2, 383	0.55	.00	0.17	.00	0.47	.00	0.17	.00	2.18	.01
Time of Day × Device	3, 383	0.59	.01	1.30	.01	0.10	.00	0.16	.00	0.28	.00
Time of Day × Total Participation Time	6, 383	2.51*	.04	1.14	.02	1.03	.02	1.20	.02	0.60	.01
Device × Total Participation Time	2, 383	0.92	.01	1.39	.01	0.42	.00	0.23	.00	2.52	.01
Time of Day × Device × Total Participation Time	4, 383	1.05	.01	1.63	.02	0.61	.01	1.79	.02	0.38	.00

Table 4 ANOVA models of the Big Five mean scores

View as image HTML

Bonferroni-adjusted post hoc analysis of the interaction effect between time of day and total participation time for Agreeableness (see Model 5), F(6, 383) = 2.51, p = .022, η² = .04, revealed a significant difference between a.m. and noon among participants who belonged to the group with the 25% lowest total participation time rates, t(68) = 3.63, p = .001. Here, Agreeableness was higher for participants who completed the assessment in the morning before noon (5–12 o’clock) compared with at noon (13–15 o’clock). Despite the Bonferroni adjustment, this effect seemed extremely specific and may thereby have resulted by chance. This interpretation may also be adequate for the effect of the device on Agreeableness, which was significant in the additive Model 4, F(1, 398) = 6.06, p = .014, η² = .02, with higher scores for stationary devices. However, this effect was not found in the interaction effect Model 5, F(1, 383) = 1.72, p = .191.

Bonferroni-adjusted post hoc analysis of the main effect of time of day on Extraversion, F(3, 383) = 4.28, p = .005, η² = .03, revealed significantly lower scores, t(279) = 3.02, p = .008, for participants who completed the assessment at noon (13–15 o’clock) compared with a.m. (5–12 o’clock).

Finally, for Openness, there was a main effect of device in Model 4, F(1, 398) = 7.18, p = .008, η² = .02. Participants who used a mobile device scored significantly lower than participants who used a stationary device. Again, this effect was prevalent in Model 4 but not in Model 5, F(1, 383) = 1.53, p = .216. In general, all effects were small.

Limitations

Many online studies use professional participants recruited from websites such as Mechanical Turk (MTurk) or Prolific. An inherent problem with using such samples in empirical studies is that it confounds the particular online assessment methodology with assessing a particular type of participant. By contrast, in the present study, we recruited participants the same way we would have recruited them for a laboratory study. Therefore, our findings more closely resemble the case in which a known sample is assessed, but we used an online assessment methodology. However, this procedure entails the potential limitation that the present findings cannot be applied to other types of samples (e.g., MTurk or Prolific) with certainty. For instance, whereas it may be easier for students to participate in online studies in the morning or at noon, MTurk participants who participate at these hours may be more likely to be professional participants with lower data quality (Goodman et al., 2013; Peer et al., 2021).

Some further limitations must be mentioned. First, as Table 1 shows, the sample sizes of the subgroups varied extensively (e.g., 34 at night as opposed to 344 in the a.m.). These sample size differences could have affected the detectability of possible effects. Second, the sample potentially had an educational bias because the majority consisted of university students. Third, we could not control whether some participants accessed the study from a different time zone. This could have influenced the time-of-day variable because time was recorded as server time rather than local time. Considering that the study was conducted in German, spread across German students and in German-speaking social media groups, we believe this issue is likely minor. Nevertheless, we cannot exclude the possibility that some participants accessed the study from a different time zone, thus resulting in a certain degree of measurement error in the time variable.

The meta-information used in this study was rich in its informative value and allowed us to investigate formal aspects in detail. Yet, some concerns remain, for example, whether participants who were registered as participating “without an intermission” may have taken a break while leaving the website running. These concerns could be resolved by adding more items to the meta-information. We decided not to use such items here, but they could be utilized in future studies.

The study was not conducted experimentally. Randomized-controlled experiments are considered the gold standard in psychological research for a good reason, as they allow for the assumption of the equal distribution of errors across the different sample groups and because of the potential to assess causality. Yet, experimental studies also decrease ecological validity. Especially in the given study where the aim was to test how different natural environments are associated with results, it would have been counterproductive to limit these environments to experimental set-ups. It could be argued that not randomizing the formal aspects (e.g., the devices people used for the study) could result in confounding effects stemming from selective distortions. But we argue that assigning device types (or any of the other formal aspects in the present study) to participants could feel unnatural to them, thus reducing the informative value of the study and creating artificial effects (e.g., possible effects might not be attributable to the devices but instead to whether the assigned device matched the participants’ preferences or not). Even with the expected trade-off of possible confounds, we prioritized ecological validity (i.e., so we could study actual behavior and its dependencies).

Finally, the present study was limited to effects on the Big Five, which are highly stable and consistent personality traits (McCrae & Allik, 2002). The influence of the formal aspects of participation might be more relevant in psychological constructs with higher expected day-to-day variation than the Big Five. An example of such a construct is self-esteem, which shows variability over the course of a day (Hank & Baltes-Götz, 2019) and weeks (Kernis, 2005). The study also focused on a questionnaire. Other measures are also used in personality psychology, most notably narrative measures. These could behave differently in a UIT setting and hence could be of interest in future studies. In general, further studies with a similar differentiation of formal aspects are welcomed as this is a necessary basis to justify using UIT-derived data.

Conclusion

In sum, the results are quite affirming as they do not call for an overly cautionary use of UIT. They show that even when isolating individual formal aspects with meta-information, differences based on these variations were not concerning. Therefore, our results provide more confidence in the use of data derived from unproctored web-based platforms in the realm of personality psychology. Debated concerns regarding device effects or diurnal preferences, for example, do not seem to hold equal bearing in the realm of noncognitive constructs.