Web-Based Research in Psychology

Web Surveys and Questionnaire Research

This type of web-based research is the most frequently used, yet it also may be the most error-prone. A general advantage of surveying as a methodology is its ease, and thus we see much ad hoc use with everything associated with quick-and-dirty approaches. Related, many errors frequently seen in web questionnaires came and come from a lack of understanding of the technology and that a direct transfer from the paper-based or computer-based formats to an Internet-based format is impossible.

Some of the typical pitfalls are technological. For example, Birnbaum (2004) observed a case of erroneous coding: in that web questionnaire, if a person were from India or if he or she did not respond, the coded value was the same, 99. Reips (2002a) highlights that the way HTML forms work may lead to overwriting of data due to the same names associated with form elements, for example, assigning the variable name “sex” to both an item reporting one’s own sex and an item asking about frequency of sexual activity.

In a recent meta-analysis, Daikeler and colleagues (2020) confirmed earlier findings that response rates in web surveys are lower than in other survey modes, roughly by 12%. The expected length of web questionnaires is, of course, a major factor in a respondent’s decision-making for participation and response. Galesic and Bosnjak (2009) experimentally varied both expected (10, 20, and 30 min) and actual questionnaire length. They found, “as expected, the longer the stated length, the fewer respondents started and completed the questionnaire. In addition, answers to questions positioned later in the questionnaire were faster, shorter, and more uniform than answers to questions positioned near the beginning” (p. 349).

Reips (2010) lists several typical errors that frequently happen when constructing online questionnaires:

• •
  preselected answer options in drop-down menus, resulting either in submission of the default answer option as a chosen answer when the item is really skipped or provoking an anchoring effect,
• •
  overlapping answer categories,
• •
  no limitations set or announced to size of text to be entered in text fields,
• •
  lack of options that indicate reluctance to answer (e.g., “don’t want to answer” or “no answer”), especially for sensitive items,
• •
  all items on one run-on web page (see OIOS Technique section below),
• •
  incorrect writing (e.g., errors in instructions or questions).

An excellent resource for many topics around web survey methodology is Callegaro and colleagues (2015). It summarizes many important learnings from the scientific literature on moving surveying to the web and supersedes many earlier attempts in scientific depth and applicability.

Despite the many benefits of web-based research, researchers and others have expressed concerns about online data collection in surveying. For the important case, in which generalization to a country population is needed, Dillman and colleagues (2010), in their editorial “Advice in surveying the general public over the Internet” for the International Journal of Internet Science, make the prediction “The first decade of this century in web surveying is likely to be recalled as a time of much uncertainty on whether random samples of the general public could be surveyed effectively over the Internet. A significant proportion of households in most countries are not connected to the Internet, and within households, some members lack the Internet skills or frequency of use needed for responding to survey requests. In addition, households without Internet access differ sharply from those with it. Non-Internet households are older, have less education, and have lower incomes. Their inclusion in surveys is critical for producing acceptable population estimates, especially for public policy purposes. Web survey response rates in general public surveys are often dismal.” (p. 1). Despite the skeptical state of web surveying the general public, the authors made five recommendations that are likely to improve the situation, namely (1) use of a mixed-mode approach, (2) delivering a token cash incentive with the initial mail request, (3) using a mail follow-up to improve response rates and obtain a better representation of the general public, (4) refraining from offering people a choice of whether to respond by web or mail in initial contacts, and (5) using an experimental approach, in order to generate estimates for the meaning and sizes of various effects on response rates and non-response. Loomis and Paterson (2018) empirically investigated the challenges identified by Dillman and colleagues and found limited differences between survey modes when aggregating all results (in their case from 11 studies). Only the non-deliverable rate seemed consistently higher in online surveying than in mail surveying. Non-response rates, item non-response, and content of the results showed no or only small differences for the aggregated data. The authors interpreted differences in response rates according to mode as “random or idiosyncratic in nature, and perhaps more a matter of study population or topic than of mode” (p. 145).

A specific form of web surveying, if it considers verbal items, is web-based psychological testing (e.g., of personality or other individual differences constructs). A researcher who has pioneered this branch of web-based research is Tom Buchanan (e.g., Buchanan, 2000, 2001; Buchanan & Smith, 1999; Buchanan et al., 2005). His and others’ studies generally find equivalency to offline testing, with notable exceptions that indicate that any web-based test should be scrutinized with the standard reliability and validity checks during test development.

Web Experiments

Web-based experimenting has evolved since 1994 and was first presented at the SCiP Conference in Chicago 1996, with Krantz presenting the first within-subjects web experiment (Krantz et al., 1997) and Reips (1996) presenting the first between-subjects web experiment and the first experimental laboratory on the world wide web, the Web Experimental Psychology Lab (Reips, 2001). On track of the question whether lab and web and type of participant recruitment are comparable, Germine and colleagues (2012) addressed the question of data quality across a range of cognitive and perceptual tasks. Their findings on key performance metrics demonstrate that collecting data from uncompensated, anonymous, unsupervised, self-selected participants on the web need not reduce data quality, even for demanding cognitive and perceptual experiments.

Reips points out the ultimate reason for using the web to conduct experiments:

As a consequence, the many advantages of web-based experimenting, for example,

1. (1)
   web experiments are more cost-effective in administration, time, space, and work in comparison with laboratory research,
2. (2)
   ease of access for participants, also from different cultures and for people with rare characteristics (for accessibility see Vereenooghe, 2021),
3. (3)
   web experiments are generally truly voluntary,
4. (4)
   detectability of confounding with motivational aspects of experiment participation,
5. (5)
   replicability and re-usability, as the materials are publicly accessible,

quickly led to the new method’s proliferation (Birnbaum, 2004; Krantz & Reips, 2017; Musch & Reips, 2000; Reips, 2000, 2007b; Wolfe, 2017). As a consequence of these characteristics of web experiments, they frequently have been shown to collect higher quality data than laboratory experiments (Birnbaum, 2001; Buchanan & Smith, 1999; Reips, 2000). For example, in a field where all past studies from more than two decades had two digit sample sizes, spatial–numerical association, web experiments finally provided detailed results with small confidence intervals (Cipora et al., 2019). Tan and colleagues (2021) were highly successful in objectively measuring singing pitch accuracy on the web. With moderate-to-high test-retest reliabilities (.65–.80), even across an average 4.5-year period between test and retest (!) they see high potential for large-scale web-based investigations of singing and music ability. In some areas of psychology, more than half of all studies published are now conducted online, for example, in social psychology – however, due to a certain participant recruitment method discussed further below, the widespread use is not without problems (Anderson et al., 2019).

Beyond what to researchers continues to be a new way and sometimes challenging and questionable advancement beyond laboratory experimental research, web experiments have truly revolutionized digital business. In his bestseller Seth Stephens-Davidowitz (2017) writes:

While early and quickly, a number of web survey generators appeared on the market, with “Internet-Rogator” by Heidingsfelder (1997) being one of the first, only a few web experiment generators are available today. For a recent listing of software for experiments (laboratory or web) and links to further resources, see the helpful Google page by Weichselgartner (2021). Figure 1 shows the newest version of WEXTOR (https://wextor.eu, also available on the iScience server at https://iscience.eu), one of the longest-standing web experiment generators.

Mobile Experience Sampling

This method is sometimes also described under the terms “ecological momentary assessment” or “ambulatory assessment” (Shiffman et al., 2008). It is a modern form of the diary method and has its strength in the current ubiquitous presence of smartphones, smartwatches, and other wearables in large proportions of populations in many technologically advanced societies. For instance, Stieger and Reips (2019) were able to replicate and refine past research about the dynamics of well-being fluctuations during the day (low in the morning, high in the evening) and over the course of a week (low just before the beginning of the week, highest near the end of the week) (Akay & Martinsson, 2009). The method is more accurate in capturing the frequency and intensity of experiences (Shiffman et al., 2008), but it is relatively burdensome for both participants and researchers, may lead to non-compliance, produces correlational data only, and its reliability is hard to determine. With context-sensitive experience sampling (see e.g., Klein & Reips, 2017), researchers can trigger questions based on times, events, app usage or location or combinations thereof, for example, ask for subjective well-being when someone leaves university only on afternoons when smartphone sensors registered a practical sports class was attended. Moreover, by using the experience sampling method, different research questions can be analyzed regarding the use of mobile devices in research, for example, whether well-being can be derived from the tilt of the smartphone (Kuhlmann et al., 2016).

Akin to upbeat statements regarding the potential of computers in psychology in the 1960s and 1970s and extension of hopes for the impact of the web in psychological research in the 1990s, the proponents of mobile research declared, “Smartphones could revolutionize all fields of psychology and other behavioral sciences if we grasp their potential and develop the right research skills, psych apps, data analysis tools, and human subjects protections.” (Miller, 2012, p. 221). Critically, however, the reliance on consumer-grade devices in research comes with their limited reliability and fast-changing variance. For example, smartwatches do not agree much in their recordings of workout parameters, and their measures depend on walking speed (Fokkema et al., 2017). Furthermore, our study on smartphone sensors showed that measurements and their reliability vary by type and brand of smartphone and operating system (Kuhlmann et al., 2021). Smartphone sensing in the field will thus systematically suffer from worse measurements than possible in controlled studies in the laboratory.

While mobile experience sampling research is often not exactly web-based, the free open-source platform “Samply” (Shevchenko et al., 2021; https://samply.uni.kn) for experience sampling enables researchers to access the complete interface via a web browser and manage their present studies. It allows researchers to easily schedule, customize and send notifications linking to online surveys or experiments created in any web-based service or software (e.g., Google Forms, lab.js, SurveyMonkey, Qualtrics, WEXTOR.eu). A flexible schedule builder enables a customized notification schedule, which can be randomized for each participant. The Samply research mobile application preserves participants’ anonymity and is freely available at the Google and Apple App Stores. Shevchenko and colleagues (2021) demonstrated via two empirical studies the app’s functionality and usability.

“Non-reactive web-based methods” refer to the use and analysis of existing databases and text or media collections on the Internet (e.g., forums, picture collections, server log files, scanned document collections, newsgroup contributions). Such data can also include geolocation, that is, information about the place that may allow analysts to identify routes and timelines. “The Internet provides an ocean of opportunities for non-reactive data collection. The sheer size of Internet corpora multiplies the specific strengths of this class of methods: Non-manipulable events can be studied in natura, facilitating the examination of rare behavioral patterns.” (Reips, 2006, p. 74). While lab-based research creates social expectations that might motivate participants to answer and perform in unusual ways, data from archival or non-reactive research will not contain biases that come from reacting to the research situation or personnel (hence “non-reactive”). Such non-reactive research is easy to do on the web.

For some archival data, there are even specific interfaces. For example, upon its initiative to scan as many of the world’s books as possible and make them available to the public, Google also created a specific search engine to search this corpus, Google Books Ngram Viewer (short: Google Ngram, https://books.google.com/ngrams/). With it, relative frequencies of words within the corpus can be analyzed per year, and thus, it is possible to create timelines that show word use over time since the year 1800. Michel and colleagues (2011) describe how this tool allows for research options unprecedented in the history of science, in data mining cultural trends as reflected in books. Younes and Reips (2019) provide guidelines for such research, for example, the use of synonyms, word inflections, and control words to assess a word’s base rate in a given year and language corpus.

Research That Could Not Be Done Without the Web

Of course, there is a lot of web-based research that cannot be done without the web because it is research that concerns the web. I will not review this research here. Instead, I will focus in this section on research that was impossible or only very difficult to do before the advancement of the web.

Design: One-Item-One-Screen (OIOS)

When designing a web-based instrument that consists of a number of items, a researcher has to decide how many items are going to be presented on one screen – or whether this may vary by device or participant. The OIOS or one-item-one-screen design has several advantages, namely “Context effects (interference between items) are reduced (Reips, 2002a), meaningful response times and drop out can be measured and used as dependent variables, and the frequency of interferences in hidden formatting is vastly reduced” (Reips, 2010, pp. 33–34). The design thus is routinely used in research studies, for example, in urban planning (Roth, 2006), where even a variant was developed, the “Two-item-one-screen design” (Lindquist et al., 2016).

OIOS has been proposed as a recommended strategy in a general framework for technology-based assessments by Kroehne and Goldhammer (2018). They write, “Item-level response times from questionnaire items (e.g., Likert type) are an interesting source of information above and beyond item responses.” (p. 543) and go on to criticize current implementations of the Programme for International Student Assessment (PISA) and other large-scale assessments as missing out on such opportunities in web-based assessments.

Including other data (para-data, meta-data) apart from the response itself, especially behavioral data that indicate timing, navigation, and switching of answers (Stieger & Reips, 2010) and thus can be important in identifying issues during test and questionnaire construction as well as diagnostic indicators beyond content responses, likely will become much more important in future surveying. Issues with adaptivity (e.g., in responsive design) and ethical application will continue to be discussed (see e.g., Hilbig & Thielmann, 2021).

Seriousness Check

The seriousness check is a technique that can be used in all reactive types of web-based research to significantly improve data quality (Aust et al., 2013; Bayram, 2018; Musch & Klauer, 2002; Reips, 2000, 2009; Verbree et al. 2020). Revilla (2016) found only limited evidence supporting the technique, but hers was an underpowered study with online panelists and a wording geared towards commitment that resulted in very few “non-serious” participants.

Nowadays, online studies are accessible to a large diversity of participants. However, many people just click through a questionnaire out of curiosity, rather than providing well-thought-out answers (Reips, 2009). This poses a serious threat to the validity of online research (Oppenheimer et al., 2009; Reips, 2002b, 2009). The seriousness check addresses this problem (Figure 2): In this approach, the respondents are asked about the seriousness of their participation or for a probability estimate that they will complete the entire study or experiment (Musch & Klauer, 2002). Thus, by using the seriousness check, irrelevant data entries can be easily identified and be excluded from the data analysis. To provide a rough idea of how large the seriousness check’s effect can be: It routinely was observed that of those answering “I would like to look at the pages only” around 75% will drop, while of those answering “I would like to seriously participate now” only ca. 10–15% will drop during the study. Overall, about 30–50% of visitors will fail the seriousness check, that is, answer “I would like to look at the pages only” (Reips, 2009).

Figure 2 depicts a possible seriousness check proposed by Reips (2009). The participants are asked whether or not they want to participate seriously in the experiment (“I would like to seriously participate now.”/“I would like to look at the pages only.”). Seriousness checks can be implemented before (e.g., Bernecker & Job, 2011) and after (Aust et al., 2013) participation in the study. Reips (2002b, 2008, 2009) has argued to conduct seriousness checks on the first page of the experiment because this is the best predictor of dropout rates and thus a measure of motivation. Additionally, conducting a seriousness check before the completion of the study can reduce the dropout rates (Reips, 2002a). The participant’s answer to the check question serves as self-commitment, as predicted by dissonance theory (Frick et al., 2001). Bayram (2018) experimentally showed that emphasizing the seriousness increased the degree of information participants accessed and their time they spent on the study. The technique has been shown to predict dropout and control for motivational confounding (Aust et al., 2013; Bayram, 2018; Musch & Klauer, 2002; Reips, 2002b, 2008, 2009). Some tools for web-based research, for example, WEXTOR (Reips & Neuhaus, 2002; https://wextor.eu), implement the seriousness check by default.

In a study with 5,077 participants representative of the Dutch population in education, gender, and age over 15 years, Verbree and colleagues (2020) recently confirmed that self-reported seriousness and motivation significantly predict multiple data quality indicators. In preparing a study that includes a seriousness check, it may be important to know that their results showed that self-reported seriousness varies with demography.

Instructional Manipulation Check (IMC) and Other Attention Checks

The IMC (Oppenheimer et al., 2009) was created with the same intention in mind as the seriousness check, to identify and avoid data from participants who are not as attentive in online studies as is required to gather quality data. In the case of the IMC, the focus is on attention during instructions. Of course, if participants do not properly attend to instructions, it is unlikely they will provide valid data during any subsequent task. Hence, screening them out at the beginning makes sense. However, there are several other attention checks that were designed to verify attention during tasks – we will also briefly look at those further below.

In the IMC, a request for unusual navigation is hidden within the instructions, for example, to not click on the submit button at the end of the page, but rather on the title of the text or a small blue dot. Only those who read the instruction carefully and comply will follow the navigation, and only their data will be analyzed later.

Another attention check, the Cognitive Reflection Test (CRT; Frederick, 2005) is a frequently used measure of cognitive vs. intuitive reflection, sometimes also discussed as a measure of numerical ability. A typical task it includes is the bat-and-the-ball problem: “A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? …cents”. The intuitive answer most inattentive participants go for is “10 cents”, but the correct answer is “5 cents”. With the widespread use of the task as a nice riddle and as an attention check in web-based research, many people have become familiar with this task. Indeed, Stieger and Reips (2016) found in a large study that 44% of the more than 2,200 participants were familiar with the task or a similar task and scored substantially higher on the test (Cohen’s d = 0.41). They also found that familiarity varies with sociodemographics. Web researchers should, therefore, better use lesser-known attention check items or the methods discussed above.

Multiple Site Entry Technique

The multiple site entry technique I first proposed in the mid-1990s (Reips, 1997, 2000) is a strategy used in web-based research to target different samples via different recruitment sites and compare their data. The method can be used in behavioral and social research to assess the presence and impact of self-selection effects. Self-selection effects can be considered a major challenge in social science research. With the invention of online research in the 1990s, the multiple site entry technique became possible because the recruitment of participants via different links (URLs) is very easy to implement. It can be assumed that there is no or very limited self-selection bias if the data sets coming from different recruitment sites do not differ systematically (Reips, 2000). This implies that the results from studies that use the multiple site entry technique and find no sample differences indicate high generalizability.

Implementing the multiple site entry technique works as follows: Several links to the study are placed on different websites, in Internet forums, social media platforms, or offline media that are likely to attract different types of participants or are mailed out to different mailing lists. In order to identify the recruitment sources, the published URLs contain source-identifying information, and the HTTP protocol is analyzed by different referrer information (Schmidt, 2000). This means a unique string of characters is appended to the URL for each recruitment source, for example, “…index.html?source=studentlist” for a study announcement mailed to a list of students. The data file will have a column (“source” in the example) containing an entry of the referring source for each participant (“studentlist” in the example). The collected datasets can then be compared for differences in results and differences in relative degree of appeal (measured via dropout), demographic data, central results, and data quality (Reips, 2002b). Figure 3 illustrates the multiple-site entry technique.

Several studies have shown that the multiple site entry technique is useful for determining the presence and impact of self-selection in web-based research (Reips, 2000, 2002a; Roth, 2006). Now, in the spring of 2021, Google Scholar shows more than 75 publications that mention the technique. It has been used in memory (Kristo et al., 2009), personality (Bluemke & Zumbach, 2012; Buchanan et al., 2005; Trapnell & Campbell, 1999), trauma surveys (Hiskey & Troop, 2002), cross-cultural music-listening (Boer & Fischer, 2011), landscape research (Roth, 2006), criminological psychology (Buchanan & Whitty, 2014), and political psychology (Kus et al., 2014) and it has entered the methodological discussion in the fields of experimental survey research (Holbrook & Lavrakas, 2019), sex research (Mustanski, 2001), and health science (Whitehead, 2007). Rodgers and colleagues (2003) used the multiple site entry technique to detect biased responses to their web questionnaire on drug use (subsequently validated by finding discussions of their research in forums on that particular recruitment website). The multiple site entry technique thus helps to detect potential sampling problems, which in turn ensures the quality of the data collection over the Internet (Dillman et al., 2010; Reips, 2002b). Therefore, the generalizability of the research results can be estimated when using the multiple-site entry technique (Reips, 2002a).

Subsampling Technique

Data quality may vary with a number of factors (e.g., whether personal information is requested at the beginning or end of a study, Frick et al., 2001, or whether participants are not allowed to leave any items unanswered and, therefore, show psychological reactance, Reips, 2002b). Subsampling analysis is a verification procedure. For a random sample drawn from all data submitted (e.g., from 50 out of 1,500 participants), every possible measure is taken to verify the responses, resulting in an estimate for the whole dataset. This technique can help estimate the prevalence of wrong answers by checking verifiable responses (e.g., age, gender, occupation) both by specific item and via the aggregate for any item. Ray and colleagues (2010) consider this technique as one possible option to better verify age and thus protect children on the Internet.

Buchanan and colleagues (2005, p. 120) noted that such “ways of estimating the degree of potential data contamination” along with other control procedures would need to be developed and researched more in the future. While a lot has happened in data analysis, for example, in the application of Benford’s law (Benford, 1938) to survey data (Judge & Schechter, 2009), the advances in design and procedure of web-based research (or all types of research, for that matter) need further research and development.

Warm-Up Technique

The warm-up technique in Internet-based experimenting, first proposed by Reips (2001), is a method that can be used to avoid dropout during the experimental phase of a study to maximize the quality of the data. This technique is based on the finding that most dropout will occur at the beginning of an online study (Reips, 2002c). Therefore, participants are presented with tasks and materials similar to the experimental materials before the actual experimental manipulation is introduced.

By using this technique, one can counter three main problems in web-based research: Firstly, the dropout during the actual experiment will be lower (Reips, 2001). Secondly, the dropout cannot be attributed to the experimental manipulation (Reips, 2002a, 2002b). Thirdly, only highly committed participants will stay in the experiment, and thus, the quality of the collected data is improved. Reips and colleagues (2001) showed that by using the warm-up technique the dropout during the actual experiment was extremely low (< 2%). In comparison, the average dropout rate in web-based research is much higher, in a review summarizing previous web-based experimental research, Musch and Reips (2000) found it to average at 34%.

The technique was used in personality and health behavior research (Hagger-Johnson & Whiteman, 2007), motivation (Bernecker & Job, 2011), landscape perception (Lindquist et al., 2016; Roth, 2006), gender research (Fleischmann et al., 2016), polar research (Summerson & Bishop, 2012), and has been implemented in software to conduct web experiments (Naumann et al., 2007). It entered the methodological discourse in Poland (Siuda, 2009) and China (Wang et al., 2015), where it is called 热身法.

Measurement

Psychology has a long history of finding strategies to measure behaviors, mental processes, attitudes, emotions, self-reported inner states, or other constructs. Pronk and colleagues (2020), following-up on various others (Garaizar et al., 2014; Plant, 2016; Reimers & Stewart, 2015; Reips, 2007a; Schmidt, 2001; van Steenbergen & Bocanegra, 2016), investigated timing accuracy of web applications, here with a comparative focus on touchscreen and keyboard devices. Their results confirm what theoretically was expected from the technical structure and limitations of the web (Reips, 1997, 2000):

An example of how the combination of computerized measurement and large sample sizes achievable on the web provided a shift in measurement accuracy that was long needed but overlooked in light of traditions and measurement burden is the switch from Likert-type rating scales to visual analogue scales. The latter has become much easier to administer via the web than on paper (where distances have to be measured with a ruler) and on computers (where software often lacks that type of scale as an option). Figure 4 shows an example of both types of scales. Note the visual analogue scale turns 100 this year (Hayes & Patterson, 1921).

As Reips and Funke (2008) point out, as a result of their experiment, data collected with web-based visual analogue scales provide better measurement than Likert-type scales and offer more options for statistical analysis. Importantly, visual analogue scales are not to be confused with slider scales, for which we know its handle causes potential problems because its default position may cause anchoring effects or erroneous ratings (Funke, 2016). Slider scales may also cause more problems for the less educated (Funke et al., 2011).

Dropout and Other Nonresponse

Dropout is more prevalent in web-based research and may have detrimental effects (Reips, 2002b; Zhou & Fishbach, 2016). Zhou and Fishbach describe how unattended selective dropout can lead to surprising yet false research conclusions. However, avoiding dropout (e.g., via high hurdle and warm-up techniques, and low tech principle) or controlling it (e.g., via seriousness check) are not the only strategies to deal with the higher prevalence of dropout on the web. Rather, dropout can be used as a dependent variable (Reips, 2002a). Bosnjak (2001) describes seven types of non-response behavior that can be observed in surveys, these all are good measures in web-based research.

In an early and widely cited study, Frick and colleagues (2001) experimentally manipulated the influence of announcing an incentive (or not), anonymity, and placement of demographic questions (beginning vs. end) on dropout. Incentive announcement and position of demographic questions showed large main effects on dropout, it ranged from 5.7% in the incentive known and demographic questions at the beginning condition to 21.9% in the incentive unknown and demographic questions at the end condition. They also found a strong effect (> 100 min difference in reported TV consumption per week) of the order of questions – asking both for time devoted to charity work and TV consumption had created a context that evoked socially desirable responses. Birnbaum (2021) also discusses dropout and reflects on early discussions and findings by those who adopted the web for research and developed the associated methodology.

DropR (http://dropr.eu) is ShinyApp and R software that we created to meet the increased need to calculate dropout rates because dropout is much more frequent in web-based research than in laboratory research. In the analysis and reporting of web experiments, the commonly high dropout makes it necessary to provide an analysis and often also visualize dropout by condition. DropR supports web researchers in both providing manuscript-ready Figures specifically designed for accessibility (see Vereenooghe, 2021) and all major survival and common dropout analyses on the fly, including Kaplan-Meier, chi-square, odds ratio, and rho family tests. Visual inspection allows for quick detection of critical differences in dropout. DropR is Open Source software available from Github (Reips & Bannert, 2015).

Dropout is particularly useful in detecting motivational confounding in experiments (Reips, 2000, 2002b). Whenever conditions differ in motivational aspects, there is the danger that this confound may explain any between-condition findings. On the web this would likely show in differential dropout rates. On the contrary, “because there is usually very minor dropout in offline experiments, the laboratory setting is not as sensitive as the Internet setting in regard to detecting motivational confounding.” (Reips, 2009). In a secondary analysis of two studies, Shapiro-Luft and Cappella (2013) confirm that motivational confounding can be detected in web-based studies with video content. It has been of consideration in conducting real-time multiplayer experiments on the web (Hawkins, 2015) and is now routinely used as an argument to conduct studies on the web rather than in the lab (e.g., Lithopoulos et al., 2020; Sheinbaum et al., 2013).

Interaction of Technology and Psychology

A chief issue in web-based research is interactions between technology and human factors, such as the type of device and personality of user. Which device someone uses is by no way randomly determined, it depends on one’s preferences – either directly, for example, because one may be an “early adopter” and likes to buy and use the newest technology or indirectly because one’s personality and demographics drive one to follow a certain education and end up in a certain profession where some type of technology is more common than in other professions. Buchanan and Reips (2001) analyzed responses of 2,148 participants to a web-based Five-Factor personality inventory and compared demographic items for users of different computing platforms. The responses of participants whose web browsers were Javascript-enabled were also compared with those whose web browsers were not. Macintosh users were significantly more “Open to Experience” than were PC users, and users using Javascript-enabled browsers had significantly lower education levels.

For research, an immediate consequence is to expect larger direct and indirect self-selection and coverage biases. If a web-based study relies on certain specific technologies, it will not reach every person with the same probability. For theory-guided and experimental basic research, this is less of an issue, but it may be for any research that tries to generalize from samples to populations. Technical and situational variance in itself in the presence of an effect strengthens the case for its generalizability (Reips, 1997, 2000), as it diminishes the probability that an effect is an artifact resulting from a specific technological setup in the laboratory. Krantz (2021) further notes that “technical variance also suggests a potential for modification of the theoretical understanding of the phenomenon” (p. 233) in web-based research and shows how this can be done with the famous illusion named after the founder of this journal, Ebbinghaus.

Recruitment

Once one understands the willingness of people to participate in web-based research (Bosnjak & Batinic, 2002), the question arises where to find them. Participants for web-based research in psychology can be found via various types of sources beyond any that also work for laboratory-based research: Mailing lists, forums/newsgroups, online panels, social media (Facebook, Twitter, Instagram, Snapchat, Tuenti, …), frequented websites (e.g., for news), special target websites (e.g., by genealogists), Google ads. Further, there are dedicated websites like “Psychological research on the net” (https://psych.hanover.edu/research/exponnet.html) by John Krantz or the “web experiment list” at https://wexlist.uni-konstanz.de

Within just a few years, it has become common to recruit workers as participants for “mini jobs” on crowdsourcing platforms like Clickworker, Prolific Academic, Cloudflare, or Amazon Mechanical Turk (AMT). Anderson and colleagues (2019) show social and personality psychology as an example of how dominant recruitment via AMT became a recruitment platform just within a few years. However, this proliferation of its use among researchers stands in stark contrast with much criticism about data quality from AMT workers (“MTurkers”) and the site’s limitations. Reips (in press) writes,

Ironically, a service that was developed to employ people who appear to work as a machine because the machine (computer) cannot do the task as well is now going down the drain because human beings have programmed scripts that pose as human workers.

Conclusion

Web-based research has enabled psychologists to explore new topics and do their research with previously impossible options to reach large heterogeneous samples and people with rare characteristics, run studies easily in several samples and cultures simultaneously (see Subsampling Technique section), and go deeper into multiple fine-grained measurements that may bring a revival of behavior instead of self-report and other measures in psychology.

At the same time, technological factors became more dominant in the ways psychological research is conducted, various dangers include (1) the dependency on non-scientific agents like big player companies who provide only selected data access to “embedded scientists”, (2) lack of reproducibility because of the many hardware and dynamic software factors involved and the quickly changing technology, and (3) the distance between the researcher and participants, who may not even be human but vague “agents” provided by commercial recruitment services.

Web methodologists are, of course, trying to keep up with the fast development and provide multiple solutions to the challenges posed by the web as a route for research. For those of us who actively have experienced research before the web revolution, it will be an important task to describe our insights from comparing pre-web with web-based research and teach a new generation of researchers in psychology. We will be the only generation to have witnessed and experienced the change.