Decrypting Log Data

General Online Activity and Learning Outcomes

One major advantage of online learning for educational research is the availability of a vast amount of information that can be derived automatically and unobtrusively. For example, courses offered via LMS allow collecting an enormous amount of log data about interactions with the platform (Campbell, 2007). However, the primary use of explorative approaches for analyzing log data results in a lack of theoretical grounding (Winne, 2020). Besides the number of studies comprising data-driven approaches for the decision which log data to examine, several researchers have considered pedagogical theories (see Tempelaar et al., 2015, for a review). For example, general online activities, such as total login time or login frequency, are considered indicators for learning engagement (Beer et al., 2010). Furthermore, according to Carroll’s (1963)model of school learning, time spent on learning is one of the crucial factors for students’ performance. Thus, time spent learning online might also be a crucial factor for web-based achievement (Jo et al., 2015). Besides, login frequency is associated with Engeström’s (1987)activity theory, which states that mere activity produces meaningful learning, leading to higher learning outcomes (Kupczynski et al., 2011). Finally, total login time and login frequency are considered proxies for time management strategies, as they are indicators for sufficient time investment and, thus, key factors for performance (Jo et al., 2015). However, there is a debate on the type of log data suitable to measure learning behavior (Agudo-Peregrina et al., 2014). Critics suggest focusing on quality, not the quantity of online learning behavior (You, 2016): As active participation is crucial for success, indicators that do not distinguish between active and passive engagement are problematic (Ransdell & Gaillard-Kenney, 2009). Overall, these contradictory assumptions on the usefulness of broad log data indicators go along with inconsistent findings on the association between those indicators of general online activity and learning outcomes: some studies reported no association (Broadbent, 2016), negative correlations (e.g., Ransdell & Gaillard-Kenney, 2009; Strang, 2016), or positive correlations (e.g., Liu & Feng, 2011; McCuaig & Baldwin, 2012; Saqr et al., 2017). Other studies that examined various online courses simultaneously obtained mixed results across different courses (e.g., Conijn et al., 2017; Gašević et al., 2016), indicating that online courses might be too heterogeneous to draw a general conclusion about the link between general online activity and learning outcomes.

The Present Study

Our aim was to systematically review findings on the relationship between two log data indicators of general online activity and learning outcomes within LMS. To guarantee a minimum level of comparability between classes, we focused on online university courses because they are more structured than informal online courses (Song & Bonk, 2016) and because they are usually graded, offering a measure of learning outcomes. Regarding general online activity, we focused on total login time and login frequency to assess the applicability of these broad measures derived from log data as a proxy of online learning behavior and examine how they are linked to educational outcomes (i.e., course grade or course score). Moreover, we examined the impact of several moderators to explain the inconsistent findings reported in previous literature, since in the course of recent technological developments (Palvia et al., 2018), teaching tools became more sophisticated and online courses became more diverse.

First, we use the term “online course” to describe all courses that include substantial online elements, that is, courses that are taught exclusively online (fully online courses), and courses that combine online and face-to-face delivered content (blended courses; Allen & Seaman, 2014). Compared to fully online courses, blended courses offer more structure through regular face-to-face sessions (Means et al., 2013), as well as the opportunity to easily get in touch with peers regularly (Broadbent, 2016). However, the varying parts of face-to-face teaching in blended courses cannot be tracked via log data (Mwalumbwe & Mtebe, 2017). Therefore, we expect a stronger relationship between general online activity and learning outcomes within fully online than within blended learning courses, as for the latter, substantial parts of learning might remain unreflected by log data (Hypothesis 1).

Second, online courses vary with respect to the emphasis put on the use of online discussion boards. An explicit instruction to use discussion boards implemented within the LMS might foster deeper learning while being online and lead to better achievement (Song et al., 2019). Although interactions in discussion forums are considered an essential part of learning (Uijl et al., 2017), the mere existence of a discussion board is not enough for promoting active participation within the LMS (Lee & Martin, 2017). Since active content engagement is crucial for students’ achievement (Ransdell & Gaillard-Kenney, 2009), we expect higher correlations between general online activity and learning outcomes for courses with instruction for discussion board usage (Hypothesis 2).

Third, online courses differ in their grading systems. For the present study, it is important to what extent online activities are explicitly incentivized. While some courses do not incentivize online participation at all, other courses either offer bonus points for regular online participation or even require online activities within the LMS (e.g., online group discussions, quizzes, or online assignments) as part of the final grade. These incentives encourage active online engagement (Tempelaar et al., 2019) and offer guidance for students to effectively use the tools and materials provided within the LMS. Therefore, we expect a higher correlation between general online activity and learning outcomes for courses that incentivize certain online activities through their grading systems (Hypothesis 3).

Fourth, we compare the operationalization of general online activity as total login time or as login frequency. Ever since the growing popularity of investigating log data, there has been the challenge of capturing log data that might be translated into psychologically meaningful variables (Seifert et al., 2018). As both operationalizations are theoretically reasoned with either Carroll’s (1963)model of school learning or Engeström’s (1987)activity theory, we want to explore if general online activity operationalized as total login time versus login frequency differs in their relationship with learning outcomes (Hypothesis 4).

Lastly, we considered publication year as a potential moderator. In the face of rapid technological advancements (Palvia et al., 2018), we expect changes in how LMS provides education. Through technological change, LMS tools become more advanced, and multiple types of learning tools can be implemented to foster students’ active engagement within the LMS (Kebritchi et al., 2017). Therefore, students ought to be enabled to benefit from a more interactive online learning experience, and online activity within recent studies might result in higher achievements than within older studies (Hypothesis 5).

Literature Search and Study Selection

We identified 33,724 potentially relevant studies from electronic searches in major scientific databases (PsycINFO, PsycArticles, PSYNDEX, and ERIC) and Google Scholar using the Boolean search term (online learning OR online course* OR web-based learning OR e-learning OR elearning OR learning management system* OR LMS OR learning analytics) AND (achievement OR performance OR outcome) in February 2021. We retrieved five further studies by calls for unpublished work (via mailing list of the German Psychological Society, ResearchGate, and Twitter). Additionally, we performed a “rolling snowball” search and identified 17 further studies by screening the reference lists of all eligible studies and by conducting forward citation tracking using Google Scholar. Finally, we contacted 19 corresponding authors of studies not reporting bivariate correlations and received them for three studies. We included all published or unpublished types of studies. See Figure S1 for the detailed literature search process, including specifications of all sources that were searched. Subsequently, these studies were included in the analysis depending on the following inclusion criteria: (a) The study investigated a fully online or blended course in an institutional setting; (b) General online activity was measured using log data and operationalized as total login time or login frequency (i.e., number of single logins or number of days with at least one login); (c) Learning outcome was measured as course grade or course score; (d) The study consisted of a sample comprising university students; (e) The study was published between 1969 (year of the first connection of the Internet) and 2021, and (f) was written in English or German; (g) The study reported at least one correlation between general online activity and learning outcome or appropriate statistics that could be transformed into correlations. Studies were excluded if: (h) General online activity was measured as a self-report as we focused on the usefulness of log data indicators of general online activity, or (i) measured as the duration or frequency of single activities in the LMS (e.g., time spent on quizzes, number of forum postings) because these types of log data fall within different categories (such as response-related or process-related log data; see Kroehne & Goldhammer, 2018), and (j) the study had a commercial e-learning course as setting as we focused on the specific context of higher education. After applying these criteria, 41 primary studies remained (see Table S1). Study selection followed a two-stage process. First, two researchers reviewed titles and abstracts of the first 50 records and discussed disagreements about eligibility until consensus was reached. Then, one researcher screened all titles and abstracts of all studies retrieved. In cases in which eligibility was unclear, the study was considered for the second stage in the form of a full-text review. A sample of full-text studies (~15%; 36/241) was independently screened by two researchers. The remaining full-text studies were screened by one researcher. Finally, the second researcher independently reviewed all included studies and those with uncertain eligibility. Again, disagreements about eligibility were resolved through discussion.

Coding Process

We developed a standardized coding manual and data extraction sheet for the data collection process (see Center for Open Science, 2021 for detailed information). The coding manual comprised eligibility criteria, guidelines for selecting effect sizes and coding, and definitions of all outcomes and other variables for which data were sought (i.e., name and description of the respective variables, guidelines regarding the format of coding, and coding examples). For each study, we extracted all relevant effect sizes for the association between general online activity and learning outcomes. Moreover, we collected data on study and sample and online course characteristics covering especially moderator variables (i.e., course format, emphasis of discussion, course activities as part of grading, operationalization of general online activity, and publication year) and general study information.

All studies were coded twice using the coding manual and data extraction sheets by two independent raters. To evaluate the coding process, Cohen’s (1960) κ for categorical variables and intraclass coefficients (ICC; Shrout & Fleiss, 1979) for continuous variables were calculated for the focal variables. Interrater reliability for the effect size was ICC = .89, 95% CI [.84, .92] and for the sample size and publication year ICC = 1, 95% CI [1, 1]. The Cohen’s κ for the remaining categorical variables was .91, overall indicating strong to excellent intercoder agreement (LeBreton & Senter, 2008). All discrepancies were solved upon discussion by comparing extracted data.

Statistical Analyses

Descriptive Statistics

The meta-analysis is based on 41 studies published between 1997 and 2021, predominantly as peer-reviewed articles (73%). The remaining studies appeared as theses (5%) or conference papers (22%). The database covered 69 independent samples that provided 104 effect sizes, with each sample comprising between 1 and 3 effect sizes. Overall, the meta-analysis included scores from 28,986 students (range of samples’ ns: 11–11,195, Mdn = 122). The mean age was 22.21 years, and 53.71% of the students were female, however, only 11 studies reported information on age, and 20 studies information on gender. The duration of the courses varied between 6 and 19 weeks (Mdn = 12 weeks), mainly covering one academic semester. Moreover, courses varied with respect to their format (24% fully online, 72% blended, 1% not reported separately, or 3% missing), emphasis of discussion (18% instructed use of discussion boards, 69% discussion boards available within the LMS without further instructions, 10% not mentioned, or 3% missing), and requirements (45% online activities within the LMS as part of grading, 54% none, or 1% missing). In 44% of the cases, general online activity was operationalized as total login time and 56% as login frequency (number of single logins or number of days with at least one login).

Overall Pooled Correlation

In total, the three-level random-effects meta-analysis identified a pooled correlation of r = .25 p = .003, 95% CI [.09, .41], indicating that students who are more active online also have a better learning outcome (Figure 1). The result of the pooled correlation was robust and replicated in the separate moderator analyses (Table 1).

Table 1 Meta-analysis on general online activity and learning outcome and separate moderator analyses

	k1	k2	N	r	SEr	95% CI	QM	Q	σ2(2)	σ2(3)	I2(2)	I2(3)	80% PI
Note. k1 = Number of effect sizes; k2 = Number of samples; r = Pooled correlation; SEr = Standard error of r; 95% CI = 95% confidence interval of r; Q = test of heterogeneity (df = k1 −1); QM = test statistic for the omnibus test of coefficients (df = 1); σ2(2) = Random effect of r between samples; σ2(3) = random effect of r within samples, I2(2) = Proportion of between-cluster heterogeneity; I2(3) = proportion of within-cluster heterogeneity; 80% PI = 80% prediction interval of r. aIncludes the categories of discussion board usage available and not mentioned. bFor illustrative purposes, subgroup analyses are reported for older versus newer studies based on a median split. *p < .05.
Overall	104	69	28,986	.25*	.08	[.09, .41]		3,960.04*	.06	< .01	.92	.05	[−.10, .59]
Course format						[−.27, .18]	0.16
Fully online	25	20	4,816	.32*	.08	[.17, .48]		125.03*	.04	< .01	.91	< .01	[.05, .60]
Blended	75	46	12,543	.28*	.07	[.15, .41]		2,919.10*	.07	< .01	.93	.04	[−.08, .63]
Discussion board usage						[−.20, .27]	0.07
Instructed	19	14	1,077	.28*	.08	[.11, .44]		112.98*	.06	< .01	.86	< .01	[−.05, 61]
Not instructeda	82	52	16,475	.29*	.06	[.17, .40]		3,052.80*	.06	< .01	.91	.06	[−.05, .63]
Requirements						[−.22, .20]	0.01
Graded activities	47	35	7,673	.29*	.10	[.10, .48]		2,313.94*	.10	< .01	.95	.03	[−.13, .72]
None	56	33	10,118	.28*	.05	[.19, .37]		420.52*	.03	< .01	.80	.10	[.06, .51]
General online activity						[−.29, .18]	0.21
Total login time	46	46	11,825	.29*	.07	[.16, .40]		2,320.63*	.04	.04	.49	.49	[−.09, .66]
Login frequency	58	53	24,842	.23*	.07	[.09, .37]		585.41*	.02	< .01	.81	.11	[.01, .45]
Publication yearb						[−.03, .05]	0.23
Older than 2017	43	31	19,860	.22*	.09	[.04, .40]		501.64*	.03	< .01	.88	.07	[−.05, .49]
Newer than 2017	61	38	9,126	.29*	.08	[.14, .44]		2,495.45*	.08	< .01	.93	.05	[−.09, .67]

Table 1 Meta-analysis on general online activity and learning outcome and separate moderator analyses

View as image HTML

Overall, these findings indicate a small but statistically significant positive association between general online activity and learning outcomes. Yet, the high random variance resulted in an exceedingly large prediction interval around the pooled effect, 80% PI [−.10, .59]. Hence, we conducted sensitivity analyses to examine the impact of certain studies on the prediction interval. Further, the studies showed higher between-cluster heterogeneity (I² = .92; see also Figure 1 for an illustration of the variability between samples) compared to within-cluster-heterogeneity (I² = .05), indicating pronounced unaccounted differences between samples that might be explained by moderator analyses, but negligible variability within samples (Higgins et al., 2003).

Moderator Analyses

We conducted meta-regression analyses to examine the effects of course format, emphasis of discussion, requirements, operationalization of general online activity, and publication year on the pooled effect (Table 2). On effect size level, correlations between moderators ranged from −.54 to .46, indicating negligible multicollinearity. None of the moderators was statistically significant. This result remained the same even when each moderator was examined separately (Table 1). Overall, moderator analyses showed no effect, indicating that our data do not provide evidence in favor of a moderating effect of course format, emphasis of discussion, requirements, operationalization of general online activity, or publication year on the relationship between general online activity and learning outcome.

Table 2 Moderator analysis including all five moderator variables simultaneously

	Moderator analysis			Correlations
	γ	SEγ	z	(1)	(2)	(3)	(4)
Note. Phi coefficients for dichotomous moderator variables and point-biserial coefficients for dichotomous and metric moderator variables on effect size level are displayed. The correlations are based on 100–104 effect sizes. γ = Fixed effects regression weight; SEγ = Standard error of γ; QM = test statistic for the omnibus test of coefficients (df = 5); σ2(2) = Random effect of r between samples; σ2(3) = Random effect of r within samples; k1 = Number of effect sizes; k2 = Number of samples.
Intercept	−5.55	25.28	−0.22
(1) Course format (1 = blended; 0 = fully online)	−0.07	0.13	−0.50
(2) Discussion board usage (1 = instructed; 0 = available, not mentioned)	−0.04	0.13	−0.28	−.54
(3) Requirements (1 = none; 0 = graded activities)	−0.03	0.10	−0.25	.12	−.27
(4) General online activity (1 = login frequency; 0 = total login time)	0.01	0.05	0.12	−.16	.12	.12
(5) Publication year (metric)	< 0.01	0.01	0.23	.46	−.45	.12	−.28
QM	0.31
σ2(2)/σ2(3)	0.06/< 0.01
k1/k2	100/66

Table 2 Moderator analysis including all five moderator variables simultaneously

View as image HTML

Sensitivity Analyses

We performed sensitivity analyses to examine the robustness of our findings. In a first step, the robustness of the presented results was investigated by removing nine extreme correlations (i.e., outliers with z > 1.96; Viechtbauer & Cheung, 2010; Figure S2) from the database to compare this pooled effect to the original pooled effect. After eliminating these effects from the database, the pooled effect was r = .24, p < .001. The 80% PI decreased from [−.10, .59] to [.04, .43], indicating a reduced random variance. However, the outliers did not distort the pooled effect. Similar patterns appeared for all subgroup analyses (see Table S2). Overall, the outlier analyses provided evidence for the robustness of the correlation between general online activity and learning outcomes. In a second step, sensitivity analyses with respect to two studies that differ in their conceptualizations from the other included studies resulted in negligible differences: r = .29, p < .001, 80% PI [−.05, .62] (Lauría et al., 2012), and r = .24, p = .011, 80% PI [−.11, .60] (Mödritscher et al., 2013), also indicating the robustness of the present meta-analysis (see Table S3).

Publication Bias

First, the meta-regression analysis that we conducted to examine publication bias indicated no statistically significant difference between effect sizes extracted from peer-reviewed versus other sources (γ = −0.06, SE = 0.13, p = .624). Second, the modified regression test for asymmetry (γ = 7.93, SE = 8.57, p = .355) revealed no statistically significant effect for measurement precision. Overall, we did not find evidence of publication bias.

Limitations of the Included Moderators

First, our potential moderator variables were restricted to broad course characteristics, which can be illustrated by the variable course format. The dichotomous classification of blended versus fully online format might be too coarse as there exist flowing transitions depending on the portion of content delivered online (Allen & Seaman, 2014) and the share of online elements in a certain course might be better depicted as a continuous variable rather than a dichotomous one. However, most studies on online courses only provide very superficial characteristics. Given the fact that faculties struggle with the transition to online teaching (Kebritchi et al., 2017), more evidence is needed so practitioners who design online courses are able to make informed choices to improve quality of online learning.

Second, there is a lack of information on contextual variables (e.g., instructional design, interactive tools, or synchronicity) reported in primary studies. Varying contexts and tools affect the learning process by providing different learning opportunities which are decisive for improved learning (Lust et al., 2012). The consideration of contextual factors might help explain ambiguous findings in the current literature (Gašević et al., 2016) as they enable more comprehensive moderator analyses. Our moderator analyses were limited to basic information about how the LMS was implemented. In the following, we discuss which aspects might help explain differences between settings.

Potential Other Moderators

An overall structure offered by online courses might help reduce individual differences in online learning behavior, as it provides guidance for students to engage in the most beneficial activities at a certain point of course processing (Winne, 2004). On the individual level, instructors can help to reduce existing heterogeneity within the association between general online activity and learning outcome, as not all learners seem to benefit equally from learning opportunities (Lust et al., 2012). Learners need to be instructed how to use LMS (Kebritchi et al., 2017) to exploit the full potential of online courses. Other forms of structure are, for example, shares of synchronous methods and applications in online courses (e.g., teaching sessions, collaborative learning, or support and monitoring by a tutor; see Kinshuk & Chen, 2006), which provide a structured schedule for learning behavior, or any form of online assessment across the course duration to monitor students’ learning progress or to provide personalized feedback (Knight, 2020), or to encourage students’ engagement (Tempelaar et al., 2019). In the present literature, systematic information on the extent of the structuredness of online courses is unfortunately limited. Future research might document the effects of these course characteristics and their impact on learning behavior.

Another aspect of online course design comprises the incentives that are used to ensure students’ participation. Apart from the extent to which participation is included in course grading, only little is known about how instructors use incentives for constant participation throughout the course. One example for these incentives is gamification – the implementation of game-design principles and elements in non-game environments (Deterding et al., 2011) – which can promote motivation (see Mora et al., 2017 for a review) for example by providing visualized immediate feedback to the learner on goal completion or students’ learning progress compared to other students. Gamification for educational purposes can be associated with increased activity (e.g., Hamari, 2017; Huang & Hew, 2015) or general engagement in online programs (Looyestyn et al., 2017). But it remains unclear if a game-based induced increase in online activity automatically leads to improvement in learning. If practitioners systematically provide online courses with and without different types of gamification, future research could examine differences in the online learning activity and its impact on learning outcomes.

Finally, our meta-analysis was based on log data indicators that specify the extent of total login time or login frequency within LMS over an entire academic semester. Differences in the distribution of online activity across the course duration could not be considered. As distributed learning is a more efficient learning strategy than cramming before examinations for an equal amount of time (Dunn et al., 2013), future research should address the mere amount of activity and the distribution of total login time or logins in a more fine-grained way. Additionally, students’ diversity and consistency of online activities might provide substantial insights into how students’ activity affects learning (Lust et al., 2012).

Log Data in Educational Research

The present meta-analysis can be seen as a starting point of how log data can be used to contribute to our understanding of complex variables like academic achievement by linking course outcomes to broad log data indicators of online learning behavior. However, given the increase of online education in higher education and the recent technological development (Kebritchi et al., 2017) there undoubtedly exist more fine-grained data in research than overall participation measures comprising platform usage. One big advantage of the use of log data is that large amounts of data are easily and immediately accessible so that learning analytics can draw on detailed and extensive log data about learners’ studying activities within modern software tools (Winne, 2010). Although log data are a more accurate reflection of the quantity of media use than self-reports (Parry et al., 2021), researchers’ degrees of freedom in data tracking, collection and analysis persist and thereby limiting the objectiveness of log data indicators (Avella et al., 2016). Moreover, the biggest issue is the connection with existing educational theories and the resulting necessity to consider the reliability of log data as well as its role in claims about validity based on this kind of data (Winne, 2020).

But how can learning analytics meet the vision to help improving learning and teaching by using generated data as people engage in learning? On the student-centered level, learning analytics facilitate predictive modeling of course completion (Clow, 2013). Information on student’s previous educational experience and demographics, as well as data on online activity and formative and summative assessment, are combined and then used to develop interventions designed to improve retention and performance. This enables early intervention systems and personalized learning, as students receive real-time feedback on their learning progress (Arnold & Pistilli, 2012). Otherwise, instructors can take advantage of learning analytics by using them to identify areas in need of improvements regarding the curriculum as well as their own performance (Avella et al., 2016). Finally, the implementation of new tools or mechanisms can be checked (Song, 2018). Multiple types of learning tools enhance the learning experience (Hathaway, 2014), but it is not always the more, the better. Instructors have to consider which tool, design element, or multimedia will add to the learning process and which ones are distracting (Kebritchi et al., 2017).

Limitations of the Present Study and Implications for Future Research

Research on learning analytics is a promising approach for an advanced understanding of the learning process (Gašević et al., 2015). This meta-analysis provides an initial insight into the value of broad log data indicators of learning behavior. However, the present analyses come with limitations. Specifically, recent developments in meta-analytic methods suggest that it might be more adequate to model the hierarchical structure of the data by including the covariances of effect sizes derived from the same sample in the model, rather than using the default model, which assumes no covariances. On a more general stance, data dependency is an important issue in meta-analytic research that is often neglected (e.g., Rodgers & Pustejovsky, 2021). While these advanced models might be even better suited to model the data structures, many issues arise not from inappropriate modeling choices but rather from shortcomings in the primary studies. How should future research look like in order to contribute to a more advanced understanding of online learning?

While the number of studies using log data increases steadily, only a few of these studies transparently describe their methodologies for data collection and cleaning, utilized measures or analyses (Bergdahl et al., 2020). In general, learning analytics has to face challenges of heterogeneous data sources and the lack of unified vocabulary (Papamitsiou & Economides, 2014). Future meta-analyses could make use of quality assessments of primary studies, something that is already common for assessing the methodological quality of intervention effectiveness research (e.g., Valentine & Cooper, 2008). Scheffel and colleagues (2014) have proposed a framework of quality indicators for learning analytics earlier on. Future meta-analyses would benefit from a standardized procedure that allows taking the methodological quality of learning analytics studies into account for weighting procedures as well as the decision whether to include or exclude a primary study. However, learning analytics can benefit from a unified framework for the use of terms and definitions, operationalizations, and methodological procedures.

Another promising development to overcome central issues (i.e., data access and transparency) in meta-analyses is the open data movement (Gurevitch et al., 2018). As soon as researchers follow standards regarding an open scientific process, design standards would reduce unclearly reported methodologies, and data sharing standards would enable to directly generate effect sizes from open data (Nosek et al., 2015).

Moreover, open science practices facilitate multi-level analyses based on raw data in the form of meta-analysis of individual participant data (IPD; e.g., Riley et al., 2010). IPD meta-analyses are considered the gold standard as they prevent aggregation biases and enable to look at the impact of heterogeneity that originates from differences within studies (Kaufmann et al., 2016). Due to the COVID-19 pandemic, educational institutions were forced to shift teaching to online learning (De’ et al., 2020), potentially with an increase in the usage of interactive tools (such as video conferencing tools) that could facilitate online discussions or group work, which may also lead to an increase in the quality of online courses. Hopefully, these changes will be documented in forthcoming research on online courses. Future research might comprise large collaborations and centrally coordinated data collections within online courses to benefit from the incoming data due to the digital surge so that they might gain deeper insights to improve the quality of online learning.

Apart from that, this review focused on formal higher education. However, informal learning gains more attention in the field of educational research (Zheng et al., 2019), and therefore is a promising extension. Especially the change toward online learning promotes informal learning (i.e., learner-directed and independent learning outside of formal educational contexts; Song & Bonk, 2016). However, due to the absence of external assessment within informal learning (Callanan et al., 2011), studies with an informal context could not be included in the present meta-analysis. Even though formal and informal learning lead to gains in knowledge and skills (Cerasoli et al., 2018), it is difficult to combine them in meta-analyses as their outcomes are operationalized differently since for informal learning, educational success is traditionally defined as course completion (Henderikx et al., 2017). Accordingly, it seems worthwhile to examine whether the positive association between general online activities and learning outcomes can be transferred to an informal context.

Conclusion

In summary, we identified an association between broad log data indicators of general online activity and learning outcomes. Although several sensitivity analyses indicated the robustness of the present meta-analysis, the high heterogeneity between studies could not be explained by our moderator variables, which were limited to basic information on course implementation. We recommend for future research to form bigger collaborations and centrally collect data to conduct IPD meta-analyses to gain deeper insight into online learning. Learning analytics have the potential to provide more fine-grained data, but it is necessary to connect generated data to existing educational theories.