New ways in fostering self-regulated learning at university: How effective are web-based courses when compared to regular attendance-based courses?
Abstract
Abstract. Self-regulated learning is essential for studying successfully at university. However, students often show deficits in their ability to learn in a self-regulated way. Consequently, it has become crucial to foster students' self-regulated learning at university. The effectiveness of such courses has primarily been investigated in regular class contexts that require physical attendance. However, web-based course formats are currently gaining in importance. Web-based courses have several advantages (e. g., that students can decide when and where they want to study). The question of whether a web-based course is as effective as an attendance-based one has yet to be answered. In a randomized intervention study (N = 186 university students) with two different treatments (attendance-based and web-based courses), it was investigated whether students in the web-based format profited to the same extent as students in the attendance-based course. Kirkpatrick's model was implemented for evaluation. The results showed that the students were very satisfied with both course formats, self-regulated learning was considered useful for studying, and the subjective and objective increases in learning were high. Furthermore, the results showed that self-regulated learning can be fostered in the web-based course as effectively as in the attendance-based course.
Zusammenfassung. Die Fähigkeit zum selbstregulierten Lernen ist eine wichtige Voraussetzung für akademischen Erfolg. Gleichzeitig gibt es Hinweise darauf, dass viele Studentinnen und Studenten deutliche Defizite im Bereich des selbstregulierten Lernens aufweisen. Die Förderung der Selbstregulationskompetenz durch geeignete Trainingsmaßnahmen stellt somit einen wichtigen Aspekt der universitären Lehre dar. Bislang wurde die Wirksamkeit von Seminaren zum selbstregulierten Lernen insbesondere für Präsenzveranstaltungen gezeigt. Gegenwärtig gewinnt die Implementation von E-Learning Formaten in der universitären Lehre zunehmend an Bedeutung. Ein Vergleich mit Präsenzveranstaltungen zur Vermittlung des selbstregulierten Lernens steht bislang noch aus. In einer randomisierten Interventionsstudie (N = 186 Studentinnen und Studenten) wurde untersucht, ob Studentinnen und Studenten in einer web-basierten Intervention gleichermaßen profitieren wie Studentinnen und Studenten in einer Präsenzveranstaltung. Zur Evaluation wurde Kirkpatricks Modell herangezogen. Die Ergebnisse zeigen, dass die Zufriedenheit der Studentinnen und Studenten mit beiden Kursen sehr hoch ist, selbstreguliertes Lernen als sehr hilfreich für das Studium angesehen wird und die Studentinnen und Studenten ihre Selbstregulationsstrategien und Wissen über selbstreguliertes Lernen verbessern. Die Ergebnisse weisen darauf hin, dass selbstreguliertes Lernen in Präsenz- sowie in E-Learning-Kursen gleichermaßen effektiv gefördert werden kann.
Self-regulated learning (SRL) is an essential condition for studying successfully at university (e. g., Bellhäuser, Lösch, Winter & Schmitz, 2016; Kitsantas, Winsler & Huie, 2008; Schunk & Ertmer, 2000). However, students often show deficits in their knowledge about SRL and their ability to learn in a self-regulated way (Peverly, Brobst, Graham & Shaw, 2003; Randi & Corno, 2000; Stark & Mandl, 2005). Consequently, it has become crucial to foster SRL in university students. Courses for fostering self-regulation competence have been shown to be effective, as has been indicated by both self-reported and objective measures of the learning process (e. g., Benz, 2010; Dörrenbächer & Perels, 2016; Masui & De Corte, 2005). The effectiveness of these courses at universities has primarily been investigated within regular class contexts that require physical attendance. However, web-based course formats are becoming more important (Benz, 2010; Waheed, Kaur & Kumar, 2016). They offer many properties that make them attractive, for example, that physical attendance is not required and that students can choose their preferred study times. Web-based course formats allow for higher levels of autonomy and self-determination because they enable students to learn the contents of a course at any time and from anywhere (e. g., Shachar & Neumann, 2003). The question of whether web-based course formats are as effective as regular attendance-based courses has yet to find a clear answer. Therefore, the aim of this study is to compare the effectiveness of a web-based course format with that of an attendance-based course format.
Process model of SRL
Several popular SRL models (e. g., Boekaerts, 1999; Schmitz & Wiese, 2006; Zimmermann & Schunk, 2011) have postulated that a learner's motivation, cognition, and metacognition are crucial for successful learning. This study refers to Schmitz and Wiese›s (2006) SRL model, which is an adaptation of Zimmerman›s (2000) three-phase cyclical SRL model, but specified for a concrete situation, namely learning, and in which SRL is exclusively defined as a process of learning states. Its phase structure can serve as the basis for the training structure, which can be divided according to the phases. Moreover, several other effective training programs have been based on this model (e. g., Bellhäuser et al., 2016; Perels, Otto, Landmann, Hertel & Schmitz, 2007). In line with Zimmerman (2000), Schmitz and Wiese (2006) described SRL as a process that comprises a cyclical sequence of three phases: the preaction, action, and postaction phases. The preaction phase is defined by preparing to learn. The learner defines goals that facilitate the evaluation of the learning outcome in the future process. Goal-setting is influenced by situational demands and the given task. Motivation is another key concept of self-regulated learning (Perels et al., 2007; Schmitz & Wiese, 2006). On the basis of these components of the preaction phase, the learner chooses strategies and plans his actions in order to achieve his goals. In the action phase, the main learning takes place. The learner implements his chosen strategies and controls his actions. Self-monitoring, understood as the observation of one's actions (Zimmerman, 2000), is therefore important during the action phase. With self-monitoring, the learner's actual performance can be checked, and it can influence self-regulation (Schmitz & Wiese, 2006). The postaction phase is defined by the reflection and evaluation of the learning process and learning outcomes. Based on this, consequences for further learning can be derived, and in due course, strategies or goals can be adapted for the following learning cycle. This can be further influenced by the attribution of the learning outcomes and the frame of reference (Abramson, Dykman & Needles, 1991; Rheinberg & Fries, 2010). All variables in one phase are both affected by previous learning phases and predictive of the subsequent learning process (Perels et al., 2007). Therefore, the postaction phase of one learning cycle influences the preaction phase of the next cycle (for a detailed description, see Schmitz & Wiese, 2006).
SRL trainings
Research has shown that students in various age groups can be successfully trained in SRL and that the acquired skills are associated with improved academic achievement. Most of these interventions were conducted in a school context and required physical attendance (e. g., Dignath & Büttner, 2008; Ferreira & Simão, 2012; Perels, Gürtler & Schmitz, 2005; Randi & Corno, 2000; Reid & Borkowski, 1987; Werth et al., 2012; Zimmerman, 1990). Nonetheless, research has shown that adults can also be trained in SRL. For example, Benz (2010) conducted a meta-analysis that revealed that 19 – 37-year-old learners indeed profited from SRL interventions. Another meta-analysis (Hattie, Biggs & Purdie, 1996), which also included samples of university students, showed that interventions focusing on SRL skills (e. g., time management, motivation) were successful in enhancing learning. Nevertheless, the samples of university students showed smaller performance increases than the younger age groups. The authors assumed a ceiling effect. However, the university students benefitted more from a positive attitude toward learning. Another study (Zhu, Au & Yates, 2016) showed that university students' SRL and self-control, which were assessed at the beginning of a blended-learning course, predicted students' learning outcomes at the end of the course. The influence of the students' self-control was mediated through their use of self-regulated learning and online course participation. The authors concluded that students have to be able to use self-regulated learning strategies in order to learn more effectively, especially in online learning environments. Therefore, they recommended that teachers support their students in building these strategies, for example, through SRL training. Moreover, a study by Masui and De Corte (2005) emphasized that training programs that included both instructional conditions (for imparting knowledge) and practice conditions (for applying the content that was learned) are effective in helping students improve their SRL skills.
Web-based interventions
Nowadays, due to digitalization, new learning formats such as web-based learning arise far more often than only in work-related contexts (Bundesministerium für Arbeit und Soziales, 2017; Kattoua, Al-Lozi & Alrowwad, 2016; Mayer, 2017).
This educational trend can also be observed in the university context (e. g., Matuga, 2009; Waheed et al., 2016). Web-based courses for students have many advantages (e. g., Kattoua et al., 2016; Nedeva & Dimova, 2010). For example, the format allows flexibility with regard to where and when the learning occurs and can be adapted to users' abilities (e. g., DeWolfe Waddill, 2006; Fariborzi & Bakar, 2010; Nedeva & Dimova, 2010). Moreover, students become familiar with a learning format that is commonly used in work-related contexts. Students are supported by the fact that they get to learn in an environment with a high degree of freedom, preparing them for their study programs and future professional careers. However, literature (e. g., Margaryan, Bianco & Littlejohn, 2014) shows that in particular massive open online courses (MOOCs), a special form of a web-based course, which is offered to a large amount of students and not only to a selected group within a university course, lack instructional quality. The authors analyzed 76 MOOCs with the course scan questionnaire, which includes a set of principles of instruction (e. g., problem-centred, activation, application). The results showed that most courses scored low.
Mayer (2017) describes twelve different principles that can be used in order to design e-learning materials to facilitate academic learning. In particular, implementing the multimedia-principle in e-learning, meaning that words (spoken or printed) and pictures (static or dynamic) should be presented together rather than alone, yields in better student learning (Mayer, 2003; 2017). Videos or animations with accompanying narration or onscreen text and computer-based interactive games including spoken or printed texts are common examples of using multimedia in e-learning (Mayer, 2017). Furthermore, on the basis of cognitive theory of multimedia learning the author describes how to create effective multimedia instructional messages because not any combination of words and pictures is equally effective. He states three instructional goals with corresponding research-based techniques: reduce extraneous processing (e. g., redundancy) manage essential processing (e. g., segmenting), and foster generative processing (e. g., personalization). In earlier research Mayer (2003) already showed that four instructional design methods (multimedia, coherence, spatial contiguity, personalization) have equal effects in book-based and computer-based environments. He concluded that instructional design methods that are effective in one media environment also promote learning in other environments as long as the same kinds of cognitive processing are promoted and the method is not unique to one media. Clark, Tanner-Smith and Killingsworth (2016) also emphasize in their review about the medium of digital games and learning the important role of the design of an intervention. They conclude that it is not only the environment, but the design within the medium that determines the efficacy of a learning environment.
Mayer (2017) presents a research agenda that requests to further examine using multimedia in e-learning in regard to, for example, long-term effects, replication of material with new learners and more realistic learning environments in order to, on the one hand, share results with instructional designers and, on the other hand, to add to theories of learning.
Web-based courses fostering SRL have thus far received positive evaluations (e. g., Bellhäuser et al., 2016; Cranwell et al., 2014; Feng & Chen, 2014; Hu, 2007; Kauffmann, Ge, Xie & Chen, 2008; Tsai, Shen & Tsai, 2011). The web-based course developed by Bellhäuser et al. (2016) was based on Schmitz and Wiese›s (2006) model. It meets the multimedia criteria presented by Mayer (2017; e. g., videos and animations with spoken texts) and by Margaryan et al. (2014; e. g., application) and was tested in an authentic learning environment. In a randomized controlled evaluation study with 211 university students, they found that training had significant effects on SRL knowledge, SRL behavior, and self-efficacy. Moreover, the participants gave high ratings to the usefulness of the SRL strategies and the quality of the web-based training program. Bellhäuser et al. (2016), who also used the web-based training program from this study reported that the students were very satisfied with the program.
However, the question of whether a web-based course is as effective at fostering SRL as an attendance-based course has yet to find a clear answer. So far, researchers have tried to examine whether web-based or distance education courses are as effective as attendance-based ones by using different criteria (e. g., attitude and achievement outcomes). However, the research findings have been ambiguous. Allen, Bourhis, Burell and Mabry›s (2002) meta-analysis included studies that compared students' satisfaction in traditional face-to-face courses versus distance education courses, which included three channels of communication (writing, audio, and video). The findings indicated that students showed a slight preference for a traditional classroom environment. However, the authors stated that satisfaction provided only one possible source of evaluation and should be complemented by other sources of evaluations. Shachar and Neumann (2003) concentrated on an objective dimension of effectiveness. In their meta-analysis, they explored the question of whether there is a difference in students' final academic performance in distance education programs compared with traditional face-to-face programs. They used learning outcome data from 86 studies and showed that distance learning outperformed classroom instruction because in 66 percent of the studies, the final academic performance grades of students in distance education programs were higher than those enrolled in traditional programs. In another meta-analysis, the effectiveness of web-based instruction and classroom instruction for teaching declarative and procedural knowledge as well as reactions to the courses were examined (Sitzmann, Kraiger, Stewart & Wisher, 2006). Participants of the 96 studies were college students or employees who acquired knowledge or skills in workplace training programs. It was shown that web-based instruction was more effective than classroom instruction for teaching declarative knowledge but equally effective for teaching procedural knowledge, and participants were equally satisfied with both delivery media.
In another meta-analysis, Bernard et al. (2004) compared classroom instruction and distance learning regarding achievement, attitude, and retention. They found great variability in effect sizes on all measures, indicating that distance education can surpass classroom instruction and that it can be less effective concerning these measures. Moreover, they criticized the quality of the literature because it lacked design features. The biggest problem that arose from comparing two learning formats was found in the lack of ecological validity (Matuga, 2009). The literature comparing the two formats often lacked a detailed description of the methodology and the contents of the formats (e. g., Bernard et al., 2004). Therefore, to contribute to this research gap, we compared the web-based course developed by Bellhäuser et al. (2016) with an attendance-based course.
Research questions and hypotheses
The aim of the present study was to analyze the effectiveness of a web-based course in comparison with an attendance-based course – differing only in the format – to provide new insights into the effectiveness of web-based courses. For this purpose, the web-based course fostering SRL developed by Bellhäuser et al. (2016) was used because it had already been approved and positively evaluated. For comparison, a parallelized attendance-based course was implemented. For the evaluation, Kirkpatrick's well-established model served as the framework (Kirkpatrick, 1979), which differentiates between four levels: reaction, learning, behavior, and results. The reaction level measures the acceptance of the training. At the learning level, the increase in participants' learning is measured; at the behavior level, the extent to which the participants adapted their behavior on the basis of the course is evaluated; and at the result level, the impact of the course on the participants' institution becomes visible. In this study, an evaluation of the reaction, learning, and behavior levels was realized.
It is expected that both course formats would be evaluated positively. Because the two courses differed only in their format, and research (e. g., Sitzmann et al., 2006) has shown that attendance-based and web-based formats are both effective in teaching declarative and in particular procedural knowledge, no differences in their general effectiveness were expected. Thus, the current study postulated the following research questions and hypotheses:
Research question 1: How do students rate the two course formats?
Hypothesis 1: Students in both course formats are satisfied with the courses and rate SRL as useful. Students' ratings of the two courses are equally positive.
Research question 2: Is the web-based course format as effective as the attendance-based course format?
Hypothesis 2a: Students in both course formats show improvements in their SRL strategies and declarative metacognitive knowledge on SRL over the semester. They do not differ in their SRL strategies and declarative metacognitive knowledge at posttests.
Hypothesis 2b: Students in both course formats report a perceived increase in SRL competence at the end of the semester and do not differ in their evaluation.
Methods
Design
A randomized intervention study was conducted. It was announced that students could register for a course with the title “Self-Regulated Learning”. Students were randomly assigned to two course formats: attendance-based and web-based. Because the effectiveness of the web-based format had previously been tested against a control group with no intervention, this study investigated only two different training conditions here. Three assessment points were scheduled. The pretest (t1) was conducted before the actual course program started. The first posttest (t2) was conducted after the training units and the second posttest (t3) was conducted after the implementation phase at the end of the semester. Students' background variables, their course evaluation, and their self-regulatory skills were assessed, and a declarative metacognitive knowledge test was implemented. All data were collected with online questionnaires.
Participants
Participants were recruited from educational science and teacher education programs at a German university. 186 students took part in the pretest session, after which they were randomly assigned to either the attendance-based group (n = 91) or the web-based group (n = 95). 184 participants took part in the posttest assessment (attendance-based group: n = 90, web-based group: n = 94), and 171 participants took part in the second posttest (attendance-based group: n = 81, web-based group: n = 90).
Due to incomplete data, however, some participants had to be excluded, leaving a final N of 162 for the analyses (12.90 % dropout). Attrition analyses revealed that the excluded individuals did not differ from the participants in the final sample concerning the effects of the demographic variables (gender, study time, GPA) and the pretest variables (SRL preaction, action, and postaction phases and declarative metacognitive knowledge) on SRL (Wilks' λ = .94, p = .51).
In the final sample of N = 162 participants (n = 38 male, n = 121 female; n = 3 unspecified; mean age: 23.70 years, SD = 3.13, Range = 18 – 44), the attendance-based group consisted of n = 77 students (n = 16 male, n = 60 female, n = 1 unspecified; mean age: 24.01 years, SD = 3.61, Range = 18 – 44), and the web-based group consisted of n = 85 students (n = 22 male, n = 61 female, n = 2 unspecified; mean age: 23.41 years, SD = 2.62, Range = 19 – 34). Their GPA was 2.23 (SD = 0.62, Range = 1.0 – 3.8) in the whole sample (attendance-based group: 2.29 (SD = 0.66, Range = 1.0 – 3.8), web-based group: 2.18 (SD = 0.58, Range = 1.0 – 3.2)). Participants had studied on average for 7.07 semesters (SD = 3.79, Range = 1 – 20) in the whole sample (attendance-based group: 7.16 semesters (SD = 3.88, Range = 1 – 17), web-based group: 6.99 semesters (SD = 3.72, Range = 1 – 20)). A MANOVA was computed to analyze differences in demographic and dependent variables between the groups (randomization check). The alpha level was set to .20 in order to test H0 and thereby minimize the Type II error rate (Bortz, 1999). The multivariate effect (Wilks' λ = .95, p = .73) was not significant. However, there were significant effects of the SRL postaction phase (F (1,93) = 2.17, p = .15) and gender (F (1,93) = 1.97, p = .16), but these effects were not interpreted because the multivariate effect was not significant.
Procedure
The students who enrolled in the SRL course were invited to attend the first meeting in the lecture hall where they received an overview of the course requirements and modalities. Participation in our study was voluntary. After this meeting, the students were randomly assigned to the different courses. They were informed via e-mail about their assignment and asked to complete the pretest. All tests were conducted online. After five weeks, when SRL training was completed, the students took the first posttest and started working on a transfer task, which was required to pass the course. After four weeks, the students presented their work in a round-table discussion during an attendance-based session. At the end of the semester, the second posttest was administered, and the students met for the final session where they had the opportunity to give feedback on their experience with the course formats.
Course program
In this study, the web-based training developed and evaluated by Bellhäuser et al. (2016), aimed at supporting students' SRL at university. The training concept was based on the process model by Schmitz and Wiese (2006), consisting of preaction, action, and postaction phases. Each 90-minute unit was released with an interval of one week. Participants acquire cognitive, metacognitive, and motivational learning strategies. The course was provided in an online learning platform (moodle), and the content was transmitted through different media, for example, videos, power point presentations, interactive exercises, and discussion forums. An attendance-based course that followed the same theoretical concept was implemented. The number of units, their duration, structure, and contents, as well as the teaching and learning methods were all identical. The only difference was in the format: web-based versus attendance-based. For example, the students in the web-based course had the opportunity to discuss questions with other students in a discussion forum or ask questions via e-mail, whereas students in the attendance-based course could discuss their questions in the classroom. Both groups had access to the course material after the materials were first released.
Unit 1 (Introduction, self-regulation model, goal-setting, time-management). The first unit is about the preaction phase and provides an overview of the course and its relevance. An introduction to the process model (Schmitz & Wiese, 2006) illustrated that all learning phases are relevant for successful learning. A further component of this unit is goal-setting (Doran, 1981; Kozlowski & Bell, 2006; Locke & Latham, 1990). Then the students come to understand and apply the SMART technique (Doran, 1981). This technique describes how effective goals are defined and divided into the following categories: specific, measurable, assignable, realistic, and time-related. The final aspects of this unit are planning and time-management. The participants identify time thieves and get to know a technique for effective time-management: the ALPEN-technique (Seiwert, 2004). This technique can be used to draw a timetable that involves the tasks with their durations, buffer time, and a subsequent check.
Unit 2 (procrastination, distractions, volition, learning strategies). The second unit deals with the action phase. The main focus lies in distractions that occur during the learning process and the learning strategies. The unit begins with a video that introduces procrastination (Tice & Baumeister, 1997). The participants reflect on their own procrastination and develop strategies to avoid it. The next section discusses distractions. In order to show the consequences of distractions, participants are asked to do the word color stroop task with and without loud noises. The comparison of the test results shows that participants perform better in the condition without noise. Afterwards, they are given more facts about two types of distractions: inner and outer distractions. Participants then receive tips on how to deal with distractions (e. g., turn off mobile phone). The third section of this unit deals with volition (Kuhl & Fuhrmann, 1998; Schmitz & Wiese, 2006) by illustrating that people can actively influence their volition. The last section deals with cognitive, metacognitive, and resource-oriented learning strategies (Bellhäuser et al., 2016; Pintrich, Smith, Garcia & McKeachie, 1991; Schmitz & Wiese, 2006; Wild & Schiefele, 1994).
Unit 3 (attribution, frame of reference, reflection, motivation). The third unit is about the postaction phase and emphasizes the handling of success, failure, reflection, and motivation. The unit begins with the theory of attribution (Abramson et al., 1991; Peterson & Barret, 1987; Weiner, 1985). In this section, the students are sensitized to different attribution styles and their resulting consequences. For example, in terms of success (e. g., doing well on an exam), the internal attribution style influences future success and motivation in the most adaptive way. The next part is about the frame of reference (Rheinberg & Fries, 2010) that serves as a standard of comparison. It is explained that the individual frame of reference has the best effect on motivation. The next section deals with reflection. The participants formulate three goals: a short-term, a middle-term, and a long-term goal. They are asked to reflect on the short-term and middle-term goals during the next week, focusing on why they have or have not reached their goals. Finally, motivation is addressed as an important factor that influences the whole learning process (Perels et al., 2007; Ryan & Deci, 2000).
The training program was followed by a three-week implementation phase, which required self-regulation skills because the students had to self-structure and work individually on a given task. The students had to choose one aspect of SRL and write a paper about the theoretical concept in APA style and prepare a presentation of their work. For this, they had to conduct a literature review about their chosen SRL aspect and about the formatting style. In this phase, they had to apply the strategies they learned, such as goal-setting, motivation, time management, and reflection on their work. Then, in an attendance-based session, the students had to present their work to their fellow students.
Instruments
Data were assessed via a multimethod approach by employing several self-assessment and objective instruments.
Evaluation scales
Students' evaluations were assessed retrospectively with questions about the SRL course at t3. Students were asked for their opinions about the usefulness of SRL for studying (five-point Likert scale; 1 = not useful at all to 5 = very useful) and about their satisfaction with the course (1 = not satisfied at all to 5 = very satisfied). Additionally, they had to indicate the extent to which they thought they profited from the course regarding the eleven SRL aspects that were part of the course (e. g., “How much competence did you build in the SRL course regarding the following aspects: For comparison, please refer to the beginning of the course. … time-management?”; 1 = no increase at all to 5 = a large increase). They were also asked about their perceptions of the overall increase in their learning (1 = very low to 5 = very high).
Knowledge test
Declarative metacognitive knowledge on SRL was measured with a knowledge test (Butz et al., 2016) at t1, t2, and t3. The questions covered the contents taught in the course and the components of SRL. The test consisted of 14 multiple-choice items with three possible answers – one correct answer and two distractors (e. g., “Please assign the following self-regulation aspect to one component. Reflection: cognitive / metacognitive / motivational component”). Questions are worth 1 point for each correct answer, so a score between 0 and 14 points could be reached. The test showed a good range for difficulty (Pit1 = 0.03 – 0.71, Pit2 = 0.12 – 0.77, Pit3 = 0.14 – 0.71), and the discriminatory power was satisfactory (Dit1 = .09-.67; Dit2 = .06-.83; Dit3 = .10-.87).
Self-regulated learning
Self-regulated learning was assessed at t1, t2, and t3 with twelve items from questionnaires for assessing SRL and learning-strategy application (Bellhäuser, Roth & Schmitz, 2015; Wild & Schiefele, 1994), representing the three phases of Schmitz and Wiese’s (2006) SRL model. The participants were asked to indicate their agreement on a six-point Likert scale ranging from 1 (not true) to 6 (true). The preaction, action, and postaction phases were capured with four items each. The preaction phase scale (Cronbach's α t1 = 0.77, Cronbach's α t2 = 0.75, Cronbach's α t3 = 0.73; e. g., “I check regularly to see if I am still following my goals”), the action phase scale (Cronbach's α t1 = 0.65, Cronbach's α t2 = 0.60, Cronbach's α t3 = 0.62, e. g., “I lack the patience for tasks I would have to do for a long time”), and the postaction phase scale (Cronbach's α t1 = 0.75, Cronbach's α t2 = 0.77, Cronbach's α t3 = 0.77, e. g., “In the evenings, I think about what worked out well and what I want to do differently tomorrow”) showed acceptable to good internal consistencies.
Results
Data were analyzed with MANOVAs and repeated-measures MANOVAs. Furthermore, pairwise tests with Bonferroni corrections were computed for comparisons of the measurement points. Means (M), standard deviations (SD), and intercorrelations between all measures are presented in table 1. For the evaluation, Kirkpatrick's model served as the framework (Kirkpatrick, 1979).
Reaction level: Course evaluations
In order to answer research question one and to measure the impact on the reaction level (Kirkpatrick, 1979), the students rated their satisfaction with their course and the usefulness of SRL for studying. All students' (N = 146) average rating of their satisfaction was 3.93 (SD = .76; nattendance-based group = 70, M = 3.90, SD = .82; nweb-based group = 76, M = 3.96, SD = .70), and the usefulness of SRL for studying was rated 4.22 (SD = .75; Mattendance-based group = 4.21, SD = .74; Mweb-based group = 4.22, SD = .76). A MANOVA revealed no differences between the groups (Wilks' λ = .95, p = .94). The alpha level was set to .20 because H0 was tested to minimize the Type II error rate (Bortz, 1999).
Learning and behavior levels: Training effectiveness
A repeated-measures MANOVA was conducted to test whether participants gained knowledge and whether their SRL strategies improved over the semester (research question 2). The SRL preaction, action, and postaction phase scales as well as declarative metacognitive knowledge on SRL were included as dependent variables. There was a significant main effect of time, indicating changes across the entire sample for the dependent variables (F (8, 136) = 25.89, p = .00, η 2 = .60). The same was true for the univariate effects of every single dependent SRL variable (see table 2). To analyze the time x treatment interaction effect, the alpha level was set to .20 because H0 was tested to minimize the Type II error rate (Bortz, 1999). The overall MANOVA was not significant, indicating that the two groups benefitted from the training equally (F (8, 136) = .78, p = .63, η 2 = .04). The pairwise tests with a Bonferroni correction for the main effect of time revealed a nonsignificant effect of t1-t2 for the SRL preaction phase (p = .79) and a significant effect of t2-t3 (p < .001). The same was true for the SRL action phase, in which the effect of t1-t2 was not significant either (p = 1.00), but the effect of t2-t3 was significant (p < .001). For the SRL postaction phase, the effects of t1-t2 and t2-t3 were both significant (p < .001, p < .001). For declarative metacognitive knowledge on SRL, the tests showed a different picture insofar as the effect of t1-t2 was significant (p < .001), but the effect of t2-t3 was not significant (p = .79).
Furthermore, the students were asked about their perceptions of the development of their SRL competence and knowledge and rated the overall increase in their learning at t3 (table 3). All aspects were rated with a mean value between three and four. Reflection was the aspect with the largest increase in both groups, and social learning had the smallest increase. They also gave high ratings to the overall increase in their learning. To analyze whether the groups differed in their evaluations, a MANOVA was computed and was not significant (Wilks' λ = .95, p = .94). Again, the alpha level was set to .20 because H0 was tested to minimize the Type II error rate (Bortz, 1999).
Discussion
The present study explored the effectiveness of a web-based course format in fostering SRL in comparison with a regular attendance-based format at university. For comparison, a web-based training and a parallelized attendance-based training were used, which differed only in their formats.
Training evaluation
The students gave high ratings for usefulness and were very satisfied with the courses (research question 1). This is an important finding because, although a positive reaction does not ensure learning or transfer, participants' attitudes can influence the success of a course (e. g., Arthur, Bennett, Edens & Bell, 2003; Bergmann & Sonntag, 2006; Burke & Hutchins, 2007; Grossman & Salas, 2011). For example, satisfaction with the training can influence participants' motivation to learn, which is a condition for learning success. Moreover, participants who believe that a training is useful are more likely to apply the skills they learned in it (Burke & Hutchins, 2007; Grossman & Salas, 2011). Therefore, if the findings at the reaction level are negative instead, the recommendation would be to modify the training program (Bergmann & Sonntag, 2006).
At the learning and behavior levels, the evaluation of the main research question (research question 2) was realized, namely, the comparison of the web-based and attendance-based training formats. In order to analyze the effectiveness of both formats, a repeated-measures MANOVA was computed for the pre-posttest comparison. The MANOVA showed a significant main effect of time, indicating positive changes in the SRL preaction, action, and postaction phases and knowledge about SRL from pre- to posttest. Unexpected was that the pairwise tests showed for t1-t2 that only the improvement in the postaction phase was significant, indicating that the training program supported students' reflection skills. Then, regarding t2-t3, the students significantly improved their SRL strategies in the preaction, action, and postaction phases. This finding indicates that the implementation phase, in which the students had to work on a task for which they were asked to apply what they had learned, is crucial for enhancing SRL strategies in all phases. In the implementation phase, the students might have paid more attention to and applied their learning strategies not only during their actual learning but also in preparing and reflecting on their learning, leading to an improvement in strategies in all phases. As opposed to strategy improvement, the significant increase in knowledge took place from t1-t2. This is not surprising because training on the content-related input ended just before t2. Thus, an increase in declarative knowledge comes along with factual input, whereas procedural knowledge and skill development require application to develop. Therefore, one can conclude that leaving room for the application of the trained strategies is crucial for actually building the trained competences. Thus, an implementation phase is crucial for success in training.
Research on training has confirmed that trainees need opportunities to use new learning for a successful transfer (e. g., Burke & Hutchins, 2007; Kauffeld, Lorenzo & Weisweiler, 2012). Leutner and Leopold (2003) demonstrated that strategy training was more successful for the group that had to apply the strategies they had learned than the group that only received training. Learning by itself (i. e., an increase in declarative knowledge) is not enough for training to be considered effective because real success requires changes in performance (Grossman & Salas, 2011). Hence, not only knowledge acquisition but also the application of the material should be included in training programs (Leutner & Leopold, 2003). This is in line with Kirkpatrick's evaluation levels: that is, the behavior level is situated above the learning level.
Furthermore, as expected, the time x treatment interaction effects were not significant, indicating that the students in the two courses showed equal gains in SRL strategies in the preaction, action, and postaction phases as well as declarative metacognitive knowledge about SRL over the semester, even under conservative testing with an alpha level of .20. The students' retrospective evaluation at t3 showed that they also perceived that they gained competence and knowledge in SRL because they gave high ratings to the development of all SRL aspects. The comparison of the groups showed no differences between the groups. The result of the objective metacognitive knowledge test confirmed that not only did the students think they gained knowledge, but they objectively did.
Because web-based courses are increasingly employed, it is important to examine whether this educational medium is effective for teaching knowledge and skills (Sitzmann et al., 2006). The findings of our study suggest that the web-based course as well as the attendance-based course format can be implemented at university in order to support SRL with comparable effectiveness. However, it is important to acknowledge that, like face-to-face instruction, web-based instruction still requires resources in terms of preparation and supervision, and delivery media such as laptops are cost-intensive. Therefore, it should be weighed carefully whether a web-based course or an attendance-based course should be implemented. For example, web-based courses could be offered to students who are not able to attend an attendance-based course. Apart from this, a web-based course to foster SRL could be offered in addition to the regular course program. It could be part of a summer school or offered to students in the introductory phase because SRL is an essential component to learn successfully (e. g., Hattie et al., 1996), and it should therefore be trained as early as possible.
Strengths, limitations, and future research
A strength of the present study is its (quasi-)experimental design. All participants were randomly assigned to the attendance-based or web-based group. Moreover, the study was conducted in a real setting. Another strength of the study is the implementation of a multimethod assessment by combining several self-assessment instruments and an objective measure. Although self-assessments are still the most common way to assess SRL, it is important to mention that it would be desirable to administer more objective measures because self-reports have several limitations (Maag-Merki, Ramseier & Karlen, 2013). The literature has shown that SRL and academic achievement are positively related, but this relation has also been discussed critically (e. g., Valle et al., 2008). Therefore, it is advantageous to collect real performance data (e. g., grades). In doing so, we were also able to analyze whether the intervention had a positive impact on academic performance. Apart from this, situational-judgment tests (SJTs) are promising for overcoming the methodological limitations of self-reports (Maag-Merki et al., 2013). Because SJTs measure the quality of learning strategies, they are able to overcome the limitation of the assessment of quantity applied with self-reports, which are based on the underlying assumption that carrying out more strategies is better (Wirth & Leutner, 2008). SJTs should be integrated into future research because a combination of many types of tests will provide a more realistic picture.
Apart from this, the study used a sample size that was drawn from only one university, and 75 percent of the sample was female. Because this is typical in educational studies, this sample can be considered representative of this field. The study by Bellhäuser et al. (2016) was conducted in mathematically oriented fields of study, and 78 percent of the sample consisted of men. Because the intervention showed similar effects, this indicates that the intervention has similar effects in samples consisting of more men and from different disciplines. However, the current results should be replicated in future studies with larger samples from more universities.
Apart from this, the web-based course had no adaptive features, which means that the students were not able to decide what they wanted to learn, only where and when in a certain time slot of a certain week. In further development, more sophisticated pedagogical approaches should be taken into consideration. For example, when students enter the course with a high prior knowledge or skill level in certain aspects, they should have the opportunity to skip selected contents. Their level of knowledge could be tested with a prior knowledge assessment so that each student can join an individual course program and thereby obtain a starting point from which to approach the course material (Rowe & Rafferty, 2013). This would be an advantage over conventional teaching at university.
Another point concerns the lengths of the sections of both course formats, which were designed for 90-minute sessions. The students who participated in the web-based training program of Bellhäuser et al. (2016) stated that they needed on average 90 minutes for each unit. However, students in the web-based format were free to choose their own study time and pace. Thus, they could divide the sections to study them in parts or repeat certain contents according to their preferences. The students in the attendance-based format could not choose the course time, but, because they had access to the material after the lesson, they also had the chance to repeat the material. Thus, future studies should investigate how much time the students spent learning in order to analyze whether study time influences the outcome variables. It is also possible, however, for students in an attendance-based course to be inattentive or to be focusing on different contents on their smartphones or laptops. For example, in the study by Zhu et al. (2016), the time spent online and the number and length of online contributions were assessed, and the effect of SRL on learning outcomes was in fact mediated by online course participation. Similar effects were found in the study by Imhof and Vollmeyer (2009) who examined the effects of a blended-learning course on SRL. It was shown that the frequency of the use of the electronic learning material was positively correlated with the final grade.
Apart from this, a modification regarding the evaluation of the behavior level and the implementation of the evaluation of the result level could be taken into consideration. The evaluation of the behavior and result levels was more difficult to obtain than the reaction and learning levels because changes in the actual performance and the training's impact on the participants' institution should be visible (Kauffeld et al., 2012; Kirkpatrick, 1979). In this study, the behavior level was gauged at three measurement points. In future research, however, it would be desirable to use an instrument that measures the extent to which participants adapted their behavior while they were actually learning. For example, in weekly reports or in a daily learning diary, the implementation of the strategies could be assessed in a student's learning environment (Schmitz & Wiese, 2006; Zhu et al., 2016). Regarding the result level, in further studies, cohorts with and without SRL training could be compared with respect to performance tests or final grades. It can be expected that the effect on the institutional level in terms of final grades would not be directly measurable after training. However, the evaluation on this level is the most challenging because several contextual factors can influence the institution's output (Kauffeld et al., 2012; Kirkpatrick, 1979). Kirkpatrick (1979) recommended that the reaction, learning, and behavior levels should be evaluated first.
In spite of the described potential improvements, this study met the requirements introduced by Matuga (2009) and Bernard et al. (2004) by showing ecological validity and providing a detailed description of the methodology and content of the formats. Moreover, in this study, delivery media and the instructional methods were not confounded, which was criticized by Clark (1994) because the two courses differed only in their format. The results of this study are also in line with Mayer’s (2017) statement saying that instructional design methods that are effective in one media environment also promote learning in other environments if the same kinds of cognitive processing are promoted and the method is not unique to one media.
This is the first study to compare an attendance-based course and a web-based course in a real setting showing that the two course formats were equally effective. The study has high relevance because it addressed an important aspect of successful learning in the university context, and the results lead us to the conclusion that it is possible to enhance SRL in university students in different training formats. In this respect, it is important to note that when employing a training program, not only a theory but also an implementation phase is crucial. This study indicates that learners need opportunities to apply the material they learned in order to actually develop the ability to apply the strategies. Although it was challenging to conduct this field study, it revealed important findings and provides implications for supporting SRL in a university setting.
References
(1991). Attributional style and theory: Let no one tear them asunder. Psychological Inquiry, 2, 11.
(2002). Comparing student satisfaction with distance education to traditional classrooms in higher education: A meta-analysis. The American Journal of Distance Education, 16, 83 – 97.
(2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88, 234 – 245.
(2016). Applying a web-based training to foster self-regulated learning – Effects of an intervention for large numbers of participants. Internet and Higher Education, 31, 87 – 100.
(2015, März). SRL@U: Validierung des Fragebogens zur Erhebung des selbstregulierten Lernens an der Universität. Vortrag auf der 3. Tagung der Gesellschaft für Empirische Bildungsforschung (GEBF), Bochum.
(2006).
Transfer: Die Umsetzung und Generalisierung erworbener Kompetenzen in den Arbeitsalltag . In K. Sonntag (Hrsg.), Personalentwicklung in Organisationen (S. 355 – 388). Göttingen: Hogrefe.(2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74, 379 – 439.
(2010). Improving the quality of e-learning by enhancing self-regulated learning. A synthesis of research on self-regulated learning and an implementation of a scaffolding concept. Dissertation, Technische Universität Darmstadt.
Bundesministerium für Arbeit und Soziales (2017). Weiss Buch: Arbeiten 4.0. Berlin: BMAS Publications.(1999) Self-regulated learning: Where we are today. International Journal of Educational Research, 31, 445 – 457.
(1999). Statistik für Sozialwissenschaftler (5. Aufl.). Berlin, Heidelberg: Springer.
(2007). Training transfer: An integrative literature review. Human Resource Development Review, 6, 263 – 296.
(2016, März). Vermittlung von Selbstreguliertem Lernen im Studium: Wie wirksam sind E-Learning Veranstaltungen im Vergleich zu Präsenz-Seminaren? Posterpräsentation auf der 4. Tagung der Gesellschaft für Empirische Bildungsforschung (GEBF), Berlin.
(1994). Media will never influence learning. Educational Technology Research and Development, 42, 21 – 29.
(2016). Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research. 86, 79 – 122.
(2014). Increasing self-regulatory energy using an internet based training application delivered by smartphone technology. Cyberpsychology, Behavior and Social Networking, 17, 181 – 186.
(2006). Action e-learning: An exploratory case study of action learning applied online. Human Resource Development International, 9, 157 – 171.
(2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3, 231 – 264.
(1981). There's a S.M.A.R.T. way to write management's goals and objectives. Management Review, 70, 35 – 36.
(2016). Self-regulated learning profiles in college students: Their relationship to achievement, personality, and the effectiveness of an intervention to foster self-regulated learning. Learning and Individual Differences, 51, 229 – 241.
(2010). Factors influencing the effectiveness of courses in iranian university e-Learning centers. International Journal of Technology, Knowledge & Society, 6, 71 – 80.
(2014). The effects of goal specificity and scaffolding on programming performance and self-regulation in game design. British Journal of Educational Technology, 45, 285 – 302.
(2012). Teaching practices that foster self-regulated learning: A case study. Educational Research, 1, 1 – 16.
(2011). The transfer of training: what really matters. International Journal of Training and Development, 15, 103 – 120.
(1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66, 99 – 136.
(2007). Effects of self-regulated learning strategy training on learners' achievement, motivation and strategy use in a web-enhanced instructional environment. Retrieved September 03, 2018, from http://purl.flvc.org/fsu/fd/FSU_migr_etd- 3689
(2009). Fördert ein Blended Learning Szenario selbstreguliertes Lernen an der Hochschule? Unterrichtswissenschaft, 37, 347 – 361.
(2016). A review of literature on e-learning systems in higher education. International Journal of Business Management and Economic Research, 7, 754 – 762.
(2012). Wann wird Weiterbildung nachhaltig? – Erfolg und Erfolgsfaktoren beim Lerntransfer. Personal Quarterly, 64, 10 – 15.
(2008). Prompting in web-based environments: Supporting self-monitoring and problem solving skill in college students. Journal of Educational Computing Research, 38, 115 – 137.
(1979). Techniques for evaluating training programs. Training and Development Journal, 33, 78 – 92.
(2008). Self-regulation and ability predictors of academic success during college: A predictive validity study. Journal of Advanced Academics, 20, 42 – 68.
(2006). Disentangling achievement orientation and goal setting: Effects on self-regulatory processes. Journal of Applied Psychology, 91, 900 – 916.
(1998).
Decomposing self-regulation and self-control: The volitional component inventory . In J. HeckhausenC. S. Dweck (Eds.), Motivation and self-regulation across the life span (pp. 15 – 49). New York: Cambridge University Press.(2003). Selbstreguliertes Lernen als Selbstregulation von Lernstrategien – Ein Trainingsexperiment mit Berufstätigen zum Lernen aus Sachtexten. Unterrichtswissenschaft, 31, 38 – 56.
(1990). A theory of goal setting and task performance. Englewood Cliffs, NJ: Prentice Hall.
(2014) Instructional quality of massive open online courses (MOOCs). Computers & Education, 80, 77 – 83.
(2005). Learning to reflect and to attribute constructively as basic components of self-regulated learning. British Journal of Educational Psychology, 75, 351 – 372.
(2009). Self-regulation, goal orientation, and academic achievement of secondary students in online university courses. Journal of Educational Technology & Society, 12, 4 – 11.
(2003). The promise of multimedia learning: Using the same instructional design methods across different media. Learning and Instruction, 13, 125 – 139.
(2017). Using multimedia for e-learning. Journal of Computer Assisted Learning, 33, 403 – 423.
(2010). Some advantages of e-learning in english language training. Trakia Journal of Sciences, 8, 21 – 28.
(2005). Training of self-regulatory and problem-solving competence. Learning and Instruction, 15, 123 – 139.
(2007). Self-Regulation from a process perspective. Journal of Psychology, 215, 194 – 204.
(1987). Explanatory style and academic performance among university freshman. Journal of Personality and Social Psychology, 53, 603 – 607.
(2003). College adults are not good at self-regulation: A study on the relationship of self-regulation, note taking, and test taking. Journal of Educational Psychology, 95, 335 – 346.
(1991). The Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor: University of Michigan.
(2000).
Teaching innovations in self-regulated learning . In M. BoekaertsP. R. PintrichM. Zeidner (Eds.), Handbook of self-regulation (pp. 651 − 686). San Diego, CA: Academic Press.(1987). Causal attributions of hyperactive children: Implications for teaching strategies and self-control. Journal of Educational Psychology, 79, 296 − 307.
(2010).
Bezugsnormorientierung . In D. H. Rost (Hrsg.), Handwörterbuch Pädagogische Psychologie (S. 61 – 68). Weinheim: Beltz.(2013). Instructional design interventions for supporting self-regulated learning: Enhancing academic outcomes in postsecondary e-learning environments. Journal of Online Learning and Teaching, 9, 590 – 601.
(2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54 – 67.
(2006). New perspectives for the evaluation of training sessions in self-regulated learning: Time-series analyses of diary data. Contemporary Educational Psychology, 31, 64 – 96.
(2000).
Self-regulation and academic learning: Self-efficacy enhancing interventions . In M. BoekaertsP.R. PintrichM. Zeidner (Eds.), Handbook of self-regulation (S. 631 – 649). San Diego, CA: Academic Press.(2004). Das 1 x 1 des Zeitmanagement (24. Aufl.). Frankfurt am Main: mvg Verlag.
(2003). Differences between traditional and distance education academic performances: A meta-analytic approach. International Review of Research in Open and Distance Learning, 4, 1 – 20.
(2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59, 623 – 664.
(2005). Lernen mit einer netzbasierten Lernumgebung im Bereich empirischer Forschungsmethoden. Unterrichtswissenschaft, 33, 3 – 29.
(1997). Longitudinal study of procrastination, performance, stress, and health: The costs and benefits of dawdling. Psychological Science, 8, 454 – 458.
(2011). Developing an appropriate design of blended learning with web-enabled self-regulated learning to enhance students' learning and thoughts regarding online learning. Behaviour & Information Technology, 30, 261 – 271.
(2008). Self-regulated profiles and academic achievement. Psicothema, 20, 724 – 731.
(2016). What role does knowledge quality play in online students' satisfaction, learning and loyalty? An empirical investigation in an eLearning context. Journal of Computer Assisted Learning, 32, 561 – 575.
(1985). An attributional theory of achievement motivation and emotion. Psychological Review, 92, 548 – 573.
(2012). Förderung des selbstregulierten Lernens durch die Lehrkräftefortbildung „Lernen mit Plan“: Effekte auf fokale Trainingsinhalte und die allgemeine Unterrichtsqualität. Zeitschrift für Pädagogische Psychologie, 26, 291 – 305.
(1994). Lernstrategien im Studium: Ergebnisse zur Faktorenstruktur und Reliabilität eines neuen Fragebogens. Zeitschrift für Differentielle und Diagnostische Psychologie, 15, 185 – 200.
(2008). Self-regulated learning as a competence: Implications of theoretical models for assessment methods. Zeitschrift für Psychologie, 216, 102 – 110.
(2016). University students' self-control and self-regulated learning in a blended course. Internet and Higher Education, 30, 54 – 62.
(1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25, 3.
(2000).
Attaining self-regulation: A social cognitive perspective . In M. BoekaertsP. R. PintrichM. Zeidner (Eds.), Handbook of self-regulation (pp. 13 – 39). San Diego, CA: Academic Press.(2011). Handbook of self-regulation of learning and performance. New York: Routledge.
