Fostering pre-service teachers' assessment skills in a video simulation
Differential effects of a utility value intervention and conceptual knowledge prompts
Abstract
Abstract: An adequate on-the-fly assessment of relevant learner characteristics is an essential professional ability for effective teaching. However, this task is challenging, particularly for teacher students, as they often struggle in applying conceptual knowledge and lack perception of personal value and utility of study contents. Video-based simulations enable the acquisition of practice-oriented abilities for student assessment in initial teacher education. Implementing additional scaffolding in simulations can increase learning gains. The present study examines whether a utility value intervention and conceptual knowledge prompts can effectively support the assessment of relevant learner characteristics and how such effects are influenced by success expectancy. The study participants were N = 108 pre-service teachers, who completed a validated video-based simulation. They were randomly assigned to both interventions (utility value intervention, conceptual prompts) in a 2 × 2 design. The results showed that conceptual prompts improved judgment accuracy effectively. The utility value intervention yielded only descriptive improvements that require further investigations. The combination of both interventions was least effective. Furthermore, the results suggest that participants with low success expectancy benefited more from conceptual prompts. These results suggest that conceptual knowledge prompts and tentatively also utility value interventions can be used as effective scaffolds in simulations in the context of assessment skills. However, they also tentatively suggest that more processing and reflection time might be required for the combined scaffold to be effective. In addition, the differential effectiveness of both scaffolds emphasizes that an adaptation of scaffolds based on, for example, success expectancy can support additional learning gains.
Zusammenfassung: Die angemessene Beurteilung wichtiger Lernvoraussetzungen gehört zu den grundlegenden professionellen Kompetenzen von Lehrpersonen. Gerade für Lehramtsstudentinnen und -studenten ist dies jedoch herausfordernd, da sie häufig Probleme bei der Anwendung konzeptuellen Wissens haben und den persönlichen Wert der Studieninhalte für entsprechende Anwendungssituationen nicht ausreichend erkennen. Video-basierte Simulationen ermöglichen den Erwerb praxisorientierter Fähigkeiten zum Diagnostizieren bereits im Universitätskontext. Durch die Implementation von Scaffolding in den Simulationen kann der Lernerfolg weiter gefördert werden. In der vorliegenden Studie wird untersucht, ob eine Utility Value Intervention und konzeptuelle Wissensprompts das Diagnostizieren von Lernvoraussetzungen effektiv unterstützen und ob dies von der Erfolgserwartung beeinflusst wird. Zur Untersuchung absolvierten N = 108 Lehramtsstudentinnen und -studenten eine validierte video-basierte Simulation und wurden den beiden Interventionstypen (Utility Value Intervention, konzeptuelle Wissensprompts) im 2 × 2 Design zufällig zugewiesen. Die Ergebnisse legen einen positiven Einfluss der Wissensprompts auf die Diagnoseakkuratheit nahe. Ein positiver Einfluss der Utility Value Intervention zeigte sich nur deskriptiv und bedarf weiterer Untersuchungen. Die Kombination beider Interventionen war hingegen am wenigsten förderlich. Die Ergebnisse zeigen zudem, dass Teilnehmende mit geringer Erfolgserwartung stärker von konzeptuellen Prompts profitierten. Diese Ergebnisse legen nahe, dass konzeptuelle Prompts sowie tendenziell auch Utility Value Interventionen im Kontext des Diagnostizierens in Simulationen als geeignete Scaffolding Maßnahmen eingesetzt werden können. Sie lassen aber auch vermuten, dass im Fall der Kombination beider Scaffolds mehr Zeit für Verarbeitung und Reflexion nötig sein könnte. Außerdem unterstreicht die differentielle Wirksamkeit der Scaffolds, dass eine adaptive Darbietung der Scaffolds, z.B. abhängig von der Erfolgserwartung, dabei helfen kann, Lernerfolge zusätzlich zu steigern.
Introduction
Assessing students' learner characteristics (e.g., prior knowledge) is a crucial task for teachers. However, particularly beginning teachers tend to be overwhelmed by this complex task (Stokking et al., 2003). As such challenges also affect early teachers' dropout rates, thorough considerations on how to address these challenges seem important. In that regard, recent conceptualizations of assessment skills (Loibl et al., 2020) have suggested that conceptual knowledge (e.g., models on how to assess students in classroom situations), which is usually the main focus in initial university teacher education, is not sufficient for acquiring practically applicable assessment skills. Thus, students need to be supported in acquiring assessment skills in applied ways (Grossman et al., 2009).
In this context, simulations have been shown as effective learning environments (Chernikova et al., 2020). They not only enable learning in relatively authentic settings but also reduce complexity by decomposing practice into distinct learning units (Grossman et al., 2009). Based on the advantages of high authenticity, good scalability, and low implementation efforts, recent research has increasingly focused on video-based simulations as a reasonable and promising way to foster pre-service teachers' (PSTs') assessment skills (Codreanu et al., 2020).
Chernikova et al. (2020) emphasized that additional scaffolding can enlarge PSTs' learning gains in simulations. While research has suggested that cognitive scaffolding is effective, the effects of motivational scaffolding have rarely been studied in this context (Belland et al., 2013). This is especially surprising, as the important role of motivation for learning has been acknowledged for quite some time and prior motivational interventions have shown positive effects on learning (Hulleman & Harackiewicz, 2021; Wigfield, 1994). Increasing learner motivation can be an important vehicle for improving learning and be based on the expectancy value theory (Wigfield, 1994). According to this theory, learner motivation depends on success expectancy (SEp) and subjective value regarding a certain task. Both can be used to increase learner motivation.
The present study empirically examines whether cognitive and motivational scaffolds can improve PSTs' assessment skills in a video-based simulation.
Assessment skills
Assessment skills include accurately assessing learner characteristics and task properties and identifying students with learning difficulties (Urhahne & Wijnia, 2021). To adapt lessons to learners' prior knowledge and best support learners individually, teacher assessment skills are particularly relevant. The accurate assessment of academic performance has been extensively studied and been found to be improvable. For instance, Hosenfeld et al. (2002) compared teachers' judgment of a given task's solution rate with the actual solution rate of their students and found that, on average, teachers overestimated the solution rate by 18% (see Urhahne & Wijnia, 2021, for an empirical overview). Current research focuses on conceptualizing assessment skills (e.g., Loibl et al., 2020). Also, first interventions for improving assessment skills have been developed and have shown to be effective, particularly when tasks include problem solving (Chernikova et al., 2020), as simulations do, for example. Simulations also allow for embedding additional scaffolding.
Scaffolding
To support PSTs' knowledge activation and application in assessment situations, studies suggest that scaffolding is effective (see Chernikova et al., 2020, for an overview). Scaffolding can be defined as a specific type of support that helps learners to focus on certain aspects of a learning situation, whereas other cognitive and motivational demands of the situation are taken over by a more knowledgeable other or a provided scaffold (e.g., a simulation tool; Belland et al., 2013).
Conceptual prompts
Scaffolding is often considered as a form of cognitive support. Conceptual prompts (CP) as one example of cognitive scaffolding support learners by “guid[ing] learner[s] in what to consider” (Hannafin et al., 1999, p. 131) in the learning environment. CP are probably the most prominent type of scaffolding in computer-based learning, also in the context of assessment skills (Chernikova et al., 2020). Due to their broad application, various types of CP have been discussed (Klepsch & Seufert, 2021). For example, in the context of assessment skills, CP can focus PSTs' attention on a particular aspect of an assessment task (Sommerhoff et al., 2023). Interpreted in the sense of scaffolding, CPs embedded in a simulation may steer PSTs' attention, allowing the PST to focus especially on the evaluation of relevant task affordances.
By guiding the learners' considerations, CP regulate cognitive learner activities as they reduce intrinsic (i.e., task-specific) or extraneous (depending on the task presentation) cognitive load (CL; Klepsch & Seufert, 2021; Sweller et al., 2019). For example, CP that guide PSTs' attention are meant to reduce intrinsic load, as the assessment task is simplified. In contrast, the extraneous load cannot be expected to decrease and might even increase, as prompts have to be additionally processed by PSTs. From a motivational perspective, CP might be perceived as extra help and, thus, can increase SEp, which, in turn, might affect performance positively (Belland et al., 2013). As particularly learners with low SEp might benefit from this expectancy “boost,” differential effectiveness of CP on performance is assumed. However, empirical data regarding the influence of motivational variables on effectiveness of CP are yet to be reported.
Utility value intervention
Although the initial concept of scaffolding included motivational support, this type of scaffolding has so far been studied to a smaller extent, also in teacher education (see Belland et al., 2013, for an overview and possible reasons for this lack).
The so-called UV interventions (UVIs) aim at enhancing learners' subjective value towards a desired learning content. As PSTs often lack perception of UV regarding university learning content (Alles et al., 2019) and UVIs have been effective in increasing motivation and performance in other contexts in higher education (Hulleman & Harackiewicz, 2021), a UVI supporting PSTs' assessment skills seems promising. For example, Cromley et al. (2020) recently reported that combining motivational scaffolding with cognitive scaffolding can increase biology course grades. Yet, it remains unclear whether this transfers to promoting PSTs' assessment skills.
The effectiveness of such UVIs may also be influenced by learners' SEp. For example, in the context of psychology students learning a math technique, Durik et al. (2015) reported high effectiveness of their UVI for learners with high SEp. As, to our knowledge, UVIs have so far not been used to foster PSTs' assessment skills, it still needs to be empirically tested whether the positive effects on motivation and performance can be transferred.
Research question (RQ)
This study investigated the effectiveness of motivational (UVI) and cognitive scaffolding (CP), as well as their combination for fostering PSTs' assessment skills in a video-based simulation:
Can CP and/or a UVI promote intrinsic CL, extraneous CL, UV, and judgment accuracy?
Based on previous results on the effectiveness of CP in simulations on assessment skills, CL is assumed to decrease, while judgment accuracy is assumed to improve. In addition, an increase in UV and judgment accuracy is expected for a UVI. Finally, we expect the combination of both scaffolds to be especially effective, as it combines the advantages of both.
Methods
Sample
PSTs from seven German universities participated in the study, which was administered online in 2021. Participation was integrated in regular university courses as a voluntary opportunity to deepen course contents and was remunerated with 50€. In total, N = 108 PSTs participated in the study [76 f (70.4%), 31 m (28.7%), 1 d (0.9%); Mage = 23.7 y, SD = 3.2 y], who were in various phases of their study programs (Msemester = 6.4, SD = 3.2).
Study design
The course of the study can be seen in Figure 1. The questionnaire instruments for the dependent variables (UV, extraneous/intrinsic CL) and the independent variable (SEp) can be found as electronic supplementary material 1 (ESM 1). All instruments were administered after the simulation.
Judgment accuracy (dependent variable) was measured at the end of the simulation: Participants rated eight items on math abilities of two simulated students. This rating was compared with an expert rating, resulting in a score from 0 to 16 points.
Video-based simulation
The video-based simulation focuses on assessing two seventh-grade students' mathematical argumentation skills. The participants' assessment was based on the information they gathered in short video clips depicting one student discussing his/her work on a mathematical proof task with the teacher as a typical type of classroom interaction during working phases in school. After the video clips, participants made their judgment as explained above (for details, see ESM 2 and Codreanu et al., 2020).
For the different kinds of scaffolding provided in the main training session, the assignment was based on a 2 × 2 intervention design:
Utility Value Intervention (UVI): Before the simulation, participants additionally watched a 5-min video on the UV of assessment skills for teachers. The video also included statements from other PSTs on the positive UV of assessment skills. Additionally, the participants were then asked to write a few sentences on why assessment skills are personally relevant and useful for them. To simulate the writing task, participants not receiving a UVI were asked to write a few sentences on the importance and characteristics of supportive performance-related feedback for students. This group did not receive CP.
Conceptual Prompts (CP): CP were presented to participants before each video. Each prompt was created according to the following example: “Pay special attention to [element of the video, e.g., how the student explains a parallelogram]. What can you conclude on [one indicator of the students' skills that should be assessed, e.g., a student's problem-solving strategies]?” These prompts have already been validated (Sommerhoff et al., 2023). This group did not receive a UVI.
Combined Condition (UVI + CP): The participants in this condition received the UVI and the CP mentioned above.
Control Condition: The participants in this condition neither received CP nor UVI.
Data analysis
The following approach was conducted separately for each dependent variable: We first compared pre-test values between the conditions (ANOVA). We then compared pre- to post-test1 values across all conditions (two-way repeated measures ANOVA) and individually for each condition (t-tests). For more details and a power analysis, see ESM 3.
Results
Effects of scaffolds on UV, CL, and judgment accuracy
For descriptive statistics, see Table 1.
UV and CL
ANOVAs did not reveal significant differences across conditions in the pre-test (all p's > .23). Regarding effects of the interventions, repeated measures ANOVAs did not reveal a significant interaction effect of time × condition (all p's > .19, ESM 4). When analyzing the development within each condition, UV did not change significantly in all four conditions (all p's > .21). This may mirror that UV scores were almost on top of the scale in the pre-test. Regarding CL, participants in the control condition perceived significantly less extraneous CL from pre- to the post-test (t(28) = –2.49, p = .019, Cohen's d = –0.47). No significant changes were found for all other conditions (all p's > .71). For intrinsic CL, there were non-significant decreases in the CP (t(21) = –1.99, p = .060, d = –0.43) and UVI + CP (t(29) = –1.92, p = .064, d = –0.36) conditions.
Judgment accuracy
Pre-test values did not differ significantly across conditions (F(3, 104) = 0.72, p = .540). Descriptively, participants improved from the pre- to post-test in all conditions (Figure 2). This is also reflected in a significant main effect of time within a repeated measures ANOVA (F(1, 104) = 14.30, p < .001). However, the interaction effect with condition was not significant (F(3, 104) = 0.53, p = .660). Analyzing the development of the individual conditions showed non-significant increases for the control (t(28) = 1.74, p = .093, d = 0.33) and UVI + CP conditions (t(29) = 1.34, p = .191, d = 0.25), but significant improvement of participants in the CP condition (t(21) = 2.90, p = .009, d = 0.63). Moreover, even though the UVI condition did not show significant improvement (t(26) = 1.57, p = .128, d = 0.31), the participants' post-test judgment accuracy was descriptively almost at the same level as that in the CP condition.
Further exploratory RQ: differential effects of scaffolds
Beyond the already addressed RQ, descriptive data suggested that participants' SEp might have influenced the scaffolds' effectiveness, which has already been document in general in prior research (Canning & Harackiewicz, 2015). Thus, we further investigated the following exploratory RQ.
Does learners' high or low initial SEp influence the effectiveness of scaffolds on judgment accuracy? Are the scaffolding conditions more effective in supporting the judgment accuracy of learners with high or low SEp?
To answer this RQ, we calculated a linear mixed model and estimated and compared marginal means of judgment accuracy within that linear mixed model for participants with one standard deviation (SD) below/above the mean of SEp. For this procedure, no explicit assignment of participants to marginal groups is necessary (see ESM 3 for more details). The model itself did not reveal any significant effects (ESM 4). However, different increases of the estimated marginal means of judgment accuracy for learners with high vs. low SEp within the conditions suggest differential effectiveness of the scaffolds (Figure 3, ESM 4). For participants one SD below mean of SEp, participants in the control condition showed a non-significant increase in their judgment accuracy (t(100) = 1.86, p = .066, d = 0.60). In the CP condition, data revealed a significant increase (t(100) = 3.01, p = .003, d = 1.40), with the participants almost doubling their initial judgment accuracy (pre-test: 4.7, post-test: 8.1). CP + UVI condition (t(100) = 1.52, p = .132, d = 0.57) and UVI condition (t(100) = 0.21, p = .831, d = 0.09) showed non-significant improvements. For participants one SD above the mean of SEp, UVI showed a non-significant effect on judgment accuracy (t(100) = 1.87, p = .065, d = 0.75). Also, all other conditions showed non-significant, yet smaller effects (all p's > .27, all d's < 0.43).
Discussion
In this study, the effects of cognitive and motivational scaffolds on PSTs' UV, CL, and judgment accuracy in a simulation were investigated, additionally exploring possible differential effects of particularly high or low initial SEp.
Effects of scaffolds on UV, CL, and judgment accuracy
Our results indicate that UV did not change in any study condition. This might, in our case, be caused by a ceiling effect, as PSTs already perceived very high utility of assessment skills prior to the intervention. Concerning CL, only the control condition perceived less extraneous CL in the post-test. A possible explanation for this might be that the participants across the intervention groups were confronted with new instructional elements that required additional cognitive processing. Decreases in intrinsic CL were twice as high for both conditions receiving CP than for the other conditions, but did not reach significance. This might indicate that CP help reducing the learners' cognitive effort when dealing with complex assessment tasks as expected (Klepsch & Seufert, 2021).
Regarding judgment accuracy, the CP condition improved significantly, which is in line with previous results (Sommerhoff et al., 2023). While judgment accuracy of UVI condition was comparable to the CP condition in the post-test, its descriptive improvement did not reach significance. Considering the power analysis and the small to medium, positive effect sizes in previous research on UVI, especially in the university context (Hulleman & Harackiewicz, 2015), it seems plausible that these descriptive effects would have been significant in studies with more power, substantiating positive effects of a UVI in the context of assessment skills. However, despite PST's general lack of UV regarding university learning content (Alles et al., 2019), applying their gathered knowledge on assessment skills in the unscaffolded simulation in the pre-test may already have convinced PSTs of the utility of assessment skills for their professional work, leading to already high UV values in the pre-test measure and thus small to negligible effects of the subsequent UVI. This, in turn, underpins the value of simulations in teacher education.
Against the backdrop that we only considered short-term improvements in our study, further studies are necessary that investigate long-term effects and use longer reflection tasks in the context of PSTs' assessment skills. Altogether, we conclude that the participants' performance on judgment accuracy can be supported with our CP, whereas the positive tendencies of the UVI need to be further investigated (Hulleman & Harackiewicz, 2021; Klepsch & Seufert, 2021). Unexpectedly, the combined CP + UVI condition did not lead to improvement in any of the investigated variables. This appears to contradict results by Cromley et al. (2020), who found that combining cognitive and motivational interventions enlarges the positive effects of each individual intervention. However, Cromley et al.’s (2020) results also emphasize the need for a good timing and sufficient time for reflection to effectively enlarge the scaffolds' effectiveness. This may explain the missing effectiveness in the present study, as the learners only had about 90 min to work on the simulation and, thus, might lack sufficient reflection time regarding both scaffolds. Against this backdrop, it seems important that future research investigates under which conditions scaffolds can be combined effectively and under which conditions this might even be detrimental.
Differential effects of scaffolds on judgment accuracy
Further explorative analyses revealed that our scaffolds might have been differentially effective for learner subgroups: Learners with comparably low SEp seem to have benefitted most from CP. As mentioned above, CP are an evident help to assess simulated students, which can boost SEp, especially for learners with low initial SEp and, thus, performance (Belland et al., 2013).
In turn, learners with high SEp could be a group to benefit from a UVI, even when our findings did not show significant effects, likely due to the ceiling effect. As most learners perceived high UV, high SEp was expected to lead to high motivation and performance (Wigfield, 1994). In our study case, comparably low SEp might even be detrimental, as all learners perceive high utility of assessment skills for later professional demands while not feeling able to cope with these demands (Meyer et al., 2019).
Thus, interventions increasing UV and SEp may be promising in teacher education, however further research is required to further clarify under which conditions and for which learner groups.
Limitations
When interpreting this study's findings, several limitations must be considered. First, the study was conducted at different German universities and, due to COVID-19 restrictions, had to be conducted online. Thus, the data quality could not be controlled as much as in a non-online setting. However, data did not reveal any peculiarities to doubt their quality. As participation in the study was voluntary, there may be a pre-selection of PSTs with high motivation and an especially high UV. However, this effect might have been reduced by the remuneration. To evaluate pre-selection effects, more research on PSTs' perception of the utility of assessment skills is necessary. Second, our analyses regarding assessment skills focused on accuracy. However, future research should also consider other relevant facets of judgment quality (e.g., efficiency). Third and, most importantly, both interventions were relatively short and cognitively demanding. This might have influenced PSTs' CL and motivation during the simulation, as well as their assessment performance in the post-test. As effectively fostering assessment skills often requires a certain amount of time, the effort of designing simulations with scaffolding may especially pay off when PSTs work with such simulations at multiple occasions throughout their studies. Future research needs to examine this in more detail to obtain a better understanding of the potential of scaffolded simulations in teacher education.
Conclusion
This study investigated effects of a motivational and a cognitive scaffold in a video-based simulation aimed at fostering PSTs' assessment skills. Validated CP aimed at focusing learner attention and a newly designed UVI were used and their effects evaluated. The results indicated that CP provide support for promoting PSTs' judgment accuracy, whereas UVI yielded only descriptive improvements that need to be substantiated in future research, using also longer-term interventions.
Electronic supplementary material
The electronic supplementary material (ESM) is available with the online version of the article at https://doi.org/10.1024/1010-0652/a000362
References
(2019). How Candidate Teachers Experience Coherence in University Education and Teacher Induction: The Influence of Perceived Professional Preparation at University and Support during Teacher Induction. Vocations and Learning, 12(1), 87–112.
(2013). A Framework for Designing Scaffolds That Improve Motivation and Cognition. Educational Psychologist, 48(4), 243–270.
(2015). Teach It, Don't Preach It: The Differential Effects of Directly-communicated and Self-generated Utility Value Information. Motivation Science, 1(1), 47–71.
(2020). Facilitating diagnostic competences in higher education – A meta-analysis in medical and teacher education. Educational Psychology Review, 32(1), 157–196.
(2020). Between authenticity and cognitive demand: Finding a balance in designing a video-based simulation in the context of mathematics teacher education. Teaching and Teacher Education, 95, 103146.
(2020). Combined cognitive–motivational modules delivered via an LMS increase undergraduate biology grades. Technology, Mind, and Behavior, 1(2).
(2015). What if I can't? Success expectancies moderate the effects of utility value information on situational interest and performance. Motivation and Emotion, 39(1), 104–118.
(2009). Teaching practice: A cross-professional perspective. Teachers College Record, 111(9), 2055–2100.
(1999).
Open Learning Environments: Foundations, methods, and models . In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory, Vol. II. (pp. 115–140). Lawrence Erlbaum Associates Publishers.(2020). A Framework for Explaining Teachers' Diagnostic Judgements by Cognitive Modeling (DiaCoM). Teaching and Teacher Education, 91, 103059.
(2002). Diagnostische Kompetenz: Unterrichts- und lernrelevante Schülermerkmale und deren Einschätzung durch Lehrkräfte in der Unterrichtsstudie SALVE [Diagnostic competence: How teachers rate the motivational and cognitive characteristics of students in the SALVE study]. Zeitschrift für Pädagogik, 45, 65–82.
(2021).
The utility-value intervention . In G. M. Walton A. J. Crum (Eds.), Handbook of wise interventions: How social psychology can help people change (pp. 100–125). The Guilford Press.(2021). Making an Effort Versus Experiencing Load. Frontiers in Education, 6, Article 645284, 56.
(2019). Expectancy value interactions and academic achievement: Differential relationships with achievement measures. Contemporary Educational Psychology, 58, 58–74.
(2001). FAM: Ein Fragebogen zur Erfassung aktueller Motivation in Lern- und Leistungssituationen [FAM: A Questionnaire for Measuring Situative Motivation in Learning and Performance Situations]. Diagnostica, 47(2), 57–66.
(2023). Pre-service teachers' learning of diagnostic skills in a video-based simulation: Effects of conceptual vs. interconnecting prompts on judgment accuracy and the diagnostic process. Learning and Instruction, 101689.
(2003). From student to teacher: Reducing practice shock and early dropout in the teaching profession. European Journal of Teacher Education, 26(3), 329–350.
(2019). Cognitive Architecture and Instructional Design: 20 Years Later. Educational Psychology Review, 31(2), 261–292.
(2021). A review on the accuracy of teacher judgments. Educational Research Review, 32, 100374.
(1994). Expectancy-value theory of achievement motivation: A developmental perspective. Educational Psychology Review, 6(1), 49–78.
1Post-test values refer to all measures that were obtained during the main training session but after the scaffolded assessment process.
