Facilitating justification, disconfirmation, and transparency in diagnostic argumentation

Teachers' diagnostic skills

Teachers' diagnostic skills include assessing pupils' performance, progress, and learning prerequisites (e.g., Südkamp et al., 2018). Teachers also play an important role in initially identifying pupils who have clinically significant learning difficulties or behavioral problems, such as dyslexia or an attention deficit hyperactivity disorder (ADHD). In many educational systems, actual clinical diagnoses are made by clinical professionals, such as school psychologists, with whom teachers need to collaborate (Albritton et al., 2021). In such situations, teachers need to explain their diagnostic reasoning to achieve a joint understanding with a collaborating professional, which we refer to as diagnostic argumentation.

We characterize diagnostic argumentation by three complementary argumentation facets that build on two basic dimensions – an epistemic dimension (i.e., epistemic activities; e.g., evaluating evidence) and a content dimension (e.g., case-specific evidence, such as hyperactive behavior; Bauer et al., 2019; Bauer et al., 2022): Justification describes evaluating evidence (e.g., inattention, hyperactivity, etc.) as a basis for drawing diagnostic conclusions (see Toulmin, 1958). Disconfirmation of alternative explanations denotes explicating and discussing differential diagnoses (e.g., ADHD, emotional stress, etc.; see Lawson, 2003; Toulmin, 1958). Transparency concerning the applied methods constitutes of describing the sources of evidence and the processes of evidence generation (e.g., tests, observations, conversations, etc.; see Chinn & Rinehart, 2016).

Prior research suggests that preservice teachers seem to focus on justification and tend to omit disconfirmation and transparency in their diagnostic argumentation (Bauer et al., 2022). However, the information associated with the three facets is complementary: Justification in relation with disconfirmation indicates discussing two or more competing diagnoses in light of the evidence; justification in relation with transparency indicates explicating and potentially evaluating methods and sources used to generate specific evidence; disconfirmation in relation with transparency indicates generating evidence for drawing conclusions concerning one or more specific diagnoses (Bauer et al., 2019). Relating the different facets' complementary information strengthens the conclusiveness of diagnostic argumentation. Therefore, we consider relations between the complementary information of justification, disconfirmation, and transparency as a relevant indicator for the quality of diagnostic argumentation. Facilitating these aspects of diagnostic skills seems relevant for preparing future teachers for professional situations that require diagnostic argumentation.

Simulation-based learning and feedback

To facilitate diagnostic argumentation in higher education, meta-analytical evidence suggests using case-based simulations (Chernikova et al., 2020), which are approximations of practice that include simplified, yet valid representations of professional situations (Grossman et al., 2009). Simulation-based learning might benefit from elaborated feedback, which provides information on the appropriate task processing (Wisniewski et al., 2020).

In online learning environments (e.g., digital case-based simulations), elaborated feedback is often implemented as static feedback, for example by providing an expert solution that exemplifies optimal task processing. However, static feedback requires learners to compare their own solution themselves, which demands high levels of learners' cognitive capacities (Sweller et al., 2019) – especially with regard to demanding tasks, such as diagnostic argumentation. In contrast, adaptive feedback adjusts to a learner's task solution by directly addressing gaps compared to the expert solution (Bimba et al., 2017), such as missing argumentation facets in learners' diagnostic argumentation. By making relevant feedback information more visible and accessible to the learners (Machts et al., 2023), adaptive feedback increases the salience of the feedback information, which might facilitate learners' cognitive processing of the feedback (Sweller et al., 2019).

While providing an expert solution as static feedback is easily automated, automating adaptive feedback on learners' argumentative task solutions is challenging. Yet, recent technological advancements in the field of artificial intelligence-based natural language processing (NLP) – namely artificial neural networks that process the context within language instead of only recognizing words – facilitate automating the analysis of written arguments and thus, adaptive feedback on argumentative task solutions (e.g., Wambsganss et al., 2021). However, the effects of NLP-based automatic adaptive feedback compared to static feedback on the relations between justification, disconfirmation, and transparency in preservice teachers' diagnostic argumentation are yet to be investigated.

The present study

In this study, we reanalyzed data from a prior study (Sailer et al., 2023), which investigated effects of NLP-based automatic adaptive feedback compared to static feedback in individual compared to collaborative learning settings on preservice teachers' diagnostic accuracy and the quality of justifications when learning with simulations. This prior study did not investigate relations between justification, disconfirmation, and transparency in preservice teachers' diagnostic argumentation. To analyze relations between argumentation facets, the present reanalysis employed the method of Epistemic Network Analysis (ENA; Shaffer, 2017). ENA analyzes relations between categories in data (e.g., Omarchevska et al., 2021), such as relations between argumentation facets within written explanations, as well as the relative emphasis on these categories.

In our reanalysis, we investigated the effects of adaptive compared to static feedback on the relations between justification, disconfirmation, and transparency and thus, on the quality of preservice teachers' diagnostic argumentation. We hypothesized that adaptive feedback compared to static feedback facilitates relations between justification, disconfirmation, and transparency, because adaptive feedback might increase the salience of feedback information addressing the argumentation facets and their relations.

Participants and research design

For our reanalysis, we used a subsample of our prior study (Sailer et al., 2023) and excluded the learners of a collaborative learning condition to focus on the effects of adaptive compared to static feedback on individual learners' diagnostic argumentation. We reanalyzed data of N = 60 German preservice teachers (50 female, 10 male; age: M = 22.47, SD = 3.45 years; semester: M = 4.62, SD = 3.13, Min = 1, Max = 14), using a randomized experimental design with the between-subjects factor type of feedback and the two experimental groups adaptive feedback (n = 30) and static feedback (n = 30).

Learning environment, materials and tasks

Learners were asked to take on the role of a teacher and process six simulated pupil cases in a learning phase and two simulated pupil cases as post-test. All cases concerned pupils with various learning difficulties or behavioral problems that might indicate a clinical diagnosis in the range of ADHD or dyslexia. The cases were implemented on the case-based online platform CASUS (http://www.casus.net). In each case, learners could access different sources of evidence, such as conversation transcripts (see ESM 1, Supplement A; for all learning materials see: https://osf.io/hn7wm/). To complete a case, preservice teachers wrote an explanation concerning their diagnostic reasoning.

Static and adaptive feedback

In the learning phase, learners in the static feedback condition (SFC) received case-specific expert solutions, which exemplified the epistemic and the content dimension of how experts would relate the complementary information of justification, disconfirmation, and transparency in their diagnostic argumentation (see ESM 1, Supplement B: Supplementary Figure 3).

In the adaptive feedback condition (AFC), learners' explanations were analyzed by an NLP-algorithm, which was trained using the Python-based web service NeuralWeb. The training data (i.e., written explanations on the same simulated cases of 118 preservice teachers) was manually coded regarding diagnostic entities (i.e., content dimension; e.g., hyperactivity) and epistemic activities (i.e., epistemic dimension; e.g., evaluating evidence). Thus, the algorithm could identify diagnostic entities and epistemic activities as correct, incorrect, or missing in new explanations written by learners in the present study (for details about the algorithm and the feedback system see Pfeiffer et al., 2019 and Schulz et al., 2019). Based on the automatic analysis, a suitable subset of around 40 case-specific feedback paragraphs were adaptively shown to the learner. Parts of the feedback addressed the epistemic activities and their relations (i.e., epistemic dimension) and other parts the diagnostic entities and their relations (i.e., content dimension; see ESM 1, Supplement B: Supplementary Figure 4). The adaptive feedback also offered highlighting diagnostic entities and activities found in a learner's submitted explanation.

Procedure

Participants spent M = 156.50 (SD = 33.65) minutes on the laboratory study. They watched a short video about navigating in CASUS and an 18-minute video input about pupils' learning difficulties and behavioral problems. Next, participants entered the learning phase with six simulated cases and received static or adaptive feedback. Participants took a short break after three cases. Finally, participants processed two post-test cases without receiving feedback.

Data sources and measurements

As data source, we used the written explanations from the two post-test cases (see ESM 1, Supplement A). We manually coded the explanations of all cases using a case-specific coding scheme for (a) justification, (b) disconfirmation, and (c) transparency. Two trained raters double coded 18% of the data and achieved substantial agreement: For (a) justification, we coded the presence or absence of the six primary supporting pieces of evidence for the correct diagnosis (Cohen's κ = .90). For (b) disconfirmation, we coded the presence or absence of the six most relevant differential diagnoses (Cohen's κ = .94). For (c) transparency, we coded the presence or absence of informational sources of the six primary evidences for the correct diagnosis (Cohen's κ = .92).

Statistical analysis

To investigate the relations between justification, disconfirmation, and transparency in preservice teachers' diagnostic argumentation, we used an ENA (Shaffer, 2017). The ENA algorithm operationalized the argumentation facets' relations by accumulating and weighting co-occurrences of the argumentation facets, first within learners' written explanations, then per simulated case, and then per experimental group (SFC vs. AFC). The resulting mathematical model is depicted in two-dimensional network graphs. The graphs show the strength of relations (i.e., relative frequencies of co-occurrences) between justification, disconfirmation, and transparency per experimental group as a network. In addition, group means provide information about the relative focus on the facets and on their relations in the argumentation. By aligning two group means on one axis in the network space, systematic variance is shifted to one dimension, which enables statistical testing of group differences. To test group differences between SFC and AFC, we used an independent-samples t-test (α = .05).

Conclusion

Artificial intelligence-based NLP proved to be an effective approach for automating adaptive feedback. Adaptive feedback compared to static feedback fosters the quality of preservice teachers' diagnostic argumentation, indicated by the relations between justification, disconfirmation, and transparency, when learning with simulated cases. Facilitating diagnostic argumentation in teacher education programs by using case-based simulations with adaptive feedback might contribute to the further professionalization of future teachers.