Source selection and source use as a basis for evidence-informed teaching
Do pre-service teachers' beliefs regarding the utility of (non-)scientific information sources matter?
Abstract
Abstract. Pre-service teachers rarely deal with problematic classroom situations in an evidence-informed way. One reason is that they often hold unfavourable beliefs about the utility of educational theories and evidence for competently dealing with classroom problems. Unfavourable beliefs about the utility of educational theories and evidence may weaken not only their actual use, but also already the selection of scientific information sources. Furthermore, beliefs about the utility of non-scientific theories and evidence might play a role in this process. We assessed N = 202 pre-service teachers' beliefs about the utility of educational research, subjective theories and anecdotal evidence in dealing with problematic classroom situations. They were presented with a case vignette of a fictitious teacher's unsuccessful attempts to deal with a problematic classroom situation and asked to provide feedback. Before giving feedback, participants could select up to four different information sources, namely (a) an educational theory, (b) the results of an empirical study, (c) a subjective theory, or (d) anecdotal evidence. Results showed that participants had significantly more positive beliefs about the utility of anecdotal evidence compared to educational research and subjective theories. Moreover, they selected educational theory significantly less often than any other source, and anecdotal evidence more frequently than educational theory and evidence. However, in students' feedback, educational theory was used as often as anecdotal evidence and significantly more often than educational evidence and subjective theories. Students' beliefs about the utility of educational theories and evidence were predictive for both selection and use of scientific sources, albeit with rather small effect sizes. The selection and use of non-scientific sources was not predicted by students' beliefs. Pre-service teacher education should not only target future teachers' skill and knowledge acquisition regarding evidence-informed teaching, but also help them develop more favourable beliefs about the utility of scientific theories and evidence.
Zusammenfassung. Lehramtsstudentinnen und -studenten referenzieren bei der Analyse von unterrichtlichen Problemsituationen nur selten auf bildungswissenschaftliche Theorien und Befunde. Eine mögliche Ursache können ungünstige Überzeugungen hinsichtlich der Nützlichkeit bildungswissenschaftlicher Theorien und Befunde zur Lösung entsprechender Problemsituationen sein. Diese Studie geht der Annahme nach, dass Überzeugungen hinsichtlich der Nützlichkeit bildungswissenschaftlicher Theorien und Befunde nicht nur die Nutzung, sondern auch bereits die Auswahl entsprechender Informationsquellen beeinflussen können. Zudem können dabei auch Überzeugungen zur Nützlichkeit nicht-wissenschaftlicher Theorien und Evidenzen bedeutsam sein. Untersucht wurden N = 202 Lehramtsstudentinnen und -studenten hinsichtlich ihrer Überzeugungen zu bildungswissenschaftlichem Wissen, subjektiven Theorien und anekdotischer Evidenz. Sie wurden gebeten, einer fiktiven Lehrkraft Feedback zu ihren suboptimalen Versuchen zur Lösung einer unterrichtlichen Problemsituation zu geben. Vor der Formulierung des Feedbacks hatten die Teilnehmerinnen und Teilnehmer die Möglichkeit, bis zu vier unterschiedliche Informationsquellen auszuwählen, die ihnen (a) eine bildungswissenschaftliche Theorie, (b) Ergebnisse einer bildungswissenschaftlichen Studie, (c) die subjektive Theorie einer Seminarlehrkraft und (d) anekdotische Evidenz einer erfahrenen Lehrkraft präsentierten. Die Ergebnisse zeigten, dass die Teilnehmerinnen und Teilnehmer signifikant günstigere Überzeugungen zur Nützlichkeit von anekdotischer Evidenz verglichen mit bildungswissenschaftlichem Wissen und subjektiven Theorien aufwiesen. Auch riefen sie die bildungswissenschaftliche Theoriequelle signifikant seltener auf als jede andere Quelle, während die anekdotische Evidenz häufiger ausgewählt wurde als die bildungswissenschaftliche Theorie und die Ergebnisse einer bildungswissenschaftlichen Studie. Überraschenderweise wurde die bildungswissenschaftliche Theoriequelle aber vergleichbar oft wie die anekdotische Evidenz im Feedback genutzt, während die bildungswissenschaftliche Evidenz sowie die subjektive Theorie signifikant seltener referenziert wurden. Die Überzeugungen der Studentinnen und Studenten waren prädiktiv für die Auswahl und Nutzung bildungswissenschaftlicher, nicht aber nicht-wissenschaftlicher Quellen. Daher sollte im Rahmen der Lehramtsausbildung nicht nur versucht werden, Lehramtsstudentinnen und -studenten Gelegenheiten zum Erwerb bildungswissenschaftlichen Wissens, sondern auch zur Entwicklung günstiger wissenschaftsbezogener Überzeugungen bereit zu stellen.
Equipping future teachers with the skills and abilities necessary to consult and use educational evidence when confronted with problematic classroom situations is an important task for pre-service teacher education (e.g., European Commission, 2007; U.S. Department of Education, 2002). The relevance of this task becomes apparent in light of research findings showing that teachers rarely ground their pedagogical decisions in educational evidence (e.g., Hetmanek et al., 2015). Potential reasons for this are manifold, including teachers' possible lack of knowledge of such evidence (Brown & Rogers, 2015; Neuweg, 2011) or difficulties transferring this knowledge to the problem at hand (Wagner, Bergner, Krause, & Stark, 2018).
In addition to such knowledge-related barriers, unfavourable beliefs about the utility of educational theories and evidence have also been discussed as a possible barrier to the application of scientific educational information to real-world classroom problems (Joram, 2007). Such unfavourable beliefs about the utility of educational theories and evidence might not only act as barriers to the actual use of educational theories and evidence. They might already come into play in the decision on whether or not to even consult scientific sources in the first place when confronted with a classroom problem. This might be especially true for pre-service teachers, whose views on good teaching are often based on gut feelings and personal experience rather than scientific evidence (Csanadi, Kollar & Fischer, in press). However, little is also known about pre-service teachers' beliefs about the utility of such non-scientific knowledge and how these impact their reasoning processes (Bråten & Ferguson, 2015).
The present study therefore seeks to uncover pre-service teachers' differential beliefs concerning the utility of scientific (i.e., educational) and non-scientific theories and evidence, their selection of different scientific and non-scientific sources of information, and their use of these sources when thinking about problematic classroom situations. Furthermore, we investigate to what extent pre-service teachers' selection and use of sources are predicted by their beliefs about the utility of (non-)scientific theories and evidence.
Evidence-informed Teaching
Knowledge of scientific, educational and psychological theories and evidence is seen as an essential component of teachers' professional competence (Baumert & Kunter, 2006; Shulman, 1986). Recent results document higher levels of instructional quality, increased teaching self-efficacy, and positive learning outcomes among pupils of teachers who possess more scientific educational/psychological knowledge (e.g., König & Pflanzl, 2016; Voss, Kunter, & Baumert, 2011). Against this backdrop, political bodies (e.g., European Commission, 2007; U.S. Department of Education, 2002) and researchers (e.g., Bauer & Prenzel, 2012) call for “the use of research evidence to support learning and decision-making“ (Williams & Coles, 2007, p.812) by teachers, that is, evidence-informed teaching.
In evidence-informed teaching, however, educational evidence should not be regarded as a “recipe” that can easily be applied to a given problem like a drug can be administered to cure a patient of a headache. When it comes to teaching, educational theories and evidence should instead be regarded as one of several possible sources to guide decisions (Cain, 2015; Stark, 2017). It is certainly true that not every decision a teacher needs to make requires an in-depth consideration of educational theories or evidence (Shavelson & Stern, 1981). Yet when problems come up repeatedly without being sufficiently addressed, teachers should be able to seek out, obtain and potentially apply what scientific research on teaching and learning has to offer to pave the way for competent action (Stark, 2017).
However, prior research suggests that applying scientific educational/psychological theories and evidence to interpret and explain classroom events presents a great challenge for teachers (e.g., Brown & Rogers, 2015; Hetmanek et al., 2015; Neuweg, 2011). A study by Franke and Wecker (2019) showed that when faced with actual classroom problems (e.g., students who seem to be unmotivated to follow the lesson), in-service teachers tend to rely on everyday theories and experiential knowledge, which Neuweg (2011) demonstrated to be often ill-reflected. Furinghetti and Pehkonen (2002) demonstrated that even when teachers possess relevant scientific knowledge, they often have difficulties applying it. Similar results have been reported for pre-service teachers as well (Wagner, Klein, Klopp, & Stark, 2014). Qualitative studies indicate that pre-service teachers tend to use educational theories that do not fit the problem – or if they do fit, apply them only superficially (e.g., Cain, 2015; Furinghetti & Pehkonen, 2002). Instead of applying scientific educational/psychological knowledge, pre-service teachers often fall back on every-day understandings, subjective theories and personal experience (De Bruyckere, Kirschner & Hulshof, 2015; Kagan, 1992; Pajares, 1992). They also tend to engage in tautological explanations, mold empirical evidence to fit their prior beliefs and assumptions or use pseudo-evidence to back up their claims (Kuhn, 1989; Kuhn & Reiser, 2005; Wenglein, Bauer, Heininger, & Prenzel, 2015).
Taken together, pre- and in-service teachers' efforts to engage in evidence-informed teaching are quite often superficial at best. It is therefore a core task for pre-service teacher education to equip future teachers with the skills and abilities necessary to engage in competent, evidence-informed teaching.
Developing a process model of the effects of pre-service teachers' beliefs about the utility of (non-)scientific theories and evidence on evidence-informed teaching
So far, little is known about the mechanisms explaining how (un)favourable beliefs of pre-service teachers about the utility of (non-)scientific theories and evidence affect their reasoning about teaching. We therefore propose a process model that provides a heuristic framework to analyse these mechanisms (see Figure 1). It differentiates between (a) pre-service teachers' beliefs about the utility of different kinds of (non-)scientific sources, (b) the selection of different kinds of (non-)scientific sources, and (c) their use during reasoning about problematic classroom situations. Moreover, it builds on a distinction between different kinds of information pre-service teachers may draw upon during evidence-informed reasoning about problematic classroom situations proposed by Franke and Wecker (2019): scientific theories, scientific (i.e., empirical) evidence, subjective theories and anecdotal evidence. The components of the model and their interrelations are described in more detail below. Note, however, that the framework in Figure 1 only acts as a heuristic tool to help the reader arrive at a better understanding of the rationale of our study. Especially among experienced in-service teachers, the boundaries between different components of the model might be less clear-cut than the model implies. For example, research on teacher expertise showed that it is often hard to disentangle the kinds of knowledge that experienced teachers, in particular, use to tackle a given classroom problem, because scientific knowledge is often integrated with experiential knowledge (Schmidt & Boshuizen, 1992).
Beliefs about the utility of scientific vs. non-scientific sources
The first distinction we propose is between beliefs about the utility of scientific theories and evidence and beliefs about the utility of non-scientific theories and evidence. Clark (1988) referred to teachers' beliefs as preconceptions and implicit theories. He stated that their use does not necessarily have to be and often is not consistent with educational theory and evidence, for they “tend to be eclectic aggregations of cause-effect propositions from many sources, rules of thumb, generalizations drawn from personal experience, beliefs, values, biases, and prejudices” (p.5). So far, substantial research has focused on teachers' beliefs regarding pupils (e.g., Suprayogi, Valcke, & Godwin, 2017), teaching and learning (e.g., Schoen, LaVenia, & Ozsoy, 2019) or epistemic beliefs (e.g., Merk, Rosman, Muis, Kelava, & Bohl, 2018). Much less is known about pre-service teachers' beliefs about the utility of (non-)scientific theory and evidence.
Existing research on this topic has strongly focused on pre-service teachers' typically rather unfavourable beliefs about the utility of scientific educational knowledge. For example, Bråten and Ferguson (2015) found pre-service teachers to more strongly believe in sources of knowledge about instruction derived from practice rather than theory. In a longitudinal study, Allen (2009) observed that pre-service teachers valued the theories they learned in their teacher education programme and the practice they observed during internships to a similar extent. However, once they entered service, they privileged the latter. In another study, Merk et al. (2017) provided pre-service teachers with three texts that were invariant in content, but presented as written either by (a) a practitioner, (b) an expert, or (c) a researcher. They asked participants to rate the perceived value they attributed to these texts for their professional practice. Surprisingly, findings that were labelled as coming from a scientific study were perceived as more relevant for practice than statements by a practitioner or expert. Thus, in sum, the evidence regarding pre-service teachers' beliefs about the utility of different information sources appears to be mixed.
The studies just reported have in common that they all compared participants' beliefs about the utility of scientific vs. non-scientific information. Yet, they ignore the fact that both scientific and non-scientific information may come in different forms. Franke and Wecker (2019) argue that a distinction should be made as to whether information is “theoretical” or “empirical”: Theoretical information refers to explanations for the occurrence or possible antecedents and effects of phenomena, which may either be scientific (e.g., a description of self-determination theory, Ryan & Deci, 2000, or cognitive load theory, Sweller, van Merriënboer, & Paas, 2019) or non-scientific (e.g., the urban myth that “learning styles” exist, see De Bruyckere, et al., 2015). Empirical information, in contrast, is based on observations, which can again either be scientific (e.g., the observation that structured collaborative learning works better with respect to knowledge acquisition than unstructured collaborative learning; see Vogel, Wecker, Kollar & Fischer, 2017), or non-scientific (e.g., a teacher's personal, perhaps repeated experience that “group work does not work”). Thus, in line with Franke and Wecker (2019), we propose to distinguish between beliefs about the utility of four different kinds of information: (a) scientific (i.e., educational) theories, (b) scientific (i.e., educational) evidence, (c) subjective theories, and (d) anecdotal experience, and investigate how these different beliefs relate to the selection and use of sources during pre-service teachers' reasoning about problematic classroom situations.
Source selection
The second component of our heuristic framework (see Figure 1) refers to source selection. During classroom practice, teachers are forced to make many decisions “on the fly”, that is, without having the time to access specific resources beyond their currently available knowledge. Yet, especially when the same problems arise several times or a problem is new and the teacher does not yet have a routine for dealing with it, they may engage in a more in-depth reasoning process. This process may involve a conscious search for and selection of information sources (Shavelson & Stern, 1981).
In line with the distinction proposed above, these information sources may be of different kinds: First, teachers may decide to seek out information sources presenting theoretical accounts which might be relevant for their current problem. For example, upon repeatedly noticing that many students display social loafing behaviour during group work, teachers may decide to look up a short description of Slavin's (1995) theory of cooperative learning and implement a group reward system along with increased individual responsibility during their next group work phase.
Second, teachers may also deliberately search for scientific (i.e., empirical) evidence informing their pedagogical decisions. For instance, when planning a new lesson, teachers may base their decisions regarding how to design an inquiry-based learning session on Lazonder and Harmsen's (2016) meta-analysis of the effects of different scaffolding techniques that can be implemented in inquiry learning.
Third, teachers may also select information sources representing subjective theories. These subjective theories might be either their own or a colleague's personal subjective theory. For example, one personal subjective theory of a biology teacher might be “Photosynthesis is a hard topic that students can only grasp via direct instruction”, while another subjective theory might be “Adding pictures to a text about photosynthesis will increase student motivation”. Such subjective theories can be very stable and resistant to change among practitioners (Cain, 2017).
Fourth, teachers may also refer to information sources representing anecdotal evidence. Again, this might refer either to their own experiential knowledge (e.g., the repeated subjective experience that using computers in the classroom distracts rather than supports students in learning subject matter content) or that of a colleague.
Prior research on teachers' selection of different information sources is scarce, especially when it comes to pre-service teachers. In one study with in-service teachers, Hetmanek et al. (2015) interviewed 25 teachers and asked them what sources they regard as useful for their daily work in the classroom. Out of 110 utterances, 43 referred to scientific evidence from educational research. However, the authors point out that this category was very inclusive and was even coded in the case of rather superficial statements such as “Well, during my internship, I have read authors such as Derbolav and Klafki …” (translation by the authors) or references to neuroscience and brain research, which have been criticised for their limited applicability in real-world educational settings (e.g., Stern, Grabner & Schumacher, 2007). Thus, there are good reasons to expect that pre- and in-service teachers' actual selection of scientific sources might be considerably less frequent, especially when alternative, non-scientific information sources are readily available (Merk et al., 2017; Porsch & Bromme, 2011).
Source use
The third component of our heuristic framework (Figure 1) refers to source use: Once teachers have selected one or several information sources to consult, they need to decide which source(s) or which parts of these sources to actually take up while reasoning about the classroom situation at hand. Here again, in line with Franke and Wecker (2019), we propose to differentiate between use of sources presenting (a) scientific theories, (b) scientific (i.e., empirical) evidence, (c) subjective theories, and (d) anecdotal evidence.
In their study, Franke and Wecker (2019) video-recorded lessons by N = 32 in-service teachers. They then presented each teacher five selected situations from their personal video and asked them to describe what led them to their decisions to engage in particular actions. The results showed that teachers referred markedly less frequently to scientific knowledge (N = 24 references to scientific theories; N = 2 references to scientific evidence) than to subjective theories (N = 87 references) and experiential knowledge (N = 83 references).
Research questions and hypotheses
To date, little is known about the relationship between pre-service teachers' beliefs about the utility of (non-)scientific theories and evidence and their actual use of these theories and evidence when faced with problematic classroom situations. Research on in-service teachers suggests that teachers tend to relate research findings to their own prior beliefs (Cain, 2015; Hetmanek et al., 2015) and that non-congruence with personal values may hamper in-service teachers' acceptance of research findings (Cain, 2016).
In contrast, the question of what role beliefs about the utility of (non-)scientific theories and evidence play for source selection has rarely been investigated, especially among pre-service teachers. We therefore developed an experimental paradigm in which pre-service teachers received a case vignette of a teaching situation and were asked to explain his/her actions and to provide feedback. Participants were first given the opportunity to access information sources of each of the four kinds. Later, they rated their beliefs about the general utility of educational theories, educational evidence, subjective theories, and anecdotal evidence. We assumed that beliefs about the utility of each of the four types of (non-)scientific theories and evidence would predict selection of the corresponding information source, which would in turn positively predict the use of this source when analysing the classroom situation.
Specifically, we asked the following four research questions:
- 1.To what extent do pre-service teachers exhibit different levels of belief about the utility of educational theories, educational evidence, subjective theories, and anecdotal evidence for professional practice?
In line with evidence from previous research (e.g., Bråten & Ferguson, 2015; Joram, 2007), we hypothesized that pre-service teachers would display significantly more negative beliefs about the utility of educational theories and evidence than about the utility of non-scientific theories and evidence (H1).
- 2.To what extent do pre-service teachers differentially select sources including descriptions of educational theories, educational evidence, subjective theories and anecdotal evidence?
Again, in line with empirical evidence from previous studies (e.g., Hetmanek et al., 2015), we hypothesized that pre-service teachers would select educational sources significantly less often than non-scientific sources (H2).
- 3.To what extent do pre-service teachers differently use educational theories, educational evidence, subjective theories, and anecdotal evidence?
In line with our conceptual model and with H1 and H2, we expected pre-service teachers to use sources containing educational theories and evidence significantly less often than sources that present non-scientific theories and evidence (H3).
- 4.To what extent do pre-service teachers' general beliefs about the utility of (non-)scientific theories and evidence predict source selection and source use?
We expected that more positive beliefs about the utility of educational theories and evidence would go along with a higher likelihood of selecting and using the corresponding educational sources (H4a), whereas more positive beliefs about the utility of non-scientific theories and evidence would correlate positively with the selection and use of non-scientific sources (H4b).
Method
Sample
The original sample consisted of N = 209 pre-service teachers from a German university. Seven participants were excluded because they either did not write case analyses at all or their case analyses were shorter than 30 words (i.e., more than 1.5 standard deviations below the mean length; (the mean length of the case analysis texts was M = 240.11, SD = 149.65). Thus, the final sample consisted of N = 202 pre-service teachers. On average, participants engaged with the learning environment for M = 2441.11 seconds (SD = 898.13), or about forty minutes. Participants were M = 22.54 years old (SD = 3.66) and in the fourth semester of their teacher education programme (M = 4.22, SD = 2.11) on average. 72.3% of the sample were female; 39.1% were studying to become primary school teachers, while 60.9% were studying to become secondary school teachers. Almost all students (98.0%) had already completed at least one school-based internship.
Design and procedure
Participants worked in a computer-based learning environment. They began by answering demographic questions, before being directed to a page with a written case description of a classroom situation (429 words). Participants were asked to imagine that they were stationed at a school for their induction phase and that the teacher in the case vignette was a colleague. The case vignette described a young teacher who faced challenges regarding his instructional practices and their effects on his pupils. At the end of the case description, participants were asked to imagine that the young teacher had come to them for advice and that they were expected to provide him with feedback. Before moving on to the case analysis task, participants saw a selection page with hyperlinks to four information sources. These sources were introduced as being potentially helpful for formulating feedback. One link was titled “Textbook excerpt” and included a description of an educational theory relevant for the case, while the second (labelled “Summary of a recent study”) presented scientific (i.e., empirical) evidence. The third link was labelled “Opinion of a mentor teacher”. The linked text presented a subjective theory by that mentor teacher. Finally, the link labelled “Field report from a colleague” linked to a text in which an experienced teacher reported her anecdotal experiences with the problems described in the case description. The sequence by which these links were presented was random. While both scientific information sources provided reliable information, the two non-scientific resources contained widely held misconceptions in education (e.g., use of pictures to motivate pupils; De Bruyckere, et al., 2015). However, the students were not informed that the non-scientific information sources contained misconceptions. Two versions of each information source (scientific vs. non-scientific) were developed, which were randomly distributed across the sample. In cases where both scientific or both non-scientific information sources were selected by the participant, the system ensured that they would receive both versions (e.g., self-determination theory and attribution theory) so as to not receive the same information twice. Participants were allowed to consult any number of the four information sources, and could go back to information sources they had visited previously. Once they felt they had enough information, they navigated to the page where they could enter their feedback (Figure 2). Lastly, participants answered a questionnaire about their beliefs concerning the utility of (non-)scientific sources for professional practice in general.
Instruments
Beliefs about the utility of different information sources
To assess participants' beliefs about the utility of different information sources, we developed an instrument based on the taxonomy by Franke and Wecker (2019). Initially, we constructed 30 items with parallel wording per subscale: beliefs about the utility of (a) educational theories, (b) educational evidence, (c) subjective theories, and (d) anecdotal evidence. Following an expert rating and an initial pilot study (N = 10 pre-service teachers), 21 items per scale were removed due to improper wording, low item difficulties and other non-optimal psychometric qualities. The remaining 36 items were further tested on N = 61 pre-service teachers. The scales showed good internal consistencies, with Cronbach's α ranging between .83 and .89. However, in an exploratory factor analysis with varimax rotation, the two subscales “beliefs about the utility of educational theory” and “beliefs about the utility of educational evidence” collapsed into one. Furthermore, we eliminated three further sets of parallel items from each subscale due to inconsistent loadings. This left us with a questionnaire of 28 items, representing three subscales: beliefs about the utility of educational research for professional decisions (14 items, sample item: “A good teacher knows about research findings from educational science”, α = .92); beliefs about the utility of subjective theories for professional decisions (7 items, sample item: “A good teacher uses his or her own explanations to make sense of a classroom situation”, α = .82), and beliefs about the utility of anecdotal evidence (7 items, sample items: “Teaching quality mostly depends on a teacher's repeated personal experiences”, α = .86). Answers were given on a 6-point Likert scale ranging from 1 = “definitely not” to 6 = “definitely yes”.
Source selection
The computer system logged which information sources pre-service teachers opted to look at. This log data was then transformed into absolute frequencies by coding it as either 0 = “did not select this page” or 1 = “selected this page”.
Source use
Two independent coders rated the case analyses for use of each information source on a 4-point Likert scale (0 = “proposition not present at all”, 3 = “proposition clearly present”). To code the presence or absence of information presented in the four information sources, the sources were first analysed for propositions (information units) relevant to the case. Afterwards, each sentence of a participant's written case analysis was coded for the presence of each proposition from each of the information sources (regardless of whether the participant had seen that information source; ICC = .73 – .93). To aggregate the data, we calculated sum scores for each information source and standardised them based on the number of propositions contained within that source. Lastly, scores for all information sources that were not seen by the participant were deleted.
Data analysis
To address the first research question, we ran a repeated measures analysis of variance (rmANOVA), looking at the overall effect as well as the within- and between-subject effects. The same approach was taken to analyse participants' source selection (RQ2) and source use (RQ3). To answer RQ4, we ran multiple regression analyses using (a) source selection and (b) source use as criterion variables. For all analyses, we assumed interval level of measurement (beliefs about the utility of different information sources and source use) or absolute level of measurement (relative frequency: source selection). The alpha level was set to 5%.
Results
RQ1: Beliefs in the utility of scientific and non-scientific information sources
Table 1 provides an overview of the descriptive statistics and statistical analysis for participants' beliefs about the utility of educational research, subjective theories, and anecdotal evidence. In line with our assumptions, participants held more positive beliefs about the utility of anecdotal evidence than about both subjective theories and educational research (p < .01). However, no significant differences were found between subjective theories and educational research. Greenhouse-Geiser corrected within-subject analyses demonstrated that beliefs about the utility of the different information sources significantly differed within persons (F(1.79;361.08) = 103.13, p < .01, η p >² = .34). Moreover, participants significantly differed between each other regarding their beliefs about the utility of different information sources (F(1;201) = 14735.28, p < .01, η p >² = .99). Thus, the data provide at least partial support for Hypothesis 1.
RQ 2: Source selection
In line with their reported beliefs about the utility of different information sources, when asked to select specific information sources for addressing a specific classroom situation, participants more often selected non-scientific rather than scientific information sources (Table 2). More specifically, pairwise comparisons showed that educational theory was selected significantly less often than any of the other information sources (educational evidence: SE = .03, p < .01; subjective theory: SE = .04, p < .01; anecdotal evidence: SE = .04, p < .01). Similarly, educational evidence was chosen by a significantly smaller percentage of participants compared to anecdotal evidence (SE = .04, p = .02; subjective theory: SE = .03, p = .11). Analysis of the within-effect showed significant differences in participants' selection of information sources (F(2.73; 548.59) = 14.99, p < .01, η p >² = .07). Most of the differences in source selection were attributable to the between-level: F(1;201) = 1411.68, p < .01, η p >² = .88). Our results thus largely support Hypothesis 2.
RQ 3: Source use
Surprisingly and contradicting H3, when analysing the feedback participants provided to their fictitious colleague, educational theory was used more frequently than any of the other information sources (F(3;199) = 6.08, p < .01, η p >² = .08, see Table 3). The differences compared to educational evidence (SE = .07, p < .01) and subjective theory (SE = .08, p = .01) were significant, but the difference to anecdotal evidence was not (SE = .09, p = .27). At the within-level, we found small but significant differences in participants' source use (F(2.24;451.00) = 6.31, p < .01, η p >² = .03). Pre-service teachers differed much more substantially between each other in their use of different information sources (F(1;201) = 138.50, p < .01, η p >² = .41).
As a consequence of this unexpected finding, we further explored the relationship between the selection and use of sources. We found that the selection and use of each information source correlated significantly. However, the correlations were only small to medium in size (educational theory: r = .40**, educational evidence: r = .24**, subjective theory: r = .27**, anecdotal evidence: r = .24**). Furthermore, we ran regression analyses for the selection and use of sources, which demonstrated that selection of the respective source was predictive for its use in the subsequent case analysis (Table 4). This suggests that participants not only clicked on the respective link, but typically also used the source material in their feedback. Furthermore, it shows that the results we found for source use, particularly with regard to educational theories, are not driven exclusively by participants' prior knowledge, but also their actual engagement with the sources.
RQ 4: Relationship between beliefs and (1) source selection and (2) source use
Linear regressions showed that beliefs about the utility of the different sources (all three types of beliefs used as predictor in a multiple regression model) significantly predicted the selection of educational theory (F(3;198) = 5.78, p < .01, R >² = .08) and educational evidence (F(3;198) = 3.23, p = .02, R >² = .05). Herein, beliefs about the utility of educational research positively predicted the selection of educational theory and educational evidence. Further, beliefs about the utility of anecdotal evidence negatively predicted the selection of educational theory (see Table 5). Beliefs about the utility of different information sources did not predict the selection of non-scientific sources, however (subjective theory: F(3;198) = 1.23, p = .30, R >² = .02; anecdotal evidence: F(3;198) = .87, p = .46, R >² = .01). Additional binary logistic regressions confirmed this pattern of effects.
Similar results were obtained when source use was treated as a criterion variable: Here, the regression models (all three types of beliefs used as predictor in four separate multiple regression models with either use of educational theory, use of educational evidence, use of subjective theory, or use of anecdotal evidence as criterion variable) showed that beliefs about the utility of different information sources significantly predicted the use of educational theory (F(3;198) = 6.54, p < .01, R >² = .09) and educational evidence (F(3;198) = 6.44, p < .01, R >² = .09) in the case analyses. Both, for the use of educational theory and the use of educational evidence, beliefs about the utility of educational research served as a positive predictor, while beliefs about the utility of subjective theory served as a negative one (see Table 6). As with source selection, beliefs about any of the information sources did not predict the use of subjective theory (F(3;198) = .49, p = .69, R >² = .01) or anecdotal evidence (F(3;200) = .14, p = .94, R >² < .01). Taken together, these results thus provide support for Hypothesis 4a, but not for Hypothesis 4b.
Discussion
We started out from the observation that pre-service teachers often analyse problematic classroom situations based on gut feelings or experiential knowledge rather than educational theories and evidence (e.g., Brown & Rogers, 2015; Hetmanek et al., 2015; Neuweg, 2011). These deficits in pre-service teachers' evidence-informed teaching abilities are worrying, given that psychological and educational research has accumulated a rich body of theories and evidence to draw upon in problematic classroom situations (e.g., Sawyer, 2006). As previously described, one possible reason for these deficits may be pre-service teachers' often rather unfavourable beliefs about the utility of educational theories and evidence for the development of effective solutions to such problems (e.g., Allen, 2009; Buehl & Fives, 2009; Joram, 2007; Knight, 2015). Moreover, prior research has so far largely neglected the role of source selection as a process that might be affected by pre-service teachers' beliefs about the utility of (non-)scientific theories and evidence when analysing problematic classroom situations. Thus, our empirical study looked at pre-service teachers' beliefs about the utility of both scientific and non-scientific theories and evidence (RQ 1), their preferences concerning the selection of sources containing information on scientific and non-scientific theories and evidence (RQ 2), and their actual use of these sources when analysing problematic classroom situations (RQ 3). Furthermore, we were interested in the extent to which pre-service teachers' beliefs would predict their selection and use of sources (RQ 4).
Regarding our first research question, our analyses showed that in line with H1, participants exhibited more positive beliefs about the utility of anecdotal evidence than regarding the utility of educational research. Yet, in partial contrast to H1, no differences were found between their beliefs about the utility of educational research and their beliefs about the utility of subjective theories. On the one hand, our results corroborate findings according to which pre-service teachers ascribe non-scientific knowledge more power for solving classroom problems than educational knowledge (e.g., Bråten & Ferguson, 2015). On the other hand, our findings indicate that this seems to be the case only for non-scientific evidence, not for non-scientific theories. This might indicate that pre-service teachers have at least some idea of the importance of empirical “data” for informing their pedagogical decisions, even though this “data” often does not meet scientific standards. Interestingly, though, we could not empirically differentiate between participants' beliefs about the utility of educational theories and the utility of educational empirical evidence. Even among scientists, there is considerable debate about what scientific educational theories actually are (e.g., whether they should or should not include normative statements; see for example Biesta, 2010) and whether it is advisable to demand that teachers ground their actions (exclusively) in educational evidence (see also Stark, 2017).
Nevertheless, the finding that pre-service teachers have difficulty differentiating between “explanations” (i.e., theory) and “observations” (i.e., evidence) at the scientific level, but not at the non-scientific level, clearly indicates a need to help pre-service teachers make this distinction. Pre-service teacher education should thus (a) put effort into helping pre-service teachers differentiate conceptually between educational theory and educational evidence, and (b) understand the value of educational evidence as compared to anecdotal evidence as a reliable source for actions in the classroom. Regarding (a), it seems advisable to use historical examples from psychological or educational research that clearly show how the empirical testing of theoretical claims can have tremendous effects on theory development. Regarding (b), the use of teacher case studies might be promising (see Kolodner, 2007). For example, pre-service teachers might be provided with cases of experienced teachers who strongly endorsed an “urban myth” regarding education (De Bruyckere et al., 2015) and based incorrect decisions on that myth, with possibly severe consequences. For example, a teacher in such a case might decide to use unstructured small group collaboration in class to “activate students” and “not interfere with the natural social learning processes” that occur within groups. Yet, it turns out that several students engage in social loafing, and thus do not acquire the necessary knowledge and fail their next test. Pre-service teachers could then be asked to explain – based on scientific, empirical evidence – why small group collaboration did not work in this case.
With respect to Research Question 2, our results are even more in line with what would be expected from prior research: As hypothesized, we found that our participants selected non-scientific sources more often than scientific sources. More specifically, we found that participants chose the source containing information about an educational theory significantly less often than any other source. Moreover, the source presenting educational evidence was selected significantly less often than the source containing anecdotal evidence. From our perspective, this result is remarkable: It shows that even when pre-service teachers can access educational theories with the same costs as other (in particular: non-scientific) sources, they deliberately choose significantly less often to even look at sources that would provide them with scientific knowledge. Thus, Hetmanek et al.'s (2015) finding that in-service teachers rarely engage with educational resources to inform their pedagogical decisions does not seem to exclusively depend on the lower accessibility of educational research that teachers often complain about (Williams & Coles, 2007). In addition, at least in terms of source selection, pre-service teachers seem to be sceptical of the notion that educational theories and evidence might be helpful for guiding their decisions. This might be related to the fact that pre-service teachers have spent thousands of hours in school themselves as pupils and thus are likely to have developed very stable subjective theories and collected a great deal of subjective, experiential knowledge on what makes good teaching.
Referring to Research Question 3, we surprisingly found much less scepticism regarding educational sources: Even though participants accessed the source presenting an educational theory less often than the other (including non-scientific) sources, we found that they actually used that educational theory significantly more often than subjective theories and educational evidence. Note that actual use of appropriate theories would have been even higher if we had let references to the theories presented in the scientific theory text count in cases in which students had not selected this information source in the first place. This result stands in contrast to prior research showing that pre- and in-service teachers seem to more often rely on non-scientific rather than scientific concepts and explanations when dealing with problematic classroom situations (e.g., Franke & Wecker, 2019). However, what we could not rule out of course is the use of prior knowledge. As the applicable educational theories (Self-Determination Theory and Attribution Theory) were quite prominent topics in the teacher education programme of the participants, some of their use might have sprung from prior knowledge. However, our subsequent analyses rule out that it might have only been prior knowledge that was used.
A rather pessimistic interpretation of our finding is that the study setting might have led participants to believe they were expected to use the educational sources they were presented with, even though it was clear that they would not have to fear poor grades if they did not. This expectation might have been stronger because participants were already rather advanced students instead of beginning pre-service teachers. A more optimistic interpretation is that pre-service teachers might indeed see some value in educational theories as tools to competently deal with authentic classroom problems (Knight, 2015). At the very least, an important goal for teacher education can be to help pre-service teachers develop this view. On the whole, however, more research is necessary to explain this finding.
Finally, with respect to Research Question 4, our results showed that, as hypothesized, participants' beliefs about the utility of educational research predicted the selection and use of educational theories and evidence positively, while in turn, beliefs about the utility of anecdotal evidence predicted the selection of educational theories negatively and beliefs in the utility of subjective theories predicted the use of educational theory and educational evidence negatively. Thus, beliefs about the utility of educational research not only seem to play a role in the use of such sources (as shown for example by Porsch & Bromme, 2011), but also already for their selection, even though these effects were rather small. Again, a possible implication of this finding is that pre-service teacher education should not only focus on supporting the acquisition of knowledge about educational theories and evidence from educational science and psychology, but also on helping pre-service teachers develop more positive beliefs concerning the utility of educational knowledge for dealing with problematic classroom situations in a more professional way. First attempts in this direction are already under way (Kerwer & Rosman, 2018; Menz, Spinath, & Seifried, 2018).
Interestingly though, we found no evidence in our data that pre-service teachers' beliefs about the utility of non-scientific theories and evidence predict the selection and use of such non-scientific sources. Thus, it appears that other factors play a role in the decision to select and use non-scientific sources when analysing problematic classroom situations. One hypothesis might be that a lack of scientific prior knowledge about educational theories and evidence positively predicts at least the use of non-scientific sources, but further research is needed to identify possible predictors at this point. A possible strategy might be to have a more detailed, qualitative look at single cases to identify potential predictors.
Limitations and conclusions
Of course, this study is not without limitations: First, we did not provide participants with parallel versions of information sources that differed only with respect to stated origin (as for example in Merk et al., 2017). Instead, we presented a text with different content for each information source, because if we had provided students with four identical texts that differed only in the stated source of the information, it would not have been possible to discern what parts of what sources the students used in their feedback. On the other hand, presenting texts with different content (and having multiple texts per category) might have produced variance originating from certain text features beyond the actual source information we were interested in. This might have affected the use of the different sources during feedback formulation, but should not have affected source selection (as students did not know at that point what content would be presented in the linked documents). Nevertheless, future research should try to find ways to rule out possible effects stemming from text features beyond source information.
Second, we included only a limited set of tasks, cases, topics and information sources in our study. Thus, we cannot rule out that our results are specific to the materials we used in our study. Future research should therefore develop more diverse materials and check whether our results can be replicated in other contexts and with other materials.
A third limitation of our study is that we did not assess participants' general feedback-giving skills. It is certainly plausible that some students had more difficulty than others in applying what they read in the sources when giving feedback, and these difficulties might even vary within persons, that is, across information sources. Thus, future research should work to develop an instrument that is able to capture pre-service teachers' general skills in applying information to produce feedback as well as check for their understanding of different information sources.
Fourth, the way we labelled the links to the four information sources (“textbook excerpt”, “summary of a recent study”, “opinion of a mentor teacher”, “field report from a colleague”) may be criticised, as these labels might have left room for interpretation regarding what to expect from each text. For example, the label “textbook excerpt” might lead students to expect to not only receive theoretical information, but perhaps also information on scientific evidence to back up the theory. Thus, future studies might want to either develop clearer labels that leave less room for interpretation or at least try to measure what kinds of sources participants associate with these labels.
Fifth, students' general utility belief ratings might have been biased because the respective items were presented after the participants had already read the case vignette and the different information sources. However, asking participants about their beliefs earlier might have also resulted in biased feedback provision, because students might have felt obliged to stick to their beliefs ratings. Nevertheless, future research should try to develop ways to make such biases less likely.
Finally, the fact that we presented non-scientific theories and evidence from other teachers (rather than giving participants the opportunity to express their own subjective theories and anecdotal evidence) might have led to an underestimation of the power of non-scientific sources. Thus, it would be very interesting for future research to compare the effects of subjective theories and anecdotal evidence that are actually held by pre-service teachers themselves to those of educational theories and evidence.
Despite these limitations, our study shows that pre-service teachers' often negative beliefs about the utility of educational theories and evidence not only have negative effects on the extent to which they use such information to reflect on and derive possible solutions to problematic classroom situations, but also and already on their decisions regarding what kinds of sources to look at when encountering such situations. This result has strong implications for augmenting pre-service teacher education with learning opportunities that not only focus on the acquisition of knowledge regarding scientific (i.e., psychological and educational) theories and evidence, but also incorporate scaffolds to help students develop more positive beliefs about the utility of educational theories and evidence for dealing with problematic classroom situations in a competent and professional way.
References
(2009). Valuing practice over theory: How beginning teachers re-orient their practice in the transition from the university to the workplace. Teaching and Teacher Education , 25 (5), 647–654. https://doi.org/10.1016/j.tate.2008.11.011
(2012). European teacher training reforms. Science , 336 (6089), 1642–1643. https://doi.org/10.1126/science.1218387
(2006). Stichwort: Professionelle Kompetenz von Lehrkräften [Keyword: Professional teacher competence]. Zeitschrift für Erziehungswissenschaft , 9 (4), 469–520. https://doi.org/10.1007/s11618-006-0165-2
(2015). Beliefs about sources of knowledge predict motivation for learning in teacher education. Teaching and Teacher Education , 50 , 13–23. https://doi.org//10.1016/j.tate.2015.04.003
(2010). Why “what works” till won't work: from evidence-based education to value-based education. Studies in Philosophy and Education , 29 , 451–503. https://doi.org/10.1007/s11217-010-9191-x
(2015). Knowledge creation as an approach to facilitating evidence informed practice: Examining ways to measure the success of using this method with early years practitioners in Camden (London). Journal of Educational Change , 16 , 79–99. doi:10.1007/s10833-014-9238-9
(2009). Exploring teachers' beliefs about teaching knowledge: Where does it come from? Does it change?. The Journal of Experimental Education , 77 (4), 367–408. https://doi.org/10.3200/JEXE.77.4.367-408
(2015). Teachers' engagement with research texts: beyond instrumental, conceptual or strategic use. Journal of Education for Teaching , 41 , 478–492. https://doi.org/10.1080/02607476.2015.1105536
(2016). Research utilisation and the struggle for the teacher's soul: A narrative review. European Journal of Teacher Education , 39 (5), 616–629. https://doi.org/10.1080/02619768.2016.1252912
(2017). Denial, opposition, rejection or dissent: why do teachers contest research evidence?. Research Papers in Education , 32 , 611–625. https://doi.org/10.1080/02671522.2016.1225807
(1988). Asking the right questions about teacher preparation: Contributions of research on teacher thinking. Educational Researcher , 17 (2), 5–12. https://doi.org/10.3102/0013189X017002005
(in press). Pre-service teachers' evidence-based reasoning during pedagogical problem-solving: better together? European Journal of Psychology of Education . https://doi.org/10.1007/s10212-020-00467-4
(2015). Urban myths about learning and education . Academic Press.
European Commission [EC] (2007). Towards more knowledge-based policy and practice in education and training . Retrieved June 26, 2019 from https://op.europa.eu/en/publication-detail/-/publication/962e3b89-c546-4680-ac84-777f8f10c590(2019, February). Das mache ich eigentlich immer so – Erfahrungsbasiertes Alltagswissen als Begründung von unterrichtsmethodischen Entscheidungen [Actually, I always do it like this – experiential everyday knowledge as justification for teaching-related decisions] . Paper presented at the Konferenz der Gesellschaft für Empirische Bildungsforschung (GEBF), Cologne, Germany.
(2002).
Rethinking characterizations of beliefs . In G. C. Leder E. Pekhonen G. Törner (Eds.), Beliefs: A hidden variable in mathematics education? (pp.39–57). Dordrecht: Kluwer.(2015). Wozu nutzen Lehrkräfte welche Ressourcen? Eine Interviewstudie zur Schnittstelle zwischen bildungswissenschaftlicher Forschung und professionellem Handeln im Bildungsbereich [For what do teachers use which resources? An interview study at the intersection of educational research and professional action in educational settings]. Unterrichtswissenschaft , 43 (3), 194–210.
(2007). Clashing epistemologies: Aspiring teachers', practicing teachers', and professors' beliefs about knowledge and research in education. Teaching and Teacher Education , 23 (2), 123–135. https://doi.org/10.1016/j.tate.2006.04.032
(1992). Implication of research on teacher belief. Educational Psychologist , 27 (1), 65–90. https://doi.org/10.1207/s15326985ep2701_6
(2018). Mechanisms of epistemic change–under which circumstances does diverging information support epistemic development?. Frontiers in Psychology , 9 , 2278. https://doi.org/10.3389/fpsyg.2018.02278
(2015). Postgraduate student teachers' developing conceptions of the place of theory in learning to teach: “more important to me now than when I started”. Journal of Education for Teaching , 41 (2), 145–160. doi:10.1018/026074276.2015.1010874
(2016). Is teacher knowledge associated with performance? On the relationship between teachers' general pedagogical knowledge and instructional quality. European Journal of Teacher Education , 39 (4), 419–436. https://doi.org/10.1080/02619768.2016.1214128
(2007).
The roles of scripts in promoting collaborative discourse in learning by design . In F. Fischer I. Kollar H. Mandl J. M. Haake (Eds.), Scripting computer-supported collaborative learning. Cognitive, computational, and educational perspectives (pp.237–262). New York: Springer.(1989). Children and adults as intuitive scientists. Psychological Review , 96 (4), 674–689. https://doi.org/10.1037/0003-295X.96.4.674
(2005, April). Students constructing and defending evidence-based scientific explanations . Paper presented at the annual meeting of the National Association for Research in Science Teaching, Dallas, TX.
(2016). Meta-Analysis of inquiry-based learning effects of guidance. Review of Educational Research , 86 (3), 681–718. https://doi.org/10.3102/0034654315627366
(2018, September). Values for and acceptance of (educational) psychology among German pre-service teachers . Paper presented at the Kongress der Deutschen Gesellschaft für Psychologie (DGPs). Frankfurt a.M., Germany.
(2018). Topic specific epistemic beliefs: Extending the theory of integrated domains in personal epistemology. Learning and Instruction , 56, 84–97. https://doi.org/10.1016/j.learninstruc.2018.04.008
(2017). Pre-service teachers' perceived value of general pedagogical knowledge for practice: Relations with epistemic beliefs and source beliefs. PloS one , 12 (9), 1–25. https://doi.org/10.1371/journal.pone.0193632
(2011).
Das Wissen der Wissensvermittler. Problemstellungen, Befunde und Perspektiven der Forschung zum Lehrerwissen [The knowledge of the knowledge transmitters. Problems, findings, and perspectives of research on teacher knowledge] . In E. Terhart H. Bennewitz M. Rothland (Eds.), Handbuch der Forschung zum Lehrerberuf (pp.451–477). Münster: Waxmann.(1992). Teachers' beliefs and educational research: Cleaning up a messy construct. Review of Educational Research , 62 (3), 307–332. https://doi.org/10.3102/00346543062003307
(2011). Effects of epistemological sensitization on source choices. Instructional Science , 39 (6), 805–819. https://doi.org/10.1007/s11251-010-9155-0
(2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology , 25 (1), 54–67. https://doi.org/10.1006/ceps.1999.1020
(2006). The Cambridge Handbook of the Learning Sciences . Cambridge: Cambridge University Press.
(1992).
Encapsulation of biomedical knowledge . In D. A. Evans V. L. Patel (Eds.), Advanced models of cognition for medical training and practice (pp.265–282). New York: Springer.(2019). Teacher beliefs about mathematics teaching and learning: Identifying and clarifying three constructs. Cogent Education , 6 (1), 1599488. https://doi.org/10.1080/2331186X.2019.1599488
(1986). Those who understand: Knowledge growth in teaching. Educational Researcher , 15 , 4–14. https://doi.org/10.3102/0013189X015002004
(1981). Research on teachers' pedagogical thoughts, judgments, decisions, and behavior. Review of Educational Research , 51 (4), 455–498. https://doi.org/10.3102/00346543051004455
(1995). Cooperative learning: Theory, research, and practice (2nd Ed.) . Boston: Allyn & Bacon.
(2017). Probleme evidenzbasierter bzw.-orientierter pädagogischer Praxis [Problems of evidence-based and evidence-oriented educational practice]. Zeitschrift für Pädagogische Psychologie , 31 (2), 99–110. https://doi.org/10.1024/1010-0652/a000201
(2007). Lehr-Lern-Forschung und Neurowissenschaften – Erwartungen, Befunde und Forschungsperspektiven [Research on learning and instruction and the neuro sciences – expectations, findings, and research perspectives]. Berlin: Bundesministerium für Bildung und Forschung.
(2017). Teachers and their implementation of differentiated instruction in the classroom. Teaching and Teacher Education , 67 , 291–301. https://doi.org/10.1016/j.tate.2017.06.020
(2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review , 31 , 261–292. https://doi.org/10.1007/s10648-019-09465-5
U.S. Department of Education, Office of Elementary and Secondary Education . (2002). No Child Left Behind: A desktop reference , Retrieved June 22, 2019 from http://www2.ed.gov/admins/lead/account/nclbreference/reference.pdf(2017). Socio-cognitive scaffolding with collaboration scripts: a meta-analysis. Educational Psycho69logy Review , 29 (3), 477–511. https://doi.org/10.1007/s10648-016-9361-7
(2011). Assessing teacher candidates' general pedagogical/psychological knowledge: Test construction and validation. Journal of Educational Psychology , 103 (4), 952–969. https://doi.org/10.1037/a0025125
(2018). Förderung wissenschaftlichen Denkens im Lehramtsstudium: Lernen aus eigenen und fremden Fehlern in multiplen und uniformen Kontexten [Fostering scientific reasoning in pre-service teacher education: Learning from own and others' failure in multiple and uniform contexts]. Zeitschrift für Pädagogische Psychologie , 32 (1–2), 5–22. https://doi.org/10.1024/1010-0652/a000219
(2014). Instruktionale Unterstützung beim Lernen aus advokatorischen Fehlern in der Lehramtsausbildung: Effekte auf die Anwendung wissenschaftlichen Wissens [Instructional support during learning from advocatory failure in pre-service teacher education: effects on the application of scientific knowledge]. Psychologie in Erziehung und Unterricht , 61 , 287–301. https://doi.org/10.2378/peu2014.art23d
(2015). Kompetenz angehender Lehrkräfte zum Argumentieren mit Evidenz: Erhöht ein Training von Heuristiken die Argumentationsqualität [Pre-service teachers competences to argue on the basis of evidence: Does a training of heuristics improve argumentation quality?]. Unterrichtswissenschaft , 43 (3), 209–224. https://doi.org/10.3262/UW1503209
(2007). Teachers' approaches to finding and using research evidence: An information literacy perspective. Educational Research , 49 (2), 185–206. https://doi.org/10.1080/00131880701369719
