The ESBW Short Scale
A Test for Assessing Teachers’ Standards-Based Educational Knowledge
Abstract
Abstract: Teacher students’ professional educational knowledge is of great importance in academic teacher education. In response to the need to continuously optimize and improve teacher education, we developed a standards-based test instrument designed along the Standards of Teacher Education of the German education administration. The so-called ESBW (Essen Test for the Assessment of Standards-Based Educational Knowledge) is intended to assess educational knowledge as it is defined in these standards. This Brief Report aims to investigate whether the ESBW, as an exclusively standards-based developed test, can empirically be distinguished from a similar, but non-originally standards-based developed test, here the BilWiss 2.0 test, which also partially covers the standards. Competing structural equation models based on a study with 216 teacher students revealed that the ESBW short scale can be empirically distinguished from the BilWiss 2.0 short version, indicating that both instruments partly measure different aspects of educational knowledge. In addition, the examination of measurement invariance revealed that the ESBW performed similarly well for both beginning and advanced teacher students. Thus, our results further underline the usefulness of the ESBW for the assessment and evaluation of the German Standards of Teacher Education.
Educational knowledge of teachers refers to knowledge that includes pedagogical (Shulman, 1986), psychological, and sociological aspects as well as knowledge about the teaching profession, school development, and educational research (Linninger et al., 2015). Several studies have shown that this type of knowledge is of great importance for both teaching students and experienced teachers, as it can increase the process and product quality of their teaching (e.g., Baumert et al., 2010; Darling-Hammond et al., 2001; König & Pflanzl, 2016; Voss et al., 2011) as well as protect them from emotional exhaustion (Dicke et al., 2015). Educational knowledge as part of the professional competence of (prospective) teachers thus provides a necessary foundation for later professional behavior. According to the standards for teacher education of the German Education Administration (hereinafter called KMK standards; KMK, 2019) all teacher students are supposed to acquire the core educational concepts during their studies, regardless of the type of school or the subjects in which they intend to teach.
This brief report aims to compare two similar test instruments, both of which measure the theoretical knowledge base regarding educational knowledge but have different operationalizations and development histories. These are the ESBW (Essen Test for the Assessment of Standards-Based Educational Knowledge; short version; Müser et al., 2022) and the BilWiss 2.0 test (short version; Kunina-Habenicht et al., 2020). Our study aimed to examine whether the ESBW, which was exclusively developed along the KMK standards, assesses different aspects of educational knowledge compared to the BilWiss 2.0 test, which covers a broader range of topics. Furthermore, it should be evaluated whether both tests can be empirically distinguished from each other.
The ESBW comprises 40 items on a short scale and is intended to capture educational knowledge on the basis of the KMK standards. These national standards in Germany, which are comparable to the American Core Teaching Standards (Council of Chief State School Officers [CCSSO], 2011), describe competencies in the areas of teaching, educating, assessing, and innovating that teachers should acquire during the university teacher program. The standards show a high acceptance within the professional discourse (Lohmann et al., 2011; Terhart, 2014). However, in contrast to the American standards, the German national standards have not yet been used for evaluation purposes in teacher education, although they were originally also designed for this purpose (Darling-Hammond, 2001; Hohenstein et al., 2014).
To pursue the goal of modification and improvement of teacher education, it seems profitable to measure educational knowledge in terms of these mandatory standards. The KMK standards formulate educational standards that are to be achieved cumulatively throughout the course of teacher education. Therefore, the administration of the ESBW may be useful both at the end of university studies and for interim screenings. The ESBW is strongly based on the structure of the standards. The items of the test (see Figure 1 for an example item) are assigned to the four competence areas of the KMK standards and are based very closely on the requirements formulated in each area (for a detailed description, see Table A1 in the Appendix).
The ESBW was developed exclusively for the purpose of assessing the mandatory KMK standards and enabling their evaluation. The ESBW was initially reviewed by experts for content validity. The 40 items were assessed by 4 experts from the field of teacher education and school development. The experts were presented with an evaluation sheet (see, e.g., Jenßen et al., 2015), in which the fit of each item was assessed with respect to (a) the intended competence (defined in the KMK standards), (b) the correctness of the item content, and (c) the plausibility of the distractors. Only those items were included where agreement on the assignment to the intended competence, the correctness of the content, and the plausibility of the distractors were achieved. Further empirical validation with teacher students indicated that the items cover the competencies of the KMK standards well and can map them on a one-dimensional short scale. Detailed testing and validation of the ESBW are described by Müser et al. (2022), in which various indications of validity are reviewed and reported, so the results of an overall validation are not covered in this Brief Report. In summary, it could be shown that the ESBW has a good psychometric quality and allows to capture of the educational knowledge of teacher students in the four competency areas of the KMK standards. The results of the expert rating on the content fit of the items, the results on the internal structure, as well as the theory-compliant correlations with other instruments and selected personal characteristics, indicate that the ESBW is suitable to validly capture the educational knowledge of prospective teachers.
The BilWiss 2.0 short test version consists of 65 items (Kunina-Habenicht et al., 2020) and is also intended to capture educational knowledge, but based on a Delphi study (Kunina-Habenicht et al., 2012). The BilWiss 2.0 test is a revised version of the initial BilWiss test in which the items and their dimensions have been slightly modified (Kunina-Habenicht et al., 2012; Linninger et al., 2015; Lohse-Bossenz et al., 2013). The revised BilWiss 2.0 test includes items from six knowledge domains: instruction, learning and development, assessment, educational theory, school as an educational institution, and teaching as a profession. Further information on the test can be found in Kunter et al. (2020). The development of this test is a university-internal approach through the Delphi study, which is less oriented towards external standards or criteria, although the KMK standards were used to pre-structure potentially relevant content (Kunina-Habenicht et al., 2012). But the BilWiss 2.0 test was not exclusively designed based on the KMK standards and was not created to evaluate the standards. Nevertheless, Kunina-Habenicht et al. (2019) showed by an expert rating that the items of the BilWiss 2.0 test could also be assigned to the KMK standards to a large extent. However, this rating also showed that many items fit several standards and that not all competencies are covered by the items of the test. A concrete and comprehensive examination of the KMK standards would therefore not be possible with the BilWiss 2.0 test.
In summary, the consistent implementation of the structure and requirements of the KMK standards, the development to evaluate the standards and the corresponding approach for generating the items distinguishes the ESBW from the BilWiss 2.0 test. Nevertheless, the question arises whether the ESBW (as an exclusively standards-based developed instrument) and the BilWiss 2.0 test (as a not exclusively standards-based developed instrument) can be empirically separated from each other. An empirical distinction of both instruments would deliver a further argument for the usefulness of a test (such as the ESBW) that is specifically designed for assessing and evaluating the KMK standards. This separability of the ESBW from the BilWiss 2.0 test is a necessary, although not sufficient, condition for the validity of intended test score interpretation and application of the ESBW. If the ESBW and the BilWiss 2.0 test cannot be distinguished, the usefulness of ESBW would have to be questioned. Therefore, we assume that both instruments measure educational knowledge but capture slightly different aspects of it due to differences in operationalization. Thus, we compared two structural equation models: a one-factor model, in which the items of both instruments were represented by a common factor, and a two-factor model, in which the items of both instruments were represented by two separate but correlated factors. We expected the two-factor model to show a better model fit and a moderate to high correlation between the two latent factors representing the ESBW and the BilWiss 2.0 test.
In addition, to examining the empirical distinction to query the reasonableness of the existence and use of the ESBW, this brief report also examines the question of whether the ESBW performs well for both beginning and advanced teacher students. This could also be an argument for the validity of the intended test-score interpretation and application of the ESBW. Therefore, we tested whether the ESBW was invariant across teacher students in the first and third semesters of their studies.
Method
Participants
The study involved 216 teacher students enrolled in the bachelor’s program of a German university (66% female, 33% male, 1% divers; Mage = 22.8 years, SD = 4.0 years). About half of the students aimed to teach at upper secondary schools (51%), whereas the other half aimed to teach at lower secondary (21%) or primary (24%) schools; only a few students (4%) aimed to teach at vocational schools. Most students reported being in the first (38%) or third (52%) semester of the bachelor’s program, with a few in the 5th or higher semester (10%).
Materials and Procedure
To measure educational knowledge, the ESBW (40 items; 0 = incorrect, 1 = partially correct, 2 = completely correct) and the short version of the BilWiss 2.0 test (65 items; same scoring system) were used in paper-pencil versions. Information about the study’s progress and sociodemographic data were collected as well. The procedure took about 80 minutes and was conducted during a lecture on educational science at the university, which is attended by all teacher students, regardless of studied subjects and type of school. Participation was voluntary, and participants were not compensated financially or otherwise.
Data Analysis
Two structural equation models were compared to investigate the separability of the two test instruments. One way to compare or distinguish the two tests and their possibly differing aspects of educational knowledge were to estimate a two-factor model (Figure 2A). In this model, the items of the BilWiss 2.0 test formed a first factor, and the items of the ESBW formed a second factor correlated with the first one. A second possibility, not assuming a distinction between the two test instruments, was to estimate a one-factor model (Figure 2B), in which the items of both test instruments were represented by a common factor. Both models were compared in terms of model fit indices using conventional cut-off criteria (χ2, preferably not significant; CFI ≥ .95; RMSEA ≤ .06; SRMR ≤ .08; TLI ≥ .95; Hu & Bentler, 1999; Marsh et al., 2004; O’Boyle & Williams, 2011), and differences in model fit were tested for statistical significance, using the likelihood ratio test (Burnham & Anderson, 2004).
With respect to the test for measurement invariance, starting from the least restrictive form of measurement invariance (configural invariance), two increasingly restrictive models (metric and scalar measurement invariance) were tested in a multigroup analysis. Each model was compared with the previous one in terms of change of model fit using the cut-off criteria ΔCFI ≥ .010 and ΔRMSEA ≥ .015 as an indication of a substantial decrease in model fit (Chen, 2007; Cheung & Rensvold, 2001).
All analyses were conducted with the software R (R Core Team, 2016) and R-packages lavaan (Rosseel, 2012), semTools (Jorgensen et al., 2018), MBESS (Kelley, 2017), and psych (Revelle, 2020).
Results
Students performed moderately on both tests (ESBW: M = 30.03, SD = 9.69; BilWiss: M = 57.31, SD = 14.56; N = 216). Descriptive analyses of the single items and their factor loadings can be found in Table A2 of the Appendix. The results for the fit values for the one-factor and the two-factor model are shown in Table 1.
While both models represented the data well in terms of CFI, RMSEA, SRMR, and TLI, only the two-factor model showed a not significant χ2-value. The result of the likelihood ratio test also indicated a significantly better fit of the two-factor model (Δχ2(Δdf = 1) = 277; p < .001). Standardized factor loadings were, except for one item in the BilWiss 2.0 test, all significant for both models. The two-factor model showed a moderate correlation between both factors (r = .61) providing further support for this model.
Table 2 shows the results of the model comparisons to test for measurement invariance of the ESBW for the two levels of study progress (first vs. third semester). While results regarding configural measurement invariance indicated good fit, the factor loadings of 22 items (partial metric) and the intercept of one item in both groups had to be freely estimated to reach the cut-off criteria for the change in model fit to assume partial scalar invariance (Byrne et al., 1989; Putnick & Bornstein, 2016).
Discussion
Overall, satisfactory results were achieved in the modeling analyses. In line with expectations, the two-factor model shows a better model fit than the one-factor model, and the factors in the two-factor model representing the ESBW and the BilWiss 2.0 test are moderately correlated. This indicates that the exclusively standards-based developed ESBW and the not exclusively standards-based developed BilWiss 2.0 test can be distinguished from each other as they appear to measure similar, but not identical aspects of educational knowledge. Therefore, the results further support the usefulness of the ESBW in order to specifically assess educational knowledge as it is defined in the KMK standards of German teacher education whereas the BilWiss 2.0 test covers many but not all aspects of educational knowledge listed in the KMK standards, as it was not developed to fully cover the KMK standards (Kunina-Habenicht et al., 2019).
Moreover, at least full configural measurement invariance can be assumed, which is sufficient to state that the model structure applies equally well to both groups. For assuming more restrictive models, half of the factor loadings as well as one intercept had to be freely estimated, in order to assume partial scalar measurement invariance. This might limit the validity of potential performance comparisons between both groups of students even though unequal factor loadings seem to have a negligible effect on factor mean differences compared to unequal intercepts (Steinmetz, 2013).
According to the results, tests such as the ESBW would allow to integrate of the KMK standards more closely into examinations in teacher education programs, especially at its end, as Terhart (2014) suggested.
One limitation of the present study refers to the used sample of students in their first and third semesters, which might have had an influence on the results (since the complete achievement of the standards cannot be expected before teacher education studies at university are completed). However, we did not find any floor effects with mean test scores ranging approximately in the middle of the scale both for the ESBW and the BilWiss 2.0 test. Furthermore, it should be mentioned that we presented each of the two tests as a testlet. We varied the order of the testlets, but not the order of the items. The fact that we did not randomize the presentation at the item level may also have influenced the results. Moreover, the sample being from only one university might restrict the generalizability of the results, which should be addressed in future research.
References
(2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress. American Education Research Journal, 47(1), 133–180. https://doi.org/10.3102/0002831209345157
(2004). Multimodel inference: Understanding AIC and BIC in Model Selection. Sociological Methods & Research, 33(2), 261–304. https://doi.org/10.1177/0049124104268644
(1989). Testing for equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456–466. https://doi.org/10.1037/0033-2909.105.3.456
(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
(2001). The effects of model parsimony and sampling error on the fit of structural equation models. Organizational Research Methods, 4(3), 236–264. https://doi.org/10.1177/109442810143004
. (2011). Interstate teacher assessment and support consortium (InTASC) model of core teaching standards: A resource for state dialogue. CCSSO. https://ccsso.org/sites/default/files/2017-11/InTASC_Model_Core_Teaching_Standards_2011.pdf
(2001).
Standards setting in teaching: Changes in licensing, certification, and assessment . In V. RichardsonEd., Handbook of research on teaching (4th ed., pp. 751–776). American Educational Research Association.(2001). Does teacher certification matter? Evaluating the evidence. Educational Evaluation and Policy Analysis, 23(1), 57–77. https://doi.org/10.3102/01623737023001057
(2015). Beginning teachers’ efficacy and emotional exhaustion: Latent changes, reciprocity, and the influence of professional knowledge. Contemporary Educational Psychology, 41, 62–72. https://doi.org/10.1016/j.cedpsych.2014.11.003
(2014). Sind die bildungswissenschaftlichen Standards für die Lehramtsausbildung in den Curricula der Hochschulen angekommen?
[Have the standards for teacher education arrived in universities’ curricula?] . Zeitschrift für Erziehungswissenschaft, 17, 497–507. https://doi.org/10.1007/s11618-014-0563-9(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
(2015).
Qualitätssicherung in der Kompetenzforschung: Empfehlungen für den Nachweis von Validität in Testentwicklung und Veröffentlichungspraxis [Quality assurance in competency research: Recommendations for demonstrating validity in test development and publication] . In S. BlömekeO. Zlatkin-TroitschanskaiaEds., Kompetenzen von Studierenden[Competencies of students] (pp. 11–31). Beltz. https://doi.org/10.25656/01:15501(2018). semTools: Useful tools for structural equation modeling. R package version 0.5-1. https://CRAN.R-project.org/package=semTools
(2017). MBESS (Version 4.0.0 and higher) [Computer software and manual]. https://CRAN.R-project.org/package=MBESS
. (2019). Standards der Lehrerbildung: Bildungswissenschaften
[Standards of teacher education: Educational science] . https://www.kmk.org/fileadmin/Dateien/veroeffentlichungen_beschluesse/2004/2004_12_16-Standards-Lehrerbildung-Bildungswissenschaften.pdf(2016). Is teacher knowledge associated with performance? On the relationship between teachers’ general pedagogical knowledge and instructional quality. European Journal of Teacher Education (39(4), 419–436). https://doi.org/10.1080/02619768.2016.1214128
(2012). Welche bildungswissenschaftlichen Inhalte sind wichtig in der Lehrerbildung?
[What educational content is important in teacher education?] Zeitschrift für Erziehungswissenschaft, 15, 649–682. https://doi.org/10.1007/s11618-012-0324-6(2019). Zur curricularen Validität des BilWiss 2.0-Tests zur Erfassung des bildungswissenschaftlichen Wissens von (angehenden) Lehrkräften
[On the curricular validity of the BilWiss 2.0 test for assessing the educational science knowledge of (prospective) teachers] . Zeitschrift für Pädagogik, 65(4), 542–555. https://doi.org/10.25656/01:23994(2020). Der BilWiss-2.0-Test: Ein revidierter Test zur Erfassung des bildungswissenschaftlichen Wissens von (angehenden) Lehrkräften
[The BilWiss-2.0-Test: A revised test to assess the educational knowledge of (prospective) teachers] . Diagnostica, 66(2), 80–92. https://doi.org/10.1026/0012-1924/a000238(2020).
Putting educational knowledge of future teachers to the test – Further development and validation of the BilWiss Test . In O. Zlatkin-TroitschanskaiaH. A. PantM. ToepperC. LautenbachEds., Student learning in German higher education (pp. 9–28). Springer.(2015). Assessing teachers’ educational knowledge: Construct specification and validation using mixed methods. Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie, 47(2), 62–74. https://doi.org/10.1026/0049-8637/a000126
(2011). Bildungswissenschaften in der universitären Lehrerbildung: Curriculare Strukturen und Verbindlichkeiten. Eine Analyse aktueller Studienordnungen an nordrhein-westfälischen Universitäten
[Educational Sciences in Teacher Education: Curricular Structures and Liabilities. An analysis of current study regulations at universities in North Rhine-Westphalia] . Lehrerbildung auf dem Prüfstand, 4(2), 271–302. https://doi.org/10.25656/01:14724(2013). The role of educational psychology in teacher education: Expert opinions on what teachers should know about learning, development, and assessment. European Journal of Psychology of Education, 28, 1543–1565. https://doi.org/10.1007/s10212-013-0181-6
(2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11(3), 320–341. https://doi.org/10.1207/s15328007sem1103_2
(2020). Die Erfassung des bildungswissenschaftlichen Wissens im Lehramtsstudium: Konstruktion und Validierung des ESBW-Tests sowie die Untersuchung des Praxisschocks im Praxissemester
[The assessment of educational knowledge in teacher education: Construction and validation of the ESBW Test and the study of practice shock in the student teaching semester] (Dissertation) . Universität Duisburg-Essen. https://duepublico2.uni-due.de/receive/duepublico_mods_00073611(2022). Entwicklung und Validierung des Essener Tests zur Erfassung des standardorientierten bildungswissenschaftlichen Wissens (ESBW)
[The construction and validation of the Essen Test for the Assessment of Standards-Based Educational Knowledge (ESBW)] . Diagnostica. Advance online publication. https://doi.org/10.1026/0012-1924/a000303(2011). Decomposing model fit: Measurement vs. theory in organizational research using latent variables. Journal of Applied Psychology, 96(1), 1–12. https://doi.org/10.1037/a0020539
(2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004
. (2016). R: A language and environment for statistical computing. https://www.R-project.org/
(2020). Package “psych”. R package version 0.5–1. https://cran.r-project.org/web/packages/psych/psych.pdf
(2012). lavaan: An R Package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
(1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. https://doi.org/10.3102/0013189X015002004
(2013). Analyzing observed composite differences across multiple groups: Is partial measurement invariance enough? Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 9, 1–12. https://doi.org/10.1027/1614-2241/a000049
(2014). Standards für die Lehrerbildung: Bildungswissenschaften – nach zehn Jahren
[Standards for teacher education: Educational sciences – After 10 years] . Die Deutsche Schule, 106(4), 300–323. https://doi.org/10.25656/01:25897(2011). Assessing teacher candidates’ general pedagogical/psychological knowledge: Test construction and validation. Journal of Educational Psychology, 103(4), 952–969. https://doi.org/10.1037/a0025125
Appendix
