Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Originalia

Assessing Teachers’ Educational Knowledge

Construct Specification and Validation Using Mixed Methods

Published Online:April 10, 2015https://doi.org/10.1026/0049-8637/a000126

Abstract

This paper presents two studies adding validity evidence to the assessment of educational knowledge (EK) measured by a recently developed test. The first study compared the test scores of a large sample of graduates from academic teacher education with those of a sample of first-semester teacher students. Results indicate higher scores for graduates in the majority of the test’s six domains. Additionally, we gained validity evidence based on response processes by conducting 46 cognitive interviews in Study 2. Participants with different professional backgrounds stated their response strategies while taking the knowledge test. Among other results, analyses revealed that graduates and advanced students in teacher education were often familiar with item topics and solved items mostly by retrieving academic knowledge from memory. Overall, results add to previously gained evidence suggesting that test scores can be taken to measure EK mainly gained by university studies.


Messung des bildungswissenschaftlichen Wissens von Lehrkräften: Konstruktspezifikation und Validierungsansätze

Im vorliegenden Beitrag werden zwei Studien vorgestellt, die Hinweise auf die Validität der Messung von bildungswissenschaftlichem Wissen anhand eines neu entwickelten Tests geben. Studie 1 verglich eine große Stichprobe von Referendar(inn)en mit einer Stichprobe von Lehramtsstudierenden im ersten Semester. Dabei wurden von Referendar(inn)en höhere Werte auf der Mehrheit der sechs Testskalen erzielt. In Studie 2 war die Untersuchung von kognitiven Prozessen während der Testbearbeitung von Interesse. Es wurden 46 kognitive Interviews durchgeführt, bei denen Teilnehmer(innen) mit unterschiedlichem professionellen Hintergrund ihre Antwortstrategien beschrieben, während sie den Wissenstest bearbeiteten. Qualitative Analysen zeigten beispielsweise, dass Teilnehmer(inne)n mit abgeschlossenem oder fortgeschrittenem Lehramtsstudium die Themen der Aufgaben häufig bekannt waren und sie richtig gelöste Aufgaben mehrheitlich mit Universitätswissen lösten. Insgesamt weisen die Ergebnisse darauf hin, dass die Testwerte im Sinne universitären bildungswissenschaftlichen Wissens interpretiert werden können.

References

  • Bauer, J., Diercks, U., Rösler, L., Möller, J., & Prenzel, M. (2012). Lehramtsausbildung in Deutschland: Wie groß ist die strukturelle Vielfalt? [Teacher education in Germany: How large is the structural variety?] Unterrichtswissenschaft, 40, 101 – 120. First citation in articleGoogle Scholar

  • Baumert, J., & Kunter, M. (2013). The COACTIV model of teachers’ professional competence. In M. KunterJ. BaumertW. BlumU. KlusmannS. KraussM. NeubrandEds., Cognitive activation in the mathematics classroom and professional competence of teachers. Results from the COACTIV project (pp. 25 – 48). New York, NY: Springer. First citation in articleCrossref, Google Scholar

  • Cochran-Smith, M., & Zeichner, K. M. (Eds.). (2005). Studying teacher education: The report of the AERA Panel on Research and Teacher Education. Washington, DC: American Educational Research Association. First citation in articleGoogle Scholar

  • Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155 – 159. doi: 10.1037//0033-2909.112.1.155 First citation in articleCrossref, Google Scholar

  • Dicke, T., Parker, P. D., Holzberger, D., Kunina-Habenicht, O., Kunter, M., & Leutner, D. (2014). Beginning teachers’ efficacy and emotional exhaustion: Latent changes, reciprocity, and the influence of professional knowledge. Manuscript submitted for publication. First citation in articleGoogle Scholar

  • Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis. Cambridge, MA: MIT Press. First citation in articleCrossref, Google Scholar

  • Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378 – 382. First citation in articleCrossref, Google Scholar

  • Floden, R. E. (2002). The measurement of opportunity to learn. In A. C. PorterA. GamoranNational Research Council/Board on International Comparative Studies in EducationEds., Methodological advances in cross-national surveys of educational achievement (pp. 231 – 266). Washington, DC: National Academy Press. First citation in articleGoogle Scholar

  • Hamilton, L., Nussbaum, E., & Snow, R. (1997). Interview procedures for validating science assessments. Applied Measurement in Education, 10, 181 – 200. doi: 10.1207/s15324818ame1002_5 First citation in articleCrossref, Google Scholar

  • Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge. First citation in articleGoogle Scholar

  • International Association for the Evaluation of Educational Achievement. (2011). Retrieved April 23, 2014, from http://www.iea.nl/studies.html First citation in articleGoogle Scholar

  • König, J., Blömeke, S., Paine, L., Schmidt, W. H., & Hsieh, F.-J. (2011). General pedagogical knowledge of future middle school teachers: On the complex ecology of teacher education in the United States, Germany, and Taiwan. Journal of Teacher Education, 62, 188 – 201. doi: 10.1177/0022487110388664 First citation in articleCrossref, Google Scholar

  • Kunina-Habenicht, O., Kunter, M., Leutner, D., Terhart, E., Baumert, J., Lohse, H. et al. (2012). Welche bildungswissenschaftlichen Inhalte sind wichtig in der Lehrerbildung? [What educational science content is important for teacher training?] Zeitschrift für Erziehungswissenschaft, 15, 649 – 682. doi: 10.1007/s11618-012-0324-6 First citation in articleCrossref, Google Scholar

  • Kunina-Habenicht, O., Lohse-Bossenz, H., Dicke, T., Holzberger, D., Linninger, C., Schulze-Stocker, F. et al. (2014). Construction and validation of a test for teacher candidates’ educational knowledge. Manuscript in preparation. First citation in articleGoogle Scholar

  • Kunter, M., Kunina-Habenicht, O., Dicke, T., Holzberger, D., Lohse-Bossenz, H., Leutner, D. et al. (in press). Bildungswissenschaftliches Wissen und professionelle Kompetenz in der Lehramtsausbildung–Ergebnisse des Projekts BilWiss [Educational knowledge and professional competence in teacher education–Results of the project BilWiss]. In C. GräselK. TremplerEds., Entwicklung von Professionalität pädagogischen Personals. Interdisziplinäre Betrachtungen, Befunde und Perspektiven [Development of the professionality of educational staff]. Springer Online. First citation in articleGoogle Scholar

  • Kuusela, H., & Pallab, P. (2000). A comparison of concurrent and retrospective verbal protocol analysis. The American Journal of Psychology, 113, 387 – 404. doi: 10.2307/1423365 First citation in articleCrossref, Google Scholar

  • Lohse-Bossenz, H., Kunina-Habenicht, O., & Kunter, M. (2013). The role of educational psychology in teacher education: Expert opinions on what teachers should know about learning, development, and assessment. European Journal of Psychology of Education, 28, 1543 – 1565. doi: 10.1007/s10212-013-0181-6 First citation in articleCrossref, Google Scholar

  • Mayring, P. (2010). Qualitative Inhaltsanalyse: Grundlagen und Techniken [Qualitative content analysis: Foundations and techniques]. Weinheim, Germany: Beltz. First citation in articleCrossref, Google Scholar

  • Merzyn, G. (2002). Stimmen zur Lehrerausbildung. Ein Überblick über die Diskussion [Voices on teacher education. A review on the discussion]. Hohengehren, Germany: Schneider Verlag. First citation in articleGoogle Scholar

  • Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159 – 176. doi: 10.1177/014662169201600206 First citation in articleCrossref, Google Scholar

  • Muthén, L. K., & Muthén, B. O. (1998 – 2010). Mplus user’s guide. Los Angeles, CA: Muthén & Muthén. First citation in articleGoogle Scholar

  • Presser, S., Couper, M. P., Lessler, J. T., & Martin, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68, 109 – 130. doi: 10.1093/poq/nfh0080 First citation in articleCrossref, Google Scholar

  • Prüfer, P., & Rexroth, M. (2005). Kognitive Interviews [Cognitive interviews]. ZUMA How-to-Reihe (Vol. 15). Mannheim, Germany: Zentrum für Umfragen, Methoden und Analysen. First citation in articleGoogle Scholar

  • Schmidt, W. H., Cogan, L., & Houang, R. (2011). The role of opportunity to learn in teacher preparation: An international context. Journal of Teacher Education, 62, 138 – 153. doi: 10.1177/0022487110391987 First citation in articleCrossref, Google Scholar

  • Schulze-Stocker, F., Holzberger, D., & Terhart, E. (2014). Spielen Studienschwerpunkte wirklich eine Rolle? Zum Zusammenhang von bildungswissenschaftlichen Studienschwerpunkten, selbsteingeschätzten Kenntnissen und gemessenem Wissen am Ende eines Lehramtsstudiums [Do major fields of study really matter? On the relationship of educational major fields of study, self-assessed knowledge, and measured knowledge at the end of academic teacher education]. Manuscript submitted for publication. First citation in articleGoogle Scholar

  • Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4 – 21. doi: 10.3102/0013189X015002004 First citation in articleCrossref, Google Scholar

  • Terhart, E., Lohmann, V., & Seidel, V. (2010). Die bildungswissenschaftlichen Studien in der universitären Lehrerbildung. Eine Analyse aktueller Studienordnungen und Modulhandbücher an Universitäten in Nordrhein-Westfalen [Educational studies in academic teacher education. An analysis of current study regulations and module descriptions at universities in North Rhine-Westphalia]. Unpublished manuscript, University of Münster. First citation in articleGoogle Scholar

  • Timperley, H., Wilson, A., Barrar, H., & Fung, I. (2007). Teacher professional learning and development: Best evidence synthesis iteration. Wellington, New Zealand: Ministry of Education. First citation in articleGoogle Scholar

  • Tittle, C. K. (2006). Assessment of teacher learning and development. In P. A. AlexanderP. H. WinneEds., Handbook of educational psychology (pp. 953 – 980). Mahwah, NJ: Lawrence Erlbaum Associates. First citation in articleGoogle Scholar

  • Veenman, S. (1984). Perceived problems of beginning teachers. Review of Educational Research, 54, 143 – 178. doi: 10.3102/00346543054002143 First citation in articleCrossref, Google Scholar

  • VERBI. (1989 – 2013). MAXQDA Software für qualitative Datenanalyse [MAXQDA software for qualitative data analysis]. Berlin, Germany: Consult, Sozialforschung GmbH. First citation in articleGoogle Scholar

  • Voss, T., Kunter, M., & Baumert, J. (2011). Assessing teacher candidates’ general pedagogical and psychological knowledge: Test construction and validation. Journal of Educational Psychology, 103, 952 – 969. doi: 10.1037/a0025125 First citation in articleCrossref, Google Scholar