Scientific Reasoning in Higher Education
Constructing and Evaluating the Criterion-Related Validity of an Assessment of Preservice Science Teachers’ Competencies
Abstract
The aim of this study was to develop a standardized test addressed to measure preservice science teachers’ scientific reasoning skills, and to initially evaluate its psychometric properties. We constructed 123 multiple-choice items, using 259 students’ conceptions to generate highly attractive multiple-choice response options. In an item response theory-based validation study (N = 2,247), we applied multiple regression analyses to test hypotheses based on groups with known attributes. As predicted, graduate students performed better than undergraduate students, and students who studied two natural science disciplines performed better than students who studied only one natural science discipline. In contrast to our initial hypothesis, preservice science teachers performed less well than a control group of natural sciences students. Remarkably, an interaction effect of the degree program (bachelor vs. master) and the qualification (natural sciences student vs. preservice teacher) was found, suggesting that preservice science teachers’ learning opportunities to explicitly discuss and reflect on the inquiry process have a positive effect on the development of their scientific reasoning skills. We conclude that the evidence provides support for the criterion-based validity of our interpretation of the test scores as measures of scientific reasoning competencies.
References
(2005). Reliability as a measurement design effect. Studies in Educational Evaluation, 31, 162–172.
(2006, April). Reliability and item response modelling. Myths, observations and applications. Paper presented at the 13th International Objective Measurement Workshop, Berkeley, CA, USA.
(1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22, 46–75.
(2012). Conquest 3.0 [computer software]. Camberwell, Australia: Australian Council for Educational Research.
. (1993). Benchmarks for science literacy: Project 2061. New York, NY: Oxford University Press.
(2015). Beyond dichotomies: Competence viewed as a continuum. Zeitschrift für Psychologie, 223, doi: 10.1027/2151-2604/a000194
(2007). Applying the Rasch model: Fundamental measurement in the human sciences. London, UK: Erlbaum.
(1993). The philosophy behind physics. Berlin, Germany: Springer.
(1985). Development of an integrated process skill test: TIPS II. Journal of Research in Science Teaching, 22, 169–177.
(2002). Teaching science as inquiry. In , Inquiring into inquiry learning and teaching in science (pp. 20–46). Washington, DC: American Association for the Advancement of Science.
(2005). Dynamic assessments of preservice teachers’ knowledge of models and modelling. In , Research and the quality of science education (pp. 309–323). Dordrecht, The Netherlands: Springer.
(1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
. (2007a). Fachübergreifende Studienordnung für das Masterstudium für das Lehramt
[Interdisciplinary study regulations for the Master of Education program] . Berlin, Germany: Humboldt University. Retrieved from www.amb.hu-berlin.de/2007/99/9920070. (2007b). Studien- und Prüfungsordnung für das Bachelorstudium Biologie: Kernfach Biologie und Beifach Chemie im Monostudiengang. Beifach Biologie im Monostudiengang
[Study regulations for the Bachelor program in biology: Major in biology and minor in chemistry. Minor in biology] . Berlin, Germany: Humboldt University. Retrieved from www.amb.hu-berlin.de/2007/62/6220070. (2007c). Studien- und Prüfungsordnung für das Bachelorstudium Biologie: Kernfach und Zweitfach im Kombinationsstudiengang mit Lehramtsoption
[Study regulations for the Bachelor program in biology: Major and minor in a dual program for preservice teachers] . Berlin, Germany: Humboldt University. Retrieved from www.amb.hu-berlin.de/2007/68/6820070(2013). Two views about explicitly teaching the nature of science. Science and Education, 22, 2109–2139.
(2006). Understanding scientific reasoning. Independence, KY: Wadsworth/Cengage Learning.
(1992). Scientific reasoning across different domains. In , Computer-based learning environments and problem solving (pp. 345–371). Heidelberg, Germany: Springer.
(2003). Theory and reality: An introduction to the philosophy of science. Chicago, IL: University of Chicago Press.
(2010). Practical approaches for choosing multiple-matrix sample designs. IEA-ETS Research Institute Monograph, 3, 125–156.
(1998). Investigative work in the science curriculum. Buckingham, UK: Open University Press.
(1991). Understanding models and their use in science: Conceptions of middle and high school students and experts. Journal of Research in Science Teaching, 28, 799–822.
(1984). Procedures for assessing the validity of tests using the “known groups” method. Applied Psychological Measurement, 8, 295–305.
(2014). Learning science, learning about science, doing science: Different goals demand different learning methods. International Journal of Science Education, 36, 2534–2553. doi: 10.1080/09500693.2014.899722
(2003). Teachers’ views on the nature of models. International Journal of Science Education, 25, 1369–1386.
(2000). Exploring science. The cognition and development of discovery processes. Cambridge, MA: MIT Press.
(2008). Current issues in competence modeling and assessment. Zeitschrift für Psychologie, 216, 61–73.
(1996). Theory and evidence. The development of scientific reasoning. Cambridge, MA: MIT Press.
(2014). Students’ levels of understanding models and modelling in biology: Global or aspect-dependent? Research in Science Education, 44, 109–132.
(1988). The development of scientific thinking skills. San Diego, CA: Academic Press.
(2012). Professionswissen von Lehrkräften der Naturwissenschaften im Kompetenzbereich Erkenntnisgewinnung
[Science teachers’ professional knowledge in scientific inquiry] . (Doctoral dissertation, University of Kassel, Germany). Retrieved from kobra.bibliothek.uni-kassel.de/bitstream/urn:nbn:de:hebis:34-2012012040403/9/DissertationHagenKunz.pdf(2000). Development of scientific reasoning in college biology: Do two levels of general hypothesis-testing skills exist? Journal of Research in Science Teaching, 37, 81–101.
(2010). Using and developing measurement instruments in science education. A Rasch modeling approach. Charlotte, NC: Information Age Publishing.
(2011). Assessing scientific inquiry. Development and evaluation of a test for the low-performing stage. Berlin, Germany: Logos.
(2007). Erkenntnisgewinnung als wissenschaftliches Problemlösen [Inquiry as scientific problem solving] In , Theorien in der biologiedidaktischen Forschung. Ein Handbuch für Lehramtsstudenten und Doktoranden (pp. 177–186). Berlin, Germany: Springer.
. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: National Academy Press.
(2011). Beyond physics content knowledge. Modeling competence regarding nature of science inquiry and nature of scientific knowledge. Berlin, Germany: Logos.
(2003). The logic of scientific discovery. London, UK: Routledge.
(2006). Validity theory In , Encyclopedia of measurement and statistics (pp. 1032–1035). Thousand Oaks, CA: Sage.
(1998). Psychometric models of student conceptions in science: Reconciling qualitative studies and distractor-driven assessment instruments Journal of Research in Science Teaching, 35, 265–296.
(2004). Developing views of nature of science in an authentic context: An explicit approach to bridging the gap between nature of science and scientific inquiry Science Education, 88, 610–645.
(2013). Modellkompetenz im Kontext Biologieunterricht – Empirische Beschreibung von Modellkompetenz mithilfe von Multiple-Choice Items
[Model competence in biology education – empirical description of model competence using multiple-choice items] . (Doctoral dissertation, HU Berlin, Germany). Retrieved from edoc.hu-berlin.de/dissertationen/terzer-eva-2012-12-19/PDF/terzer.pdf(2012). Kompetenzstruktur und -niveaus von Methoden der naturwissenschaftlichen Erkenntnisgewinnung
[Competence structure and levels of methods of scientific inquiry] . Berlin, Germany: Logos.(2010). Developing a paper-and-pencil-test to assess students’ skills in scientific inquiry In , Contemporary science education research: Learning and assessment (pp. 289–294). Ankara, Turkey: ESERA.
(2005). The role of plausible values in large-scale surveys Studies in Educational Evaluation, 31, 114–128.
(2007). The development of scientific thinking skills in elementary and middle school Developmental Review, 27, 172–223.
