Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

Validating a Test for Measuring Knowledge and Understanding of Economics Among University Students

Published Online:August 26, 2019https://doi.org/10.1024/1010-0652/a000239

Abstract

Abstract. University students' knowledge and understanding of economics have mostly been investigated cross-sectionally; however, longitudinal analyses are needed to determine which factors influence knowledge development and to draw valid conclusions based on test results. In the WiWiKom II project, a quasi-experimental longitudinal study was conducted assessing bachelor students of business and economics over the course of their studies in Germany (N = 39 universities). In this project, running from 2016 – 2019, the test-takers complete the WiWiKom II-test of economic knowledge (adapted TUCE4G and TEL4G items) and a general cognitive ability test (BEFKI 11) at four measurement points, each one year apart. In this paper, we describe the validation of the test instrument based on the data from the first measurement (winter semester 2016/17). We compare students' results on the economic knowledge test and the general cognitive abilities test between first-year students of economic sciences (N = 3,710) and social sciences (N = 1,347) to determine the discriminant validity of the economic knowledge test. The findings from the confirmatory factor analyses presented here show that the items on general cognitive ability and economic knowledge are empirically separable. As expected, there were no differences in the factorial structure between the comparison groups (economic vs. social science students) at the beginning of their studies.


Validierung eines Tests zur Messung von ökonomischem Wissen und Verstehen bei Studierenden der Wirtschaftswissenschaften

Zusammenfassung. Das ökonomische Wissen und Verstehen wurde in bisherigen Studien meist querschnittlich erfasst. Um zu untersuchen, welche Faktoren den Erwerb des Wissens im Verlauf des Studiums beeinflussen und valide Testwertinterpretationen ziehen zu können, sind längsschnittliche Studien notwendig. In dem Projekt WiWiKom II werden mit einem quasi-experimentellen Studiendesign Studierende der Wirtschaftswissenschaften und der Sozialwissenschaften im Verlauf des Bachelorstudiums in einem Large-Scale-Design (N = 39 Universitäten) erfasst. In der Studie mit einer vierjährigen Laufzeit (2016 – 2019) werden die Studierenden mit dem WiWiKom-Test zur Erfassung des ökonomischen Wissens und Verstehens (ins Deutsche adaptierte Items aus dem amerikanischen TUCE4 und dem amerikanischen TEL4) sowie zur allgemeinen kognitiven Leistungsfähigkeit (BEFKI 11) über vier Messzeitpunkte im Abstand von einem Jahr befragt. Der vorliegende Beitrag beschreibt die Validierung der eingesetzten Testinstrumente zum ersten Messzeitpunkt (Wintersemester 2016 / 17). Im Rahmen der diskriminanten Validierung betrachten wir Befunde zum ökonomischen Wissenstest und zur kognitiven Leistungsfähigkeit bei Erstsemesterstudierenden der Wirtschaftswissenschaften (N = 3.710) und der Sozialwissenschaften (N = 1.347). Die Dimensionalitätsanalysen mittels konfirmatorischer Faktorenanalysen zeigen, dass sich die Items zur allgemeinen kognitiven Leistungsfähigkeit von dem ökonomischen Wissen faktor-analytisch trennen lassen. Zwischen den Vergleichsgruppen (Studierende der Wirtschafts- vs. Sozialwissenschaften) gibt es zu Studienbeginn erwartungsgemäß keine Unterschiede in der faktoriellen Struktur.

References

  • American Educational Research Association (AERA), American Psychological Association (APA) & National Council on Measurement in Education (NCME) (2014). The standards for educational and psychological testing. Washington, DC: American Psychological Association. First citation in articleGoogle Scholar

  • Bacdayan, P. (1996). Characterizing expert leaders of quality improvement task forces: Towards applying research on expertise to training design. Academy of Management Proceedings, 1, 181 – 185. First citation in articleCrossref, Google Scholar

  • Beck, K. & Wuttke, E. (2004). Eingangsbedingungen von Studienanfängern – Die prognostische Validität wirtschaftskundlichen Wissens für das Vordiplom bei Studierenden der Wirtschaftswissenschaften. Zeitschrift für Berufs- und Wirtschaftspädagogik, 100 (1), 116 – 124. First citation in articleGoogle Scholar

  • Beck, K., Krumm, V. & Dubs, R. (1998). Wirtschaftskundlicher Bildungs-Test (WBT). Göttingen: Hogrefe. First citation in articleGoogle Scholar

  • Black, H. T. & Duhon, D. L. (2003). Evaluating and improving student achievement in business programs: The effective use of standardized assessment tests. Journal of Education for Business, 79 (2), 90 – 98. First citation in articleCrossref, Google Scholar

  • Bothe, T. Wilhelm, O. & Beck, K. (2006). Business administration knowledge. Assessment of declarative business administration knowledge: Measurement development and validation. Unpublished manuscript. First citation in articleGoogle Scholar

  • Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: Guilford. First citation in articleGoogle Scholar

  • Brückner, S., Förster, M., Zlatkin-Troitschanskaia, O., & Walstad, W. B. (2015). Effects of prior economic education, native language, and gender on economic knowledge of first-year students in higher education. A comparative study between Germany and the USA. Studies in Higher Education, 40 (3), 437 – 453. First citation in articleCrossref, Google Scholar

  • Brückner, S., & Pellegrino, J. W. (2016). Integrating the Analysis of Mental Operations Into Multilevel Models to Validate an Assessment of Higher Education Students' Competency in Business and Economics. Journal of Educational Measurement, 53, 293 – 312. doi:10.1111/jedm.12113 First citation in articleCrossref, Google Scholar

  • Brückner, S., & Pellegrino, J. W. (2017). Contributions of Response Processes Analysis to the Validation of an Assessment of Higher Education Students' Competence in Business and Economics. In B. ZumboA. Hubley (Eds.), Understanding and Investigating Response Processes in Validation Research (pp. 31 – 52). New York: Springer. First citation in articleGoogle Scholar

  • Christ, O. & Schlüter, E. (2012). Strukturgleichungsmodelle mit Mplus. Eine praktische Einführung. München: Oldenbourg. First citation in articleCrossref, Google Scholar

  • Collins, L. M., Schafer, J. L. & Kam, C.-M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6 (4), 330 – 351. First citation in articleCrossref, Google Scholar

  • Cox, S. S., Charles, L., Chen, L. J.-S. & Totten, L. J. (2011). Is it more than just GPA? An examination of work experience and test preparation effects on MFT-B scores. Journal of Academic Administration in Higher Education, 7 (1), 53 – 70. First citation in articleGoogle Scholar

  • Cronbach, L. J. & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological bulletin, 52 (4), 281 – 302. First citation in articleCrossref, Google Scholar

  • de Ayala, R. J. (2009). The theory and practice of item response theory. New York / London: The Guildford Press. First citation in articleGoogle Scholar

  • Finch, H. (2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT Likelihood Ratio. Applied Psychological Measurement, 29 (4), 278 – 295. First citation in articleCrossref, Google Scholar

  • Förster M., Zlatkin-Troitschanskaia, O., Brückner, S., Happ, R., Hambleton, R. K., Walstad, W. B., (,... )& Yamaoka, M. (2015a). Validating Test Score Interpretations by CrossNational Comparison: Comparing the Results of Students From Japan and Germany on an American Test of Economic Knowledge in Higher Education. Zeitschrift für Psychologie, 223 (1), 14 – 23. First citation in articleLink, Google Scholar

  • Förster, M., Zlatkin-Troitschanskaia, O., & Happ, R. (2015b). Adapting and Validating the Test of Economic Literacy to Assess the Prior Economic Knowledge of First-Year Students in Business and Economic Studies in Germany. (Presented Paper; Annual Meeting of the American Economic Association). Boston: AEA. First citation in articleGoogle Scholar

  • Geisinger, K., Shaw, L. H. & McCormick, C. (2012). The validation of tests in higher education. In C. SecolskyD. B. Denison (Eds.), Handbook on measurement, assessment, and evaluation in higher education (pp. 194 – 207). New York, NY: Routledge. First citation in articleGoogle Scholar

  • German Council of Science and Humanities (2012). Arbeitsbericht Prüfungsnoten an Hochschulen im Prüfungsjahr 2010 (online verfügbar). First citation in articleGoogle Scholar

  • Happ, R., Förster, M., & Zlatkin-Troitschanskaia, O. (2016a). Assessing the previous economic knowledge of beginning students in Germany: Implications for teaching economics in basic courses. Citizenship, Social and Economics Education, 15 (1): 45 – 57. First citation in articleCrossref, Google Scholar

  • Happ, R., Zlatkin-Troitschanskaia, O., & Förster, M. (2018). How Prior Economic Education Influences Beginning University Students’ Knowledge of Economics. Empirical Research in Vocational Education and Training, 10 (5), 1 – 20. doi:10.1186/s40461-018-0066-7 First citation in articleCrossref, Google Scholar

  • Happ, R., Zlatkin-Troitschanskaia, O., & Schmidt, S. (2016b). An analysis of economic learning among undergraduates in introductory economics courses in Germany. The Journal of Economic Education, 47 (4), 300 – 310. doi:10.1080/00220485.2016.1213686 First citation in articleCrossref, Google Scholar

  • Hartig, J. & Frey, A. (2012). Konstruktvalidierung und Skalenbeschreibung in der Kompetenzdiagnostik durch die Vorhersage von Aufgabenschwierigkeiten. Psychologische Rundschau, 63 (1), 43 – 49. First citation in articleLink, Google Scholar

  • Hell, B., Trapmann, S. & Schuler, H. (2007). Eine Metaanalyse der Validität von fachspezifischen Studierfähigkeitstests im deutschsprachigen Raum. Empirische Pädagogik, 21, 251 – 270. First citation in articleGoogle Scholar

  • Hubley, A. & Zumbo, B. (2011). Validity and the consequences of test interpretation and use. Social Indicators Research, 103 (2), 219 – 230. First citation in articleCrossref, Google Scholar

  • Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50 (1), 1 – 73. First citation in articleCrossref, Google Scholar

  • Kuckartz, U. (2014). Mixed Methods: Methodologie, Forschungsdesigns und Analyseverfahren. Lehrbuch. Wiesbaden: Springer VS. First citation in articleGoogle Scholar

  • Kuhn, C., Zlatkin-Troitschanskaia, O, Pant, H. A., & Hannover, B. (2016). Valide Erfassung der Kompetenzen von Studierenden in der Hochschulbildung. Eine kritische Betrachtung des nationalen Forschungsstandes. Zeitschrift für Erziehungswissenschaft, 19(2), 275 – 298. https://doi.org//10.1007/s11618-016-0673-7 First citation in articleCrossref, Google Scholar

  • Kyllonen, P. C. & Christal, R. E. (1990). Reasoning ability is (little more than) working-memory capacity?! Intelligence, 14, 389 – 433. First citation in articleCrossref, Google Scholar

  • Larsen, M. R., Sommersel, H. B. & Larsen, M. S. (2013). Evidence on dropout phenomena at universities. Kopenhagen: Aarhus University. First citation in articleGoogle Scholar

  • Leighton, J. P. & Gierl, M. J. (2011). The learning sciences in educational assessment: The role of cognitive models. New York, NY: Cambridge University Press. First citation in articleCrossref, Google Scholar

  • Little, R. J. A. & Rubin, D. B. (2002). Statistical analysis with missing data. Wiley series in probability and statistics (2nd ed.). Hoboken, N.J.: Wiley. First citation in articleCrossref, Google Scholar

  • Lüdtke, O., Robitzsch, A., Trautwein, U., & Köller, O. (2007). Umgang mit fehlenden Werten in der psychologischen Forschung. Psychologische Rundschau, 58, 103 – 117. First citation in articleLink, Google Scholar

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13 – 103). New York: Macmillan Publishing. First citation in articleGoogle Scholar

  • Mirchandani, D., Lynch, R. & Hamilton, D. (2001). Using the ETS Major Field Test in Business: Implications for Assessment. Journal of Education for Business, 77 (1), 51 – 56. First citation in articleCrossref, Google Scholar

  • Mislevy, R. J. & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25 (4), 6 – 20. First citation in articleCrossref, Google Scholar

  • Organisation for Economic Cooperation and Development (OECD) (2013). AHELO Feasibility Study Volume 1: Design and implementation. Paris: OECD. First citation in articleGoogle Scholar

  • Pellegrino, J. W., Chudowsky, N. & Glaser, R. (2001). The nature of assessment and reasoning from evidence. In J. W. PellegrinoN. ChudowskyR. Glaser (Eds.), Knowing what students know: The science and design of educational assessment (pp. 37 – 54). Washington, DC: National Academy Press. First citation in articleGoogle Scholar

  • Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New Jersey: John Wiley & Sons. First citation in articleCrossref, Google Scholar

  • Schipolowski, S., Wilhelm, O., & Schroeders, U. (2016). Berliner Test zur Erfassung fluider und kristalliner Intelligenz ab der 11. Jahrgangsstufe (BEFKI 11+). Göttingen: Hogrefe. First citation in articleGoogle Scholar

  • Schmidt, S. (2018). Veränderungsmessung des fachlichen Wissens von Studierenden – Eine Längsschnittanalyse des Wissenserwerbs in einem latenten Mehrebenenmo dell (Economics Education and Human Resource Management). Springer Gabler: Wiesbaden. First citation in articleCrossref, Google Scholar

  • Schmidt, S., Zlatkin-Troitschanskaia, O. & Fox, J.-P. (2016). Pretest-Posttest-Posttest Multilevel IRT modeling of competence growth of students in higher education in Germany. Journal of Educational Measurement, 53 (3), 332 – 351. https://doi.org/10.1111/jedm.12115 First citation in articleCrossref, Google Scholar

  • Schuhmann, P. W., McGoldrick K. M. & Burrus, R. T. (2005). Student Quantitative Literacy: Importance, Measurement, and Correlation with Economic Literacy. The American Economist, 49 (1), 49 – 65. First citation in articleCrossref, Google Scholar

  • Settlage, D. M. & Settlage, L. A. (2011). A statistical framework for assessment using the ETS Major Field Test in business. Journal of Education for Business, 86 (5), 274 – 278. First citation in articleCrossref, Google Scholar

  • Shadish, W. R., Cook, T. D. & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston: Hoghton Mifflin. First citation in articleGoogle Scholar

  • Shavelson, R. J., Zlatkin-Troitschanskaia, O. & Marino, J. (2018). Performance indicators of learning in higher-education institutions: Overview of the field. In E. HazelkornH. CoatesA. Cormick (Eds.), Research Handbook on Quality, Performance and Accountability in Higher Education. Edward Elgar. First citation in articleGoogle Scholar

  • Verhoeven, B. H., Verwijnen, G. M., Scherpbier, A J J A, & Vleuten, C. P. M. Van Der. (2002). Growth of medical knowledge. Medical Education, 36, 711 – 717. https://doi.org/10.1046/j.1365-2923.2002.01268.x First citation in articleCrossref, Google Scholar

  • Walstad, W. B. & J. Wagner (2016). The disaggregation of value-added test scores to assess learning outcomes in economics courses. Journal of Economic Education, 47 (2), 121 – 131. First citation in articleCrossref, Google Scholar

  • Walstad, W. B. & Rebeck, K. (2008). The Test of Understanding of College Economics. American Economic Review, 98 (2), 547 – 551. First citation in articleCrossref, Google Scholar

  • Walstad, W. B., Rebeck, K. & Butters, R. B. (2013). The Test of Economic Literacy: Development and results. Journal of Economic Education, 44 (3), 298 – 309. First citation in articleCrossref, Google Scholar

  • Walstad, W. B., Schmidt, S., Zlatkin-Troitschanskaia, O. & Happ, R. (2018). Pretest-posttest measurement of the economic knowledge of undergraduates – Estimating guessing effects. Discussion Paper of the Annual AEA Conference on Teaching and Research in Economic Education, Philadelphia, AEA. First citation in articleGoogle Scholar

  • Walstad, W. B., Watts, M. & Rebeck, K. (2007). Test of Understanding in College Economics: Examiner's manual (4th ed.). New York: National Council on Economic Education. First citation in articleGoogle Scholar

  • Whitely, R. (1989). On the nature of managerial tasks and skills: Their distinguishing characteristics and organization. Journal of Management Studies, 26 (3), 209 – 224. First citation in articleCrossref, Google Scholar

  • Woods, C. M. (2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44 (1), 1 – 27. First citation in articleCrossref, Google Scholar

  • Yamaoka, M., Walstad, W. B., Watts, M. W., Asano, T. & Abe, S. (2010). Comparative studies on economic education in Asia-Pacific region. Tokyo: Shumpusha Publishing. First citation in articleGoogle Scholar

  • Zlatkin-Troitschanskaia, O., Happ, R., Förster, M., Preuße, D., Schmidt, S. & Kuhn, C. (2013). Analyse der Ausprägung und Entwicklung der Fachkompetenz von Studierenden der Wirtschaftswissenschaften und der Wirtschaftspädagogik. Lehrerbildung auf dem Prüfstand, 69 – 92. First citation in articleGoogle Scholar

  • Zlatkin-Troitschanskaia, O., Förster, M., Brückner, S., & Happ, R. (2014). Insights from a German assessment of business and economics competence. In H. Coates (Ed.), Higher Education Learning Outcomes Assessment: International Perspectives (pp. 175 – 197). Frankfurt am Main: Lang. First citation in articleGoogle Scholar

  • Zlatkin-Troitschanskaia, O., Förster, M., Schmidt, S., Brückner, S., & Beck, K. (2015). Erwerb wirtschaftswissenschaftlicher Fachkompetenz im Studium – Eine mehrebenenanalytische Betrachtung von hochschulischen und individuellen Einflussfaktoren. Zeitschrift Für Pädagogik, Beiheft, 61, 116 – 135. First citation in articleGoogle Scholar

  • Zlatkin-Troitschanskaia, O., & Kühling-Thees, C. (2019). Study-related Domain-specific and Generic Competencies of Economics Students: Insights from a German-Japanese Study. Journal of Asia Pacific Studies (Waseda University, Japan), 35, 285 – 297. First citation in articleGoogle Scholar

  • Zlatkin-Troitschanskaia, O., Pant, H. A., Kuhn, C., Toepper, M., & Lautenbach, C. (2016a). Messung akademischer Kompetenzen von Studierenden und Hochschulabsolventen. Ein Überblick zum nationalen und internationalen Forschungsstand. Wiesbaden: Springer. First citation in articleGoogle Scholar

  • Zlatkin-Troitschanskaia, O., Schmidt, S., Brückner, S., Förster, M., Yamaoka, M., & Asano, T. (2016b). Macroeconomic knowledge of higher education students in Germany and Japan – a multilevel analysis of contextual and personal effects. Assessment & Evalua-tion in Higher Education, 41 (5): 787 – 801. First citation in articleGoogle Scholar

  • Zlatkin-Troitschanskaia, O., Pant, H. A., Lautenbach, C., Molerov, D., Toepper, M. & Brückner, S. (2017). Modeling and measuring competencies in higher education – Approaches to challenges in higher education policy and practice. Wiesbaden: Springer. First citation in articleCrossref, Google Scholar