Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigation
Previous article
Next article
Originalarbeit

Entwicklung und Skalierung eines Tests zur Erfassung des Verständnisses multipler Dokumente von Studierenden

Published Online:April 20, 2020https://doi.org/10.1026/0012-1924/a000231

Abstract

Zusammenfassung. Das Verständnis multipler Dokumente (Multiple Document Comprehension, MDC) wird als Fähigkeit verstanden, aus verschiedenen Informationsquellen eine integrierte Repräsentation eines inhaltlichen Gegenstandsbereichs zu konstruieren. Als solche ist sie sowohl für die erfolgreiche Bewältigung eines Studiums als auch für gesellschaftliche Partizipation eine wichtige Kompetenz. Bislang gibt es jedoch kein etabliertes Diagnostikum in diesem Bereich. Um diese Lücke zu schließen, wurde ein Test entwickelt, der vier zentrale kognitive Anforderungen von MDC abdeckt und auf Basis der Daten von 310 Studierenden sozial- und geisteswissenschaftlicher Fächer überprüft wurde. Die im MDC-Test gemessene Kompetenz erwies sich als eindimensional. Der MDC-Testwert wies theoriekonforme Zusammenhänge mit der Abiturnote, dem Studienabschnitt und der Leistung in einer Essay-Aufgabe auf. Insgesamt liefern die Ergebnisse empirische Belege dafür, dass der Testwert aus dem MDC-Test die fächerübergreifende Fähigkeit von Studierenden wiedergibt, multiple Dokumente zu verstehen.


Developing and Scaling a Test of Multiple Document Comprehension in University Students

Abstract. Multiple document comprehension (MDC) is defined as the ability to construct an integrated representation based on different sources of information on a particular topic. It is an important competence for both the successful accomplishment of university studies and participation in societal discussions. Yet, there is no established assessment instrument for MDC. Therefore, we developed a test covering four theory-based cognitive requirements of MDC. Based on the data of 310 university students of social sciences and humanities, the MDC test proved to be a unidimensional measure. Furthermore, the test score was related to the final school exam grade, the study level (bachelor / master), and the performance in an essay task. The empirical results suggest that the score of the MDC test can be interpreted as the generic competence of university students to understand multiple documents.

Literatur

  • American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. First citation in articleGoogle Scholar

  • Anmarkrud, Ø., Bråten, I. & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64 – 76. https://doi.org/10.1016/j.lindif.2013.01.007 First citation in articleCrossref, Google Scholar

  • Arbeitskreis Deutscher Qualifikationsrahmen (2011). Deutscher Qualifikationsrahmen für lebenslanges Lernen. Verfügbar unter: http://www.dqr.de/media/content/Der_Deutsche_Qualifikationsrahmen_fue_lebenslanges_Lernen.pdf First citation in articleGoogle Scholar

  • Bates, D., Mächler, M., Bolker, B. & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67 (1), 1 – 48. https://doi.org/10.18637/jss.v067.i01 First citation in articleCrossref, Google Scholar

  • Braasch, J. L. G. & Bråten, I. (2017). The discrepancy-induced source comprehension (D-ISC) model: Basic assumptions and preliminary evidence. Educational Psychologist, 52, 167 – 181. https://doi.org/10.1080/00461520.2017.1323219 First citation in articleCrossref, Google Scholar

  • Braasch, J. L. G., Bråten, I., Strømsø, H. I. & Anmarkrud, Ø. (2014). Incremental theories of intelligence predict multiple document comprehension. Learning and Individual Differences, 31, 11 – 20. https://doi.org/10.1016/j.lindif.2013.12.012 First citation in articleCrossref, Google Scholar

  • Braasch, J. L. G., Rouet, J.-F., Vibert, N. & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40, 450 – 465. https://doi.org/10.3758/s13421-011-0160-6 First citation in articleCrossref, Google Scholar

  • Bråten, I., Ferguson, L. E., Strømsø, H. I. & Anmarkrud, Ø. (2014). Students working with multiple conflicting documents on a scientific issue: Relations between epistemic cognition while reading and sourcing and argumentation in essays. British Journal of Educational Psychology, 84 (1), 58 – 85. https://doi.org/10.1111/bjep.12005 First citation in articleCrossref, Google Scholar

  • Bråten, I., Salmerón, L. & Strømsø, H. I. (2016). Who said that? Investigating the Plausibility-Induced Source Focusing assumption with Norwegian undergraduate readers. Contemporary Educational Psychology, 46, 253 – 262. https://doi.org/10.1016/j.cedpsych.2016.07.004 First citation in articleCrossref, Google Scholar

  • Britt, M. A. & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485 – 522. https://doi.org/10.1207/s1532690xci2004_2 First citation in articleCrossref, Google Scholar

  • Britt, M. A., Perfetti, C. A., Sandak, R. & Rouet, J.-F. (1999). Content integration and source separation in learning from multiple texts. In S. R. GoldmaA. C. GraesserP. Van den Broek (Eds.), Narrative, comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 209 – 233). Mahwah, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Britt, M. A. & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In J. R. KirbyM. J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276 – 314). New York: Cambridge University Press. First citation in articleGoogle Scholar

  • Britt, M. A. & Sommer, J. (2004). Facilitating textual integration with macro-structure focusing tasks. Reading Psychology, 25, 313 – 339. https://doi.org/10.1080/02702710490522658 First citation in articleCrossref, Google Scholar

  • Cerdán, R. & Vidal-Abarca, E. (2008). The effects of tasks on integrating information from multiple documents. Journal of Educational Psychology, 100, 209 – 222. https://doi.org/10.1037/0022-0663.100.1.209 First citation in articleCrossref, Google Scholar

  • Chen, W.-H. & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265 – 289. https://doi.org/10.2307/1165285 First citation in articleCrossref, Google Scholar

  • Deutscher Bibliotheksverband (2009). Standards der Informationskompetenz für Studierende. Verfügbar unter www.bibliotheksverband.de/fileadmin/user_upload/Kommissionen/Kom_Dienstleistung/Publikationen/Standards_Infokompetenz_03.07.2009_endg.pdf First citation in articleGoogle Scholar

  • Douglas, J. & Cohen, A. (2001). Nonparametric item response function estimation for assessing parametric model fit. Applied Psychological Measurement, 25, 234 – 243. https://doi.org/10.1177/01466210122032046 First citation in articleCrossref, Google Scholar

  • Gehrer, K., Zimmermann, S., Artelt, C. & Weinert, S. (2013). NEPS framework for assessing reading competence and results from an adult pilot study. Journal for Educational Research Online, 5 (2), 50 – 79. First citation in articleGoogle Scholar

  • Gil, L., Bråten, I., Vidal-Abarca, E. & Strømsø, H. I. (2010a). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157 – 173. https://doi.org/10.1016/j.cedpsych.2009.11.002 First citation in articleCrossref, Google Scholar

  • Gil, L., Bråten, I., Vidal-Abarca, E. & Strømsø, H. I. (2010b). Understanding and integrating multiple science texts: Summary tasks are sometimes better than argument tasks. Reading Psychology, 31 (1), 30 – 68. https://doi.org/10.1080/02702710902733600 First citation in articleCrossref, Google Scholar

  • Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C. et al. (2016). Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51, 219 – 246. https://doi.org/10.1080/00461520.2016.1168741 First citation in articleCrossref, Google Scholar

  • Goldman, S. R. & Scardamalia, M. (2013a). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31, 255 – 269. https://doi.org/10.1080/10824669.2013.773217 First citation in articleCrossref, Google Scholar

  • Goldman, S. R. & Scardamalia, M. (2013b). Mutiple document comprehension [Special Issue]. Cognition and Instruction, 31 (2) First citation in articleCrossref, Google Scholar

  • Homann, B. (2000). Das Dynamische Modell der Informationskompetenz (DYMIK) als Grundlage für bibliothekarische Schulungen. In G. KnorzR. Kuhlen (Hrsg.), Informationskompetenz – Basiskompetenz in der Informationsgesellschaft. Proceedings des 7. Internationale Symposiums für Informationswissenschaft (ISI 2000), Darmstadt, 8. – 10. November 2000 (S. 195 – 206). Konstanz: UVK Verlag. First citation in articleGoogle Scholar

  • Kaakinen, J. K. & Hyönä, J. (2008). Perspective-driven text comprehension. Applied Cognitive Psychology, 22, 319 – 334. First citation in articleCrossref, Google Scholar

  • Kammerer, Y. & Gerjets, P. (2014). Quellenbewertungen und Quellenverweise bei Lesen und Zusammenfassen wissensbezogener Informationen aus multiplen Webseiten. Unterrichtswissenschaft, 42 (1), 7 – 23. First citation in articleGoogle Scholar

  • Kammerer, Y., Kalbfell, E. & Gerjets, P. (2016). Is this information source commercially biased? How contradictions between web pages stimulate the consideration of source information. Discourse Processes, 53, 430 – 456. https://doi.org/10.1080/0163853x.2016.1169968 First citation in articleCrossref, Google Scholar

  • Keck, D., Kammerer, Y. & Starauschek, E. (2015). Reading science texts online: Does source information influence the identification of contradictions within texts? Computers & Education, 82, 442 – 449. https://doi.org/10.1016/j.compedu.2014.12.005 First citation in articleCrossref, Google Scholar

  • Kintsch, W. (1998). Comprehension. A paradigm for cognition. Cambridge: Cambridge University Press. First citation in articleGoogle Scholar

  • Kultusministerkonferenz (2012). Bildungsstandards im Fach Deutsch für die Allgemeine Hochschulreife (Beschluss der Kultusministerkonferenz vom 18. 10. 2012). Verfügbar unter: www.kmk.org/fileadmin/veroeffentlichungen_beschluesse/2012/2012_10_18-Bildungsstandards-Deutsch-Abi.pdf First citation in articleGoogle Scholar

  • Landis, J. R. & Koch, G. (1977). The measurement of observer agreement for categorial data. Biometrics, 33, 159 – 174. First citation in articleCrossref, Google Scholar

  • Lau, J. (2006). Guidelines on information literacy for lifelong learning. Verfügbar unter archive.ifla.org/VII/s42/pub/IL-Guidelines2006.pdf First citation in articleGoogle Scholar

  • Magis, D., Béland, S., Tuerlinckx, F. & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42, 847 – 862. https://doi.org/10.3758/brm.42.3.847 First citation in articleCrossref, Google Scholar

  • Maier, J. & Richter, T. (2013). Text belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31, 151 – 175. https://doi.org/10.1080/07370008.2013.769997 First citation in articleCrossref, Google Scholar

  • Maier, J. & Richter, T. (2014). Verstehen multipler Texte zu kontroversen wissenschaftlichen Themen: Die Rolle der epistemischen Validierung. Unterrichtswissenschaft, 42 (1), 24 – 38. First citation in articleGoogle Scholar

  • McCrudden, M. T. & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19 (2), 113 – 139. First citation in articleCrossref, Google Scholar

  • Mühlen, S. v. d., Richter, T., Schmid, S., Schmidt, E. M. & Berthold, K. (2016). The use of source-related strategies in evaluating multiple psychology texts: A student–scientist comparison. Reading and Writing, 29, 1677 – 1698. https://doi.org/10.1007/s11145-015-9601-0 First citation in articleCrossref, Google Scholar

  • Muthén, B., Muthén, L. & Asparouhov, T. (2015). Estimator choices with categorical outcomes. Verfügbar unter https://www.statmodel.com/download/EstimatorChoices.pdf First citation in articleGoogle Scholar

  • Paul, J., Macedo-Rouet, M., Rouet, J.-F. & Stadtler, M. (2017). Why attend to source information when reading online? The perspective of ninth grade students from two different countries. Computers & Education, 113, 339 – 354. https://doi.org/10.1016/j.compedu.2017.05.020 First citation in articleCrossref, Google Scholar

  • Perfetti, C. A., Rouet, J.-F. & Britt, M. A. (1999). Toward a theory of documents representation. In H. van OostendorpS. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99 – 122). Mahwah, NJ: Lawrence Erlbaum Associates. First citation in articleGoogle Scholar

  • Pohl, S., Gräfe, L. & Rose, N. (2014). Dealing with omitted and not-reached items in competence tests. Educational and Psychological Measurement, 74, 423 – 452. https://doi.org/10.1177/0013164413504926 First citation in articleCrossref, Google Scholar

  • R Core Team (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Verfügbar unter https://www.R-project.org/ First citation in articleGoogle Scholar

  • Richter, T. & Maier, J. (2017). Comprehension of multiple documents with conflicting information: A two-step model of validation. Educational Psychologist, 52, 148 – 166. https://doi.org/10.1080/00461520.2017.1322968 First citation in articleCrossref, Google Scholar

  • Robitzsch, A., Kiefer, T. & Wu, M. (2017). TAM: Test analysis modules. R package version 2.3 – 18. Verfügbar unter https://cran.r-project.org/package=TAM First citation in articleGoogle Scholar

  • Rölke, H. (2012). The ItemBuilder: A graphical authoring system for complex item development. In T. BastiaensG. Marks (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2012 (Vol. 2012, pp. 344 – 353). Chesapeake, VA: AACE. First citation in articleGoogle Scholar

  • Rose, N., Von Davier, M. & Xu, X. (2010). Modeling nonignorable missing data with item response theory (IRT). Princeton: ETS. First citation in articleCrossref, Google Scholar

  • Rouet, J.-F., Favart, M., Britt, M. A. & Perfetti, C. A. (1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15 (1), 85 – 106. https://doi.org/10.2307/3233756 First citation in articleCrossref, Google Scholar

  • Sailer, M. O. (2013). crossdes: Construction of Crossover Designs. R package version 1.1 – 1. Verfügbar unter: https://CRAN.R-project.org/package=crossdes First citation in articleGoogle Scholar

  • Salmerón, L., Gil, L., Bråten, I. & Strømsø, H. (2010). Comprehension effects of signalling relationships between documents in search engines. Computers in Human Behavior, 26, 419 – 426. https://doi.org/10.1016/j.chb.2009.11.013 First citation in articleCrossref, Google Scholar

  • Scharrer, L. & Salmerón, L. (2016). Sourcing in the reading process [Special issue]. Reading and Writing, 29 (8) First citation in articleCrossref, Google Scholar

  • Schmalhofer, F. & Glavanov, D. (1986). Three components of understanding a programmer’s manual: Verbatim, propositional, and situational representations. Journal of Memory and Language, 25, 279 – 294. https://doi.org/10.1016/0749-596X(86)90002-1 First citation in articleCrossref, Google Scholar

  • Schwabe, F., McElvany, N. & Trendtel, M. (2015). The school age gender gap in reading achievement: Examining the influences of item format and intrinsic reading motivation. Reading Research Quarterly, 50, 219 – 232. https://doi.org/10.1002/rrq.92 First citation in articleCrossref, Google Scholar

  • Stadtler, M. & Bromme, R. (2014). The content–source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. RappJ. L. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379 – 402). Cambridge, MA: MIT Press. First citation in articleGoogle Scholar

  • Stadtler, M., Scharrer, L., Skodzik, T. & Bromme, R. (2014). Comprehending multiple documents on scientific controversies: Effects of reading goals and signaling rhetorical relationships. Discourse Processes, 51 (1 – 2), 93 – 116. https://doi.org/10.1080/0163853x.2013.855535 First citation in articleCrossref, Google Scholar

  • Strømsø, H. I., Bråten, I., Britt, M. A. & Ferguson, L. E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 31, 176 – 203. https://doi.org/10.1080/07370008.2013.769994 First citation in articleCrossref, Google Scholar

  • Wang, W.-C. & Wilson, M. (2005). The Rasch testlet model. Applied Psychological Measurement, 29 (2), 126 – 149. https://doi.org/10.1177/0146621604271053 First citation in articleCrossref, Google Scholar

  • Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427 – 450. https://doi.org/10.1007/bf02294627 First citation in articleCrossref, Google Scholar

  • Weinert, F. E. (1999). Definition and selection of competencies – concepts of competence. München: Max Planck Institute for Psychological Research. First citation in articleGoogle Scholar

  • Weinert, F. E. (2001). Concept of competence. A conceptual clarification. In D. S. RychenL. H. Salganik (Hrsg.), Defining and selecting key competencies (pp. 45 – 65). Göttingen: Hogrefe & Huber. First citation in articleGoogle Scholar

  • Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83 (1), 73 – 87. https://doi.org/10.1037/0022-0663.83.1.73 First citation in articleCrossref, Google Scholar

  • Wright, B. & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370. First citation in articleGoogle Scholar