Abstract
Abstract. The goal of this study was to investigate sources of evidence of convergent validity supporting the construct interpretation of scores on a simulation-based information and communication technology (ICT) skills test. The construct definition understands ICT skills as reliant on ICT-specific knowledge as well as comprehension and problem-solving skills. On the basis of this, a validity argument comprising three claims was formulated and tested. (1) In line with the classical nomothetic span approach, all three predictor variables explained task success positively across all ICT skills items. As ICT tasks can vary in the extent to which they require construct-related knowledge and skills and in the way related items are designed and implemented, the effects of construct-related predictor variables were expected to vary across items. (2) A task-based analysis approach revealed that the item-level effects of the three predictor variables were in line with the targeted construct interpretation for most items. (3) Finally, item characteristics could significantly explain the random effect of problem-solving skills, but not comprehension skills. Taken together, the obtained results generally support the validity of the construct interpretation.
References
2014). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
. (2014). lme4: Linear mixed-effects models using Eigen and S4. R package version 1.1-7. Retrieved from http://CRAN.R-project.org/package=lme4
(2012).
(Defining 21st century skills . In P. GriffinB. McGawE. CareEds., Assessment and teaching of 21st century skills (pp. 17–66). Dordrecht, The Netherlands: Springer.2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology & Evolution, 24, 127–135. https://doi.org/10.1016/j.tree.2008.10.008
(2011). The word frequency effect: A review of recent developments and implications for the choice of frequency estimates in German. Experimental Psychology, 58, 412–424. https://doi.org/10.1027/1618-3169/a000123
(2009). Models and instruments for assessing digital competence at school. Journal of e-Learning and Knowledge Society-English Version, 4, 183–193. Retrieved from http://www.je-lks.org/ojs/index.php/Je-LKS_EN/article/view/288/270
(1970). Note on a simple algorithm for a computer-produced reading ease score. Journal of Applied Psychology, 54, 208–210. https://doi.org/10.1037/h0029067
(1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179–197. https://doi.org/10.1037/0033-2909.93.1.179
(2017). Experimental validation strategies for heterogeneous computer-based assessment items. Computers in Human Behavior, 76, 683–692. https://doi.org/10.1016/j.chb.2017.02.020
(1953). Reliability of the original and the simplified Flesch reading ease formulas. Journal of Applied Psychology, 37, 111–113. https://doi.org/10.1037/h0055346
(2004). Digital literacy: A conceptual framework for survival skills in the digital era. Journal of Educational Multimedia and Hypermedia, 13, 93–107. Retrieved from http://www.openu.ac.il/Personal_sites/download/Digital-literacy2004-JEMH.pdf
(2006). Recommendation of the European Parliament and the Council of 18 December 2006 on key competences for lifelong learning. Official Journal of the European Union, L394. Retrieved from http://www.alfa-trall.eu/wp-content/uploads/2012/01/EU2007-keyCompetencesL3-brochure.pdf
. (2012).
(Understanding digital competence in the 21st century: An analysis of current frameworks . In A. RavenscroftS. LindstaedtC. D. KloosD. Hernández-LeoEds., 21st Century Learning for 21st Century Skills (pp. 79–92). Berlin, Heidelberg, Germany: Springer. https://doi.org/10.1007/978-3-642-33263-0_71948). A new readability yardstick. Journal of Applied Psychology, 32, 221–233. https://doi.org/10.1037/h0057532
(2010). The IEA international study of computer and information literacy (ICILS). Retrieved from http://www.researchgate.net/profile/John_Ainley/publication/268297993_The_IEA_International_Study_of_Computer_and_Information_Literacy_%28ICILS%29/links/54eba4330cf2082851be49a9.pdf
(2013). Assessing individual differences in basic computer skills. European Journal of Psychological Assessment, 29, 263–275. https://doi.org/10.1027/1015-5759/a000153
(1980). Structural components of reading time. Journal of Verbal Learning and Verbal Behavior, 19, 135–151.
(2013). Computer-based assessment of complex problem-solving: Concept, implementation, and application. Educational Technology Research and Development, 61, 407–421.
(2002). Digital Transformation: A Framework for ICT Literacy. Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/ICTREPORT.pdf
. (1987).
(Speed reading . In M. A. JustP. A. CarpenterEds., The psychology of reading and language processing (pp. 425–452). Newton, MA: Allyn & Bacon.2010). Task effects on eye movements during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 1561–1566. https://doi.org/10.1037/a0020693
(2016). TAM: Test analysis modules. R package version 1.99-6. Retrieved from http://CRAN.R-project.org/package=TAM
(1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.
(2005).
(The effect of age and language structure on working memory performance . In B. G. B. L. BarsalouM. BucciarelliEds., Proceedings of the XXVII. Annual Meeting of the Cognitive Science Society (pp. 1413–1418). Mahwah, NJ: Erlbaum.2015). Third graders’ reading proficiency reading texts varying in complexity and length: Responses of students in an urban, high-needs school. Journal of Literacy Research, 47, 473–504. https://doi.org/10.1177/1086296x16631923
(2017). koRpus: An R package for text analysis. Retrieved from https://reaktanz.de/?c=hacking&s=koRpus
(2014). Erfolgreiches Problemlösen in technologiebasierten Umgebungen: Wechselwirkungen zwischen Interaktionsschritten und Aufgabenanforderungen
([Successful problem-solving in technology rich environments: Interactions between number of actions and task demands] . Zeitschrift für Pädagogische Psychologie, 28, 193–203. https://doi.org/10.1024/1010-0652/a0001342017). Digital reading proficiency in german 15-year olds: Evidence from PISA 2012. Zeitschrift für Erziehungswissenschaft, 20, 585–603. https://doi.org/10.1007/s11618-017-0758-y
(2012). Literacy, numeracy, and problem-solving in technology-rich environments: Framework for the OECD Survey of Adult Skills. Paris, France: OECD Publishing. https://doi.org/10.1787/9789264128859-en
. (2007). Reading ability: Lexical quality to comprehension. Scientific Studies of Reading, 11, 357–383.
(2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
. (2013). Lexical quality and reading comprehension in primary school children. Scientific Studies of Reading, 17, 415–434. https://doi.org/10.1080/10888438.2013.764879
(2010). Das Inventar zur Computerbildung (revidierte Fassung)
([A revised version of the Computer Literacy Inventory] . Zeitschrift für Pädagogische Psychologie, 24, 23–37. https://doi.org/10.1024/1010-0652/a0000022012).
(The ItemBuilder: A graphical authoring system for complex item development . In T. BastiaensG. MarksEds., Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2012 (pp. 344–353). Chesapeake, VA: AACE.2018). Construct validity of a process-oriented test assessing syntactic skills in German primary school children. Language Assessment Quarterly: An International Journal, 15, 183–203. https://doi.org/10.1080/15434303.2018.1446142
(2007). LGVT 6–12: Lesegeschwindigkeits- und -verständnistest für die Klassen 6–12
([Reading Speed and Comprehension Test for Grades 6 to 12] . Göttingen, Germany: Hogrefe.2005).
(An integrated model of text and picture comprehension . In R. E. MayerEd., The Cambridge handbook of multimedia learning (pp. 49–69). New York, NY: Cambridge University Press.2016). Taking a future perspective by learning from the past–A systematic review of assessment instruments that aim to measure primary and secondary school students’ ICT literacy. Educational Research Review, 19, 58–84.
(1971). Human problem-solving: The state of the theory in 1970. American Psychologist, 26, 145–159. https://doi.org/10.1037/h0030806
(2015). On the advantages of word frequency and contextual diversity measures extracted from subtitles: The case of Portuguese. The Quarterly Journal of Experimental Psychology, 68, 680–696. https://doi.org/10.1080/17470218.2014.964271
(1982). Further studies of the Tower of Hanoi problem-solving performance of retarded young adults and nonretarded children. Developmental Psychology, 18, 922–930. https://doi.org/10.1037/0012-1649.18.6.922
(2016). Easily too difficult: Estimating item difficulty in computer simulated microworlds. Computers in Human Behavior, 65, 100–106. https://doi.org/10.1016/j.chb.2016.08.025
(2015). How readability and topic incidence relate to performance on mathematics story problems in computer-based curricula. Journal of Educational Psychology, 107, 1051–1074. https://doi.org/10.1037/edu0000036
(2016).
(Computergestützte, adaptive und verhaltensnahe Erfassung Informations- und Kommunikationstechnologie-bezogener Fertigkeiten (ICT-Skills) (CavE-ICT) [Computer-based, adaptive and behavior related assessment of information and communication-related competencies (ICT skills)] . In In BMBFEds., Forschung in Ankopplung an Large-Scale Assessments (pp. 161–180). Bonn, Berlin: BMBF.2015). Eye movements during reading and topic scanning: Effects of word frequency. Journal of Experimental Psychology: Human Perception and Performance (pp. 233–248). 41. https://doi.org/10.1037/xhp0000020
(2008).
(Explanatory item response models: A brief introduction . In J. HartigE. KliemeD. LeutnerEds., Assessment of competencies in educational contexts (pp. 91–120). Göttingen, Germany: Hogrefe.