Abstract
Electronic devices can be used to enhance or improve cognitive ability testing. We compared three reasoning-ability measures delivered on handheld computers, notebooks, and paper-and-pencil to test whether or not the same underlying abilities were measured irrespective of the test medium. Rational item-generative principles were used to generate parallel item samples for a verbal, a numerical, and a figural reasoning test, respectively. All participants, 157 high school students, completed the three measures on each test medium. Competing measurement models were tested with confirmatory factor analyses. Results show that 2 test-medium factors for tests administrated via notebooks and handheld computers, respectively, had small to negligible loadings, and that the correlation between these factors was not substantial. Overall, test medium was not a critical source of individual differences. Perceptual and motor skills are discussed as potential causes for test-medium factors.
References
1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least square estimation in confirmatory factor analysis. Structural Equation Modeling, 13, 186–203.
(2002). True scores, latent variables, and constructs: A comment on Schmidt and Hunter. Intelligence, 30, 505–514.
(2003). Effects of screen size, screen resolution, and display rate on computer-based test performance. Applied Measurement in Education, 16, 191–205.
(1990). What one intelligence test measures: A theoretical account of processing in the Raven’s progressive matrices test. Psychological Review, 97, 404–431.
(1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge, MA: Cambridge University Press.
(2005). New tests and new items: Opportunities and issues. In , Computer-based testing and the Internet: Issues and advances. New York: Wiley.
(2000). A multitrait–multimethod model with minimal assumptions. Psychometrika, 65, 241–261.
(1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 380–396.
(2009). Transitioning to computer-based assessments: A question of costs. In , The transition to computer-based assessment (pp. 108–116). Office for Official Publications of the European Communities, Luxembourg. Retrieved from crell.jrc.it/RP/reporttransition.pdf
(2005). International guidelines on computer-based and Internet-delivered testing. Retrieved from www.intestcom.org/guidelines
. (2002). The foundations of item generation for mass testing. In , Item generation for test development (pp. 3–34). Mahwah, NJ: Erlbaum.
(1997). Rules and illusions: A critical study of Rip’s The Psychology of Proof. Minds and Machines, 7, 387–407.
(1999, October). Meta-analysis of equivalence of computerized and P&P tests on ability measures. Paper presented at the annual meeting of the Midwestern Educational Research Association, Chicago, IL.
(1990). Reasoning ability is (little more than) working-memory capacity? Intelligence, 14, 389–433.
(2006). The mode effect: A literature review of human and technological issues in computerized testing. International Journal of Testing, 6, 1–24.
(2006). Distraction as a determinant of processing speed. Psychonomic Bulletin and Review, 13, 619–625.
(1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.
(1993). Equivalence of computerized and paper-and-pencil cognitive-ability tests: A meta-analysis. Psychological Bulletin, 114, 449–458.
(1998). Computerization of paper-and-pencil tests: When are they equivalent? Applied Psychological Measurement, 22, 71–83.
(1996). The pen-based computer as an alternative platform for test administration. Personnel Psychology, 49, 455–464.
(2001). Complexity of geometric inductive reasoning tasks: Contribution to the understanding of fluid intelligence. Intelligence, 30, 41–70.
(2009). Testing for equivalence of test data across media. In , The transition to computer-based assessment (pp. 164–170). Office for Official Publications of the European Communities, Luxembourg. Retrieved from crell.jrc.it/RP/reporttransition.pdf
(2010). Internet-based ability testing: Problems and opportunities. In , Advanced methods for behavioral research on the Internet (pp. 131–148). Washington, DC: American Psychological Association.
(2004). A usability comparison of PDA-based quizzes and paper-and-pencil quizzes. Computers and Education, 45, 417–432.
(1904). “General intelligence,” objectively determined and measured. American Journal of Psychology, 15, 201–293.
(1999). Examining the relationship between computer familiarity and performance on computer-based language tasks. Language Learning, 49, 219–274.
(1998). Computer anxiety: A comparison of pen-based personal digital assistants, conventional computer, and paper assessment of mood and performance. British Journal of Psychology, 89, 599–610.
(2008). Important considerations in e-assessment. In , Toward a research agenda on computer-based assessment (pp. 97–103). Office for Official Publications of the European Communities, Luxembourg. Retrieved from crell.jrc.it/CBA/EU-Report-CBA.pdf
(2007). A meta-analysis of testing mode effects in grade K-12 mathematics tests. Educational and Psychological Measurement, 67, 219–238.
(2008). Comparability of computer-based and paper-and-pencil testing in K-12 reading assessments: A meta-analysis of testing mode effects. Educational and Psychological Measurement, 68, 5–24.
(2005). Measuring reasoning ability. In , Understanding and measuring intelligence (pp. 373–392). London: Sage.
(