Abstract
Abstract. Skilful collaborative problem-solving is becoming increasingly important in various life areas. However, researchers are still seeking ways to assess and foster this skill in individuals. In this study, we developed a computer-assisted assessment for collaborative behavior (COLBAS) following the experiment-based assessment of behavior approach (objective personality tests; Cattell, 1958). The instrument captures participants’ collaborative behavior in problem-solving tasks using the MicroDYN approach while participants work collaboratively with a computer-agent. COLBAS can thereby assess problem-solving and collaborative behavior expressed through communication acts. To investigate its validity, we administered COLBAS to 483 German seventh graders along with MicroDYN as a measure of individual problem-solving skills and questions regarding the motivation to collaborate. A latent confirmatory factor analysis suggested a five-dimensional construct with two problem-solving dimensions (knowledge acquisition and knowledge application) and three collaboration dimensions (questioning, asserting, and requesting). The results showed that extending MicroDYN to include collaborative aspects did not considerably change the measurement of problem-solving. Finally, students who were more motivated to collaborate interacted more with the computer-agent but also obtained worse problem-solving results.
References
(1992). Principles for measuring teamwork skills. Human Factors, 34, 469–475.
(2000). Achieving coordination in collaborative problem-solving groups. The Journal of the Learning Sciences, 9, 403–436.
(2003). When smart groups fail. The Journal of the Learning Sciences, 12, 307–359.
(2012).
Defining 21st Century Skills . In P. GriffinB. McGawE. CareEds., Assessment and teaching 21st century skills (pp. 17–66). Heidelberg, Germany: Springer.(1958). A need for alertness to multivariate experimental findings in integrative surveys. Psychological Bulletin, 55, 253–256.
(1999).
What do you mean by “collaborative learning?” . In P. DillenbourgEd., Collaborative-learning: Cognitive and computational approaches (pp. 1–19). Oxford, UK: Elsevier.(2006). Sharing solutions: Persistence and grounding in multimodal collaborative problem solving. Journal of the Learning Sciences, 15, 121–151.
(2001). Dynamic systems as tools for analysing human judgement. Thinking & Reasoning, 7, 69–89.
(2012). Dynamic problem solving: A new measurement perspective. Applied Psychological Measurement, 36, 189–213.
(2013). Complex problem solving in educational contexts – something beyond g: concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology, 105, 364–379.
P. GriffinB. McGawE. Care (2012). Assessment and teaching of 21st century skills. New York, NY: Springer.
(2005). Intelligence assessment with computer simulations. Intelligence, 33, 347–368.
(2009). The technique of objective personality-tests sensu R. B. Cattell. nowadays: The Viennese pool of computerized tests aimed at experiment-based assessment of behavior. Acta Psychologica Sinica, 41, 1024–1036.
(2002). Statistical analysis with missing data (2nd ed.). New York, NY: Wiley.
(2006).
Problem solving . In P. A. AlexanderP. H. WinneEds., Handbook of educational psychology (2nd ed., pp. 287–304). Mahwah, NJ: Erlbaum.(1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.
(2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning, 2, 63–86.
(2012). Impact of two types of partner, perceived or actual, in human-human and human-agent interaction. Computers in Human Behaviour, 28, 1286–1297.
(1998–2014). Mplus User’s Guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
(2003). Issues in the computer-based assessment of collaborative problem solving. Assessment in Education, 10, 361–373.
(2013). PISA 2015 draft collaborative problem solving framework. Paris, France: OECD.
T. M. OrtnerR. T. ProyerK. D. Kubinger (Eds.). (2006). Theorie und Praxis objektiver Persönlichkeitstests
[Theory and practice of objective personality tests] . Bern, Switzerland: Hans Huber.(1995).
The construction of shared knowledge in collaborative problem-solving . In C. E. O’MalleyEd., Computer-supported collaborative learning (pp. 69–97). Berlin, Germany: Springer.(2012). Scripted collaborative learning with the cognitive tutor algebra. Computer-Supported Collaborative Learning, 7, 307–339.
(1975). Objektive Persönlichkeitsmessung in diagnostischer und klinischer Psychologie
[Objective personality assessment in psychodiagnostics and clinical psychology] . Weinheim, Germany: Beltz.(1975).
Indirect speech acts . In P. ColeJ. L. MorganEds., Syntax and Semantics, 3: Speech Acts (pp. 59–82). New York, NY: Academic Press.(2011). MICE: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45, 1–67.
(1999). Meta-analyses of fakability estimates: Implications for personality measurement. Educational and Psychological Measurement, 59, 197–210.
(1995). Group collaboration in assessment: Multiple objectives, processes, and outcomes. Educational Evaluation and Policy Analysis, 17, 239–261.
(2011). A closer look on social presence as a causing factor in computer-mediated collaboration. Computers in Human Behaviour, 27, 513–521.
(2012). Complex problem solving. More than reasoning? Intelligence, 40, 1–14.
(1998). A distributed representation approach to group problem solving. Journal of the American Society for Information Science, 49, 801–809.
