Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Multistudy Report

Social-Scientific Research Competency

Validation of Test Score Interpretations for Evaluative Purposes in Higher Education

Published Online:February 16, 2018https://doi.org/10.1027/1015-5759/a000451

Abstract

Abstract. Although the development of research competency is an important goal of higher education in social sciences, instruments to measure this outcome often depend on the students’ self-ratings. To provide empirical evidence for the utility of a newly developed instrument for the objective measurement of social-scientific research competency, two validation studies across two independent samples were conducted. Study 1 (n = 675) provided evidence for unidimensionality, expected differences in test scores between differently advanced groups of students as well as incremental validities over and above self-perceived research self-efficacy. In Study 2 (n = 82) it was demonstrated that the competency measured indeed is social-scientific and relations to facets of fluid and crystallized intelligence were analyzed. Overall, the results indicate that the test scores reflected a trainable, social-scientific, knowledge-related construct relevant to research performance. These are promising results for the application of the instrument in the evaluation of research education courses in higher education.

References

  • Ackerman, P. L. (2000). Domain-specific knowledge as the “dark matter” of adult intelligence: Gf/Gc, personality and interest correlates. The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 55, P69–P84. https://doi.org/10.1093/geronb/55.2.P69 First citation in articleCrossref, Google Scholar

  • Ackerman, P. L. & Heggestad, E. D. (1997). Intelligence, personality, and interests: evidence for overlapping traits. Psychological Bulletin, 121, 219–245. https://doi.org/10.1037/0033-2909.121.2.219 First citation in articleCrossref, Google Scholar

  • AERA, APA, NMCE . (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. First citation in articleGoogle Scholar

  • American Political Science Association. (2004). APSA task force on graduate education. Washington, DC: American Political Science Association. Retrieved from http://files.eric.ed.gov/fulltext/ED495969.pdf First citation in articleGoogle Scholar

  • Arendasy, M., Hornke, L. F., Sommer, M., Häusler, J., Wagner-Menghin, M., Gittler, G., … Körtner, T. (2015). Manual Intelligence Structure Battery. Mödling, Austria: Schuhfried. First citation in articleGoogle Scholar

  • Beauducel, A. & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186–203. https://doi.org/10.1207/s15328007sem1302_2 First citation in articleCrossref, Google Scholar

  • Bieschke, K. J. (2006). Research self-efficacy beliefs and research outcome expectations: Implications for developing scientifically minded psychologists. Journal of Career Assessment, 14, 77–91. https://doi.org/10.1177/1069072705281366 First citation in articleCrossref, Google Scholar

  • Blömeke, S., Gustafsson, J.-E. E. & Shavelson, R. J. (2015). Beyond dichotomies: Competence viewed as a continuum. Zeitschrift für Psychologie/Journal of Psychology, 223, 3–13. https://doi.org/10.1027/2151-2604/a000194 First citation in articleLink, Google Scholar

  • Boswell, S. S. (2013). Undergraduates’ perceived knowledge, self-efficacy, and interest in social science research. The Journal of Effective Teaching, 13, 48–57. First citation in articleGoogle Scholar

  • Böttcher, F. & Thiel, F. (2017). Evaluating research-oriented teaching: A new instrument to assess university students’ research competences. Higher Education, 1–20. https://doi.org/10.1007/s10734-017-0128-y First citation in articleGoogle Scholar

  • British Academy. (2012). Society counts: Quantitative skills in the social sciences. London, UK: British Academy. https://www.britac.ac.uk/sites/default/files/BA%20Position%20Statement%20-%20Society%20Counts.pdf First citation in articleGoogle Scholar

  • Bryant, F. B. & Satorra, A. (2012). Principles and practice of scaled difference chi-square testing. Structural Equation Modeling: A Multidisciplinary Journal, 19, 372–398. https://doi.org/10.1080/10705511.2012.687671 First citation in articleCrossref, Google Scholar

  • Bryman, A. (2016). Social research methods (5th ed.). Oxford, UK: Oxford University Press. First citation in articleGoogle Scholar

  • Cassidy, S. & Eachus, P. (2000). Learning style, academic belief systems, self-report student proficiency and academic achievement in higher education. Educational Psychology, 20, 307–322. https://doi.org/10.1080/713663740 First citation in articleCrossref, Google Scholar

  • Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14, 464–504. https://doi.org/10.1080/10705510701301834 First citation in articleCrossref, Google Scholar

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. First citation in articleCrossref, Google Scholar

  • Csapó, B. (2004). Knowledge and competencies. In J. LetschertJ. BronH. HooghoffEds., The integrated person. How curriculum development relates to new competencies (pp. 35–50). Enschede, The Netherlands: CIDREE. First citation in articleGoogle Scholar

  • Dietrich, H., Zhang, Y., Klopp, E., Brünken, R., Krause, U. M., Spinath, F. M., … Spinath, B. (2015). Scientific competencies in the social sciences. Psychology Learning and Teaching, 14, 115–130. https://doi.org/10.1177/1475725715592287 First citation in articleCrossref, Google Scholar

  • Eid, M., Geiser, C., Koch, T. & Heene, M. (2016). Anomalous results in G-factor models: explanations and alternatives. Psychological Methods, 22, 541–562. https://doi.org/10.1037/met0000083 First citation in articleCrossref, Google Scholar

  • Evers, A., Hagemeister, C., Høstmaelingen, A., Lindley, P., Muñiz, J. & Sjöberg, A. (2013). EFPA Review Model for the description and evaluation of psychological and educational tests. Brussels, Belgium: European Federation of Psychology Associations. First citation in articleGoogle Scholar

  • Forester, M., Kahn, J. H. & Hesson-McInnis, M. S. (2004). Factor structures of three measures of research self-efficacy. Journal of Career Assessment, 12, 3–16. https://doi.org/10.1177/1069072703257719 First citation in articleCrossref, Google Scholar

  • Fox, J. & Weisberg, S. (2011). An {R} companion to applied regression (2nd ed.). Thousand Oaks, CA: Sage. First citation in articleGoogle Scholar

  • Gess, C., Wessels, I. & Blömeke, S. (2017). Domain-Specificity of Research Competencies in the Social Sciences: Evidence from Differential Item Functioning. Journal for Educational Research Online, 9, 11–36. First citation in articleGoogle Scholar

  • Gess, C., Rueß, J. & Blömeke, S. (2017). Was ist Forschungskompetenz? Entwicklung und Expertenbegutachtung eines fach- und paradigmenübergreifenden Kompetenzmodells für die Sozialwissenschaften [What is research competency? Development and expert review of a competency model spanning across social scientific disciplines and research paradigms]. First citation in articleGoogle Scholar

  • Groß Ophoff, J., Schladitz, S., Leuders, J., Leuders, T. & Wirtz, M. A. (2015). Assessing the development of educational research literacy: The effect of courses on research methods in studies of educational science. Peabody Journal of Education, 90, 560–573. https://doi.org/10.1080/0161956X.2015.1068085 First citation in articleCrossref, Google Scholar

  • Guba, E. G. & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. In N. K. DenzinY. S. LincolnEds., Handbook of qualitative research (pp. 163–194). Thousand Oaks, CA: Sage. First citation in articleGoogle Scholar

  • Hancock, G. R. (2001). Effect size, power, and sample size determination for structured means modeling and MIMIC approaches to between-groups hypothesis testing of means on a single latent construct. Psychometrika, 66, 373–388. https://doi.org/10.1007/BF02294440 First citation in articleCrossref, Google Scholar

  • Hancock, G. R. & Mueller, R. O. (2015). Rethink construct reliability within latent variable systems. In R. CudeckS. du ToitD. SörbomEds., Structural equation modeling: Present and future. A Festschrift in honor of Karl Jöreskog (pp. 195–216). Lincolnwood, IL: Scientific Software International. First citation in articleGoogle Scholar

  • Harrell, F. E. J. (2016). Hmisc: Harrell miscellaneous. Retrieved from https://cran.r-project.org/package=Hmisc First citation in articleGoogle Scholar

  • Hartig, J. & Klieme, E. (2006). Kompetenz und Kompetenzdiagnostik [Competency and competency diagnostics]. In K. SchweizerEd., Leistung und Leistungsdiagnostik (pp. 127–143). Heidelberg, Germany: Springer Medizin. First citation in articleCrossref, Google Scholar

  • Hartmann, S., Upmeier zu Belzen, A., Krüger, D. & Pant, H. A. (2015). Scientific reasoning in higher education. Zeitschrift für Psychologie/Journal of Psychology, 223, 47–53). https://doi.org/10.1027/2151-2604/a000199 First citation in articleLink, Google Scholar

  • Hothorn, T., Bretz, F. & Westfall, P. (2008). Simultaneous inference in general parametric models. Biometrical Journal, 50, 346–363. https://doi.org/10.1002/bimj.200810425 First citation in articleCrossref, Google Scholar

  • Hu, L. & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. https://doi.org/10.1080/10705519909540118 First citation in articleCrossref, Google Scholar

  • John, O. P. & Robins, R. W. (1994). Accuracy and bias in self-perception: Individual differences in self-enhancement and the role of narcissism. Journal of Personality and Social Psychology, 66, 206–219. https://doi.org/10.1037/0022-3514.66.1.206 First citation in articleCrossref, Google Scholar

  • Kenny, S. S., Alberts, B., Booth, W. C., Glaser, M., Glassick, C. E., Ikenberry, S. O., … Yang, C. N. (1998). Reinventing undergraduate education: A blueprint for America’s research universities. Stoney Brook, NY: Boyer Commission on Educating Undergraduates in the Research University. First citation in articleGoogle Scholar

  • Klahr, D. & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12, 1–48. First citation in articleCrossref, Google Scholar

  • Koeppen, K., Hartig, J., Klieme, E. & Leutner, D. (2008). Current issues in competence modeling and assessment. Zeitschrift für Psychologie/Journal of Psychology, 216, 61–73. https://doi.org/10.1027/0044-3409.216.2.61 First citation in articleLink, Google Scholar

  • Köller, O., Trautwein, U. & Lüdtke, O. (2006). Zum Zusammenspiel von schulischer Leistung, Selbstkonzept und Interesse in der gymnasialen Oberstufe [On the interplay of academic achievement, self-concept, and Interest in upper secondary schools. Abstract]. Zeitschrift für Pädagogische Psychologie, 20, 27–39. https://doi.org/10.1024/1010-0652.20.1.27 First citation in articleLink, Google Scholar

  • Kruger, J. & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121–1134. https://doi.org/10.1037/0022-3514.77.6.1121 First citation in articleCrossref, Google Scholar

  • Mabe, P. A. & West, S. G. (1982). Validity of self-evaluation of ability: A review and meta-analysis. Journal of Applied Psychology, 67, 280–296. https://doi.org/10.1037/0021-9010.67.3.280 First citation in articleCrossref, Google Scholar

  • Marsh, H. W., Hau, K.-T. & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11, 320–341. https://doi.org/10.1207/s15328007sem1103_2 First citation in articleCrossref, Google Scholar

  • McClelland, D. C. (1973). Testing for competence rather than for “intelligence”. The American Psychologist, 28, 1–14. https://psycnet.apa.org/doi/10.1037/h0034092 First citation in articleCrossref, Google Scholar

  • McGrew, K. S. (2005). The Cattell-Horn-Carroll theory of cognitive abilities. In D. P. FlanaganP. L. HarrisonEds., Contemporary intellectual assessment: Theories, tests, and issues (2nd ed., pp. 136–181). New York, NY: Guilford. First citation in articleGoogle Scholar

  • McKinney, K., Howery, C. B., Strand, K. J., Kain, E. L. & Berheide, C. W. (2004). Liberal learning and the sociology major updated: Meeting the challenge of teaching sociology in the twenty-first century. Washington, DC: American Sociological Association. Retrieved from http://www.asanet.org/documents/teaching/pdfs/Lib_Learning_FINAL.pdf First citation in articleGoogle Scholar

  • Muthén, L. K. & Muthén, B. O. (2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén. First citation in articleGoogle Scholar

  • R Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation. First citation in articleGoogle Scholar

  • Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667–696. https://doi.org/10.1080/00273171.2012.715555 First citation in articleCrossref, Google Scholar

  • Revelle, W. (2016). psych: Procedures for psychological, psychometric, and personality research. Retrieved from https://cran.r-project.org/package=psych First citation in articleGoogle Scholar

  • Revelle, W. & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145–154. https://doi.org/10.1007/s11336-008-9102-z First citation in articleCrossref, Google Scholar

  • Roussos, L. & Stout, W. (1996). A multidimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20, 355–371. https://doi.org/10.1177/014662169602000404 First citation in articleCrossref, Google Scholar

  • Sass, D. A. (2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29, 347–363. https://doi.org/10.1177/0734282911406661 First citation in articleCrossref, Google Scholar

  • Schladitz, S., Groß Ophoff, J. & Wirtz, M. (2015). Konstruktvalidierung eines Tests zur Messung bildungswissenschaftlicher Forschungskompetenz [Construct validation of a test to measure educational research competency]. Zeitschrift für Pädagogik (Beiheft), 61, 167–184. First citation in articleGoogle Scholar

  • Schneider, W. J. & McGrew, K. S. (2012). The Cattell-Horn-Carroll model of intelligence. In D. FlanaganD. HarrisonEds., Contemporary intellectual assessment: Theories, tests, and issues (3rd ed., pp. 99–144). New York, NY: Guilford Press. First citation in articleGoogle Scholar

  • Schneider, W. J. & Newman, D. A. (2015). Intelligence is multidimensional: Theoretical review and implications of specific cognitive abilities. Human Resource Management Review, 25, 12–27. https://doi.org/10.1016/j.hrmr.2014.09.004 First citation in articleCrossref, Google Scholar

  • Schwarzer, R. & Jerusalem, M. (2002). Das Konzept der Selbstwirksamkeit [The concept of self-efficacy]. In M. JerusalemD. HopfEds., Selbstwirksamkeit und Motivationsprozesse in Bildungsinstitutionen (pp 28–53). Weinheim, Germany: Beltz. First citation in articleGoogle Scholar

  • Seaton, M., Marsh, H. W. & Craven, R. G. (2009). Big-fish-little-pond effect: Generalizability and moderation – two sides of the same coin. American Educational Research Journal (pp. 390–433). 47. https://doi.org/10.3102/0002831209350493 First citation in articleCrossref, Google Scholar

  • Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251. https://doi.org/10.1037/0033-2909.87.2.245 First citation in articleCrossref, Google Scholar

  • Swank, J. M. & Lambie, G. W. (2016). Development of the research competencies scale. Measurement and Evaluation in Counseling and Development, 49, 91–108. https://doi.org/10.1177/0748175615625749 First citation in articleCrossref, Google Scholar

  • Tiffin-Richards, S. P. & Pant, H. A. (2017). Arguing validity in educational assessment. In D. LeutnerJ. FleischerJ. GrünkornE. KliemeEds., Competence assessment in education: Research, models and instruments (pp. 469–485). Cham, Switzerland: Springer International. https://doi.org/10.1007/978-3-319-50030-0 First citation in articleGoogle Scholar

  • Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450. First citation in articleCrossref, Google Scholar

  • Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Ziegler, M. (2014). Stop and state your intentions! Let’s not forget the ABC of test construction. European Journal of Psychological Assessment, 30, 239–242. https://doi.org/10.1027/1015-5759/a000228 First citation in articleLink, Google Scholar

  • Ziegler, M. & Bühner, M. (2009). Modeling socially desirable responding and its effects. Educational and Psychological Measurement, 69, 548–565. First citation in articleCrossref, Google Scholar

  • Ziegler, M., Kemper, C. J. & Lenzner, T. (2015). The issue of fuzzy concepts in test construction and possible remedies. European Journal of Psychological Assessment, 31, 1–4. https://doi.org/10.1027/1015-5759/a000255 First citation in articleLink, Google Scholar

  • Zinbarg, R. E., Yovel, I., Revelle, W. & McDonald, R. P. (2006). Estimating generalizability to a latent variable common to all of a scale’s indicators: A comparison of estimators for ωh. Applied Psychological Measurement, 30, 121–144. https://doi.org/10.1177/0146621605278814 First citation in articleCrossref, Google Scholar

  • Zlatkin-Troitschanskaia, O., Pant, H. A. & Coates, H. (2016). Assessing student learning outcomes in higher education: Challenges and international perspectives. Assessment & Evaluation in Higher Education, 41, 655–661. https://doi.org/10.1080/02602938.2016.1169501 First citation in articleCrossref, Google Scholar