Validity Improvement in Two Reasoning Measures Following the Elimination of the Position Effect
Abstract
We report on an investigation of the consequences of preventing the position effect from contributing to the correlation between two reasoning scales considered an indicator of convergent validity. Confirmatory factor models served to separate this effect from the ability-specific components of measurement in Raven’s Advanced Progressive Matrices and Horn’s Numeric Reasoning Scale. These models additionally enabled investigation of the convergent validity of these measures. In a sample of 230 participants the ability-specific components of the two reasoning measures showed a very high correlation (.88), whereas the correlation of the position-related components was not significant. Since the measures represented different types of reasoning, we conclude that the ability-specific components represent fluid intelligence and the position-related components reflect the specificities of the different types of reasoning.
References
(2005). The impact of the Cattell-Horn-Carroll theory on test development and interpretation of cognitive and academic abilities. In , Contemporary intellectual assessment: Theories, tests, and issues (pp. 185–202). New York: Guilford.
(1993). Human cognitive abilities. New York: Cambridge University Press.
(1963). Theory of fluid and crystallized intelligence: A critical experiment. Journal of Educational Psychology, 54, 1–22.
(2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189–225.
(1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
(1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349–354.
(1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 56, 495–515.
(2008). Higher-order models versus direct hierarchical models: g as superordinate or breadth factor? Psychology Science Quarterly, 50, 21–43.
(2007). A confirmatory analysis of item reliability trends (CAIRT): Differentiating true score and error variance in the analysis of item context effects. Multivariate Behavioral Research, 42, 157–183.
(1983). Leistungsprüfsystem [
Performance-test-system ] (2nd ed.). Göttingen: Hogrefe.(1998). The g factor. The science of mental ability. Westport: Praeger.
(1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36, 109–133.
(2001). LISREL 8.5: User’s reference guide. Chicago, IL: Scientific Software International.
(1988). Item context effects on personality scales: Measuring changes the measure. Journal of Personality and Social Psychology, 55, 312–320.
(1996). Reliability shifts in measurement reactivity: Driven by content engagement or self-engagement? Journal of Personality and Social Psychology, 70, 1080–1090.
(1991). Psychometric shortcomings of Raven’s Standard Progressive Matrices SPM) in particular for computerized testing. European Review of Applied Psychology, 41, 295–300.
(1968). Statistical theories of mental scores. Reading, MA: Addison-Wesley.
(1985). Generalized linear models. London: Chapman and Hall.
(2009). CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37, 1–10.
(1989). Validity. In , Educational measurement (pp. 13–103). Washington, DC: American Council of Education and National Council on Measurement in Education.
(2007). Method effects due to social desirability as a parsimonious explanation of the deviation from unidimensionality in LOT-R scores. Personality and Individual Differences, 42, 1597–1607.
(1962). Advanced progressive matrices. London: Lewis & Co.
(1997). Raven’s progressive matrices and vocabulary scales. Edinburgh: J. C. Raven.
(2000). The Armed Services Vocational Aptitude Battery (ASVAB) little more than acculturated learning (Gc)!? Learning and Individual Differences, 12, 81–103.
(1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41, 1101–1114.
(2003). Perceptual processes and cognitive ability. Intelligence, 31, 211–235.
(2008). An investigation of the structure of the Social Optimism Scale in considering the dimensionality problem. Journal of Individual Differences, 29, 223–230.
(2010). Avoiding the effect of item wording by means of bipolar instead of unipolar items: An application to social optimism. European Journal of Personality, 24, 137–150.
(2009). The confirmatory investigation of APM items with loadings as a function of the position and easiness of items: A two-dimensional model of APM. Psychology Science Quarterly, 51, 47–64.
(1992). Thoughts on the nature of context effects. In N. Schwarz S. Sudman, Eds., Context effects in social and psychological research (pp. 163–184). New York: Springer.
(2009). Exploring the relations among executive functions, fluid intelligence, and personality. Journal of Individual Differences, 30, 194–200.
(2003). Dimensionality of the Revised Life Orientation Test and the status of filler items. International Journal of Psychology, 38, 390– 400.
(2000). A Rasch model for detecting learning while solving an intelligence test. Applied Psychological Measurement, 24, 151–162.
