Anchoring Vignettes
Can They Make Adolescent Self-Reports of Social-Emotional Skills More Reliable, Discriminant, and Criterion-Valid?
Abstract
Abstract. Individuals differ in the way they use rating scales to describe themselves, and these differences are particularly pronounced in children and early adolescents. One promising remedy is to correct (or “anchor”) an individual’s responses according to the way they use the scale when they rate an anchoring vignette (a set of hypothetical targets differing on the attribute of interest). Studying adolescents’ self-reports of their socio-emotional attributes, we compared traditional self-report scores with vignette-corrected scores in terms of reliability (internal consistency), discriminant validity (scale intercorrelations), and criterion validity (predicting achievement test scores in language and math). A large and representative sample of 12th grade Brazilian students (N = 8,582, 62% female, mean age 18.2) were administered a Portuguese-language self-report inventory assessing social-emotional skills related to the Big Five personality dimensions. Correcting scores according to vignette ratings led to increases in the reliability of scales measuring Conscientiousness and Openness, but discriminant validity and criterion validity increased only when each scale was corrected using its own corresponding vignette set. Moreover, accuracy in rating the vignettes was correlated with language achievement test scores, suggesting that verbal factors play a role in providing both normative vignette ratings of others and self-reports that are reliable and valid.
References
2009). 21st century skills and competences for New Millennium Learners in OECD countries. Paris, France: Centre for Educational Research and Innovation (CERI) – New Millennium Learners.
(2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21, 1086–1120.
(2006). Individual differences in response scale use: Mixed Rasch modelling of responses to NEO-FFI items. Personality and Individual Differences, 40, 1235–1245.
(2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33, 335–352. doi: 10.1177/0146621608329891
(1946). Response sets and test validity. Educational and Psychological Measurement, 6, 475–494.
(2006). Five types of personality continuity in childhood and adolescence. Journal of Personality and Social Psychology, 91, 538–552. doi: 10.1037/0022-3514.91.3.538
(2015). Employability in the 21st century: Complex (interactive) problem solving and other essential skills. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 276–281.
(2008). Cultural differences in response styles: The role of dialectical thinking. Personality and Individual Differences, 44, 932–942.
(1968). Personality attributes associated with extreme response style. Psychological Bulletin, 69, 192–203.
(2006). Response styles in cross-national survey research. International Journal of Cross-Cultural Management, 6, 243–266.
(1981). Panel data and unobservable individual effects. Econometrica, 49, 1377–1398.
(2014). Response styles and personality traits: A multilevel analysis. Journal of Cross-Cultural Psychology, 45, 1028–1045. doi: 10.1177/0022022114534773
(2010). Multilevel Analysis: Techniques and Applications. New York, NY: Routledge.
(2004). Acquiescence as differential person functioning. Assessment & Evaluation in Higher Education, 29, 535–548. doi: 10.1080/02602930410001689126
(1994). The “Little Five”: Exploring the nomological network of the five-factor model of personality in adolescent boys. Child Development, 65, 160–178.
(2008).
(Paradigm shift to the integrative Big Five trait taxonomy: Discovery, measurement, and conceptual issues . In O. P. JohnR. W. RobinsL. A. PervinEds., Handbook of personality: Theory and research (3rd ed., pp. 114–158). New York, NY: Guilford.1993). Determinants of interjudge agreement: The Big Five, observability, evaluativeness, and the unique perspective of the self. Journal of Personality, 61, 521–551.
(2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98, 191–207.
(2007). Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis, 15, 46–66.
(2014).
(Innovative questionnaire assessment methods to increase cross-country comparability . In L RutkowskiM. Von DavierD. RutkowskiEds., Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis (pp. 277–286). Boca Raton: Taylor & Francis.2014). Draft report: anchoring vignettes reduce bias in noncognitive rating scale responses. Princeton, NJ: ETS/OECD.
(2008). Personality, motivation, and college readiness: a prospectus for assessment and development.Non printed technical report. Educational Testing Service, Princeton. Retrieved from: http://onlinelibrary.wiley.com/doi/10.1002/ets2.12004/epdf
(1938). Acquiescence as a factor in the measurement of personality. Psychological Bulletin, 35, 659.
(1992). Extreme response style and acquiescence among Hispanics. Journal of Cross-Cultural Psychology, 23, 498–509.
(1992). An introduction to the five-factor model and its applications. Journal of Personality, 60, 175–215. doi: 10.1111/j.1467-6494.1992.tb00970.x
(2005). Personality profiles of cultures: Aggregate personality traits. Journal of Personality and Social Psychology, 89,(3), 407–425. doi: 10.1037/0022-3514.89.3.407
(2012). Comparability of self-reported conscientiousness across 21 countries. European Journal of Personality, 26, 303–317. doi: ez1.periodicos.capes.gov.br/10.1002/per.840
(2012). The effect of response style on self-reported conscientiousness across 20 countries. Personality and Social Psychology Bulletin, 38, 1423–1436. doi: 10.1177/0146167212451275
(1991).
(Measurement and control of response bias . In J. P. RobinsonP. R. ShaverL. S. WrightsmanEds., Measures of social psychological attitudes, Vol. 1. Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press.2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological bulletin, 135, 322–338.
(2016). The development of a nationwide inventory assessing social and emotional skills in Brazilian youth. European Journal of Psychological Assessment, 32, 5–16. doi: 10.1027/1015-5759/a000343
(2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org
(2014). psych: Procedures for personality and psychological research. Evanston, IL, USA: Northwestern University.
(2014). Social and emotional development and school learning: a measurement proposal in support of public policy Technical report for Organization for Economic Cooperation and Development (OCDE) Rio de Janeiro State Education Department (SEEDUC) and Ayrton Senna Institute. São Paulo, Brazil: Ayrton Senna Institute.
(2009). Ten facet scales for the Big Five Inventory: Convergence with NEO PI-R facets, self-peer agreement, and discriminant validity. Journal of Research in Personality, 43, 84–90.
(2008). The developmental psychometrics of Big Five self-reports: Acquiescence, factor structure, coherence, and differentiation from ages 10 to 20. Journal of Personality and Social Psychology, 94, 718–737.
(2003). Statistical models for enhancing cross-population comparability In Health systems performance assessment: Debates, methods and empiricism (pp. 727–746). Geneva: World Health Organization. Retrieved from http://www.who.int/healthinfo/paper42.pdf
(2009). 21st century skills: Learning for life in our times. San Francisco, CA: Jossey-Bass.
(2004). Response styles in rating scales evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35(3), 346–360. doi: 10.1177/0022022104264126
(2012). Investment and intellect: A review and meta-analysis. Psychological Bulletin, 139, 841–869.
(2011). Anchors: Software for anchoring vignette data. Journal of Statistical Software, Forthcoming, Retrieved from http://www.jstatsoft.org
(2013). Expanding and reconceptualizing aberrant personality at work: Validity of Five-Factor Model aberrant personality tendencies to predict career outcomes. Personnel Psychology, 66, 173–223.
(