Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

Anchoring Vignettes

Can They Make Adolescent Self-Reports of Social-Emotional Skills More Reliable, Discriminant, and Criterion-Valid?

Published Online:April 22, 2016https://doi.org/10.1027/1015-5759/a000336

Abstract

Abstract. Individuals differ in the way they use rating scales to describe themselves, and these differences are particularly pronounced in children and early adolescents. One promising remedy is to correct (or “anchor”) an individual’s responses according to the way they use the scale when they rate an anchoring vignette (a set of hypothetical targets differing on the attribute of interest). Studying adolescents’ self-reports of their socio-emotional attributes, we compared traditional self-report scores with vignette-corrected scores in terms of reliability (internal consistency), discriminant validity (scale intercorrelations), and criterion validity (predicting achievement test scores in language and math). A large and representative sample of 12th grade Brazilian students (N = 8,582, 62% female, mean age 18.2) were administered a Portuguese-language self-report inventory assessing social-emotional skills related to the Big Five personality dimensions. Correcting scores according to vignette ratings led to increases in the reliability of scales measuring Conscientiousness and Openness, but discriminant validity and criterion validity increased only when each scale was corrected using its own corresponding vignette set. Moreover, accuracy in rating the vignettes was correlated with language achievement test scores, suggesting that verbal factors play a role in providing both normative vignette ratings of others and self-reports that are reliable and valid.

References

  • Ananiadou, K. & Claro, M. (2009). 21st century skills and competences for New Millennium Learners in OECD countries. Paris, France: Centre for Educational Research and Innovation (CERI) – New Millennium Learners. First citation in articleCrossref, Google Scholar

  • Antonakis, J., Bendahan, S., Jacquart, P. & Lalive, R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21, 1086–1120. First citation in articleCrossref, Google Scholar

  • Austin, E. J., Deary, I. J. & Egan, V. (2006). Individual differences in response scale use: Mixed Rasch modelling of responses to NEO-FFI items. Personality and Individual Differences, 40, 1235–1245. First citation in articleCrossref, Google Scholar

  • Bolt, M. D. & Johnson, T. R. (2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33, 335–352. doi: 10.1177/0146621608329891 First citation in articleCrossref, Google Scholar

  • Cronbach, L. J. (1946). Response sets and test validity. Educational and Psychological Measurement, 6, 475–494. First citation in articleCrossref, Google Scholar

  • De Fruyt, F., Bartels, M., Van Leeuwen, K. G., De Clercq, B., Decuyper, M. & Mervielde, I. (2006). Five types of personality continuity in childhood and adolescence. Journal of Personality and Social Psychology, 91, 538–552. doi: 10.1037/0022-3514.91.3.538 First citation in articleCrossref, Google Scholar

  • De Fruyt, F., Wille, B. & John, O. P. (2015). Employability in the 21st century: Complex (interactive) problem solving and other essential skills. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 276–281. First citation in articleCrossref, Google Scholar

  • Hamamura, T., Heine, S. J. & Paulhus, D. L. (2008). Cultural differences in response styles: The role of dialectical thinking. Personality and Individual Differences, 44, 932–942. First citation in articleCrossref, Google Scholar

  • Hamilton, D. L. (1968). Personality attributes associated with extreme response style. Psychological Bulletin, 69, 192–203. First citation in articleCrossref, Google Scholar

  • Harzing, A.-W. (2006). Response styles in cross-national survey research. International Journal of Cross-Cultural Management, 6, 243–266. First citation in articleCrossref, Google Scholar

  • Hausman, J. & Taylor, W. (1981). Panel data and unobservable individual effects. Econometrica, 49, 1377–1398. First citation in articleCrossref, Google Scholar

  • He, J., Bartram, D., Inceoglu, I. & van de Vijver, F. J. R. (2014). Response styles and personality traits: A multilevel analysis. Journal of Cross-Cultural Psychology, 45, 1028–1045. doi: 10.1177/0022022114534773 First citation in articleCrossref, Google Scholar

  • Hox, J. J. (2010). Multilevel Analysis: Techniques and Applications. New York, NY: Routledge. First citation in articleCrossref, Google Scholar

  • Johanson, G. A. & Osborn, C. J. (2004). Acquiescence as differential person functioning. Assessment & Evaluation in Higher Education, 29, 535–548. doi: 10.1080/02602930410001689126 First citation in articleCrossref, Google Scholar

  • John, O. P., Caspi, A., Robins, R., Moffitt, T. E. & Stouthamer-Loeber, M. (1994). The “Little Five”: Exploring the nomological network of the five-factor model of personality in adolescent boys. Child Development, 65, 160–178. First citation in articleCrossref, Google Scholar

  • John, O. P., Naumann, L. & Soto, C. J. (2008). Paradigm shift to the integrative Big Five trait taxonomy: Discovery, measurement, and conceptual issues. In O. P. JohnR. W. RobinsL. A. PervinEds., Handbook of personality: Theory and research (3rd ed., pp. 114–158). New York, NY: Guilford. First citation in articleGoogle Scholar

  • John, O. P. & Robins, R. W. (1993). Determinants of interjudge agreement: The Big Five, observability, evaluativeness, and the unique perspective of the self. Journal of Personality, 61, 521–551. First citation in articleCrossref, Google Scholar

  • King, G., Murray, C. J., Salomon, J. A. & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98, 191–207. First citation in articleCrossref, Google Scholar

  • King, G. & Wand, J. (2007). Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis, 15, 46–66. First citation in articleCrossref, Google Scholar

  • Kyllonen, P. C. & Bertling, J. P. (2014). Innovative questionnaire assessment methods to increase cross-country comparability. In L RutkowskiM. Von DavierD. RutkowskiEds., Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis (pp. 277–286). Boca Raton: Taylor & Francis. First citation in articleGoogle Scholar

  • Kyllonen, P. C. & Bertling, J. P. (2014). Draft report: anchoring vignettes reduce bias in noncognitive rating scale responses. Princeton, NJ: ETS/OECD. First citation in articleGoogle Scholar

  • Kyllonen, P. C., Lipnevich, A. A., Burrus, J. & Roberts, R. D. (2008). Personality, motivation, and college readiness: a prospectus for assessment and development.Non printed technical report. Educational Testing Service, Princeton. Retrieved from: http://onlinelibrary.wiley.com/doi/10.1002/ets2.12004/epdf First citation in articleGoogle Scholar

  • Lentz, T. F. (1938). Acquiescence as a factor in the measurement of personality. Psychological Bulletin, 35, 659. First citation in articleGoogle Scholar

  • Marin, G., Gamba, R. J. & Marin, B. V. (1992). Extreme response style and acquiescence among Hispanics. Journal of Cross-Cultural Psychology, 23, 498–509. First citation in articleCrossref, Google Scholar

  • McCrae, R. R. & John, O. P. (1992). An introduction to the five-factor model and its applications. Journal of Personality, 60, 175–215. doi: 10.1111/j.1467-6494.1992.tb00970.x First citation in articleCrossref, Google Scholar

  • McCrae, R. R. & Terracciano, A. (2005). Personality profiles of cultures: Aggregate personality traits. Journal of Personality and Social Psychology, 89,(3), 407–425. doi: 10.1037/0022-3514.89.3.407 First citation in articleCrossref, Google Scholar

  • Mõttus, R., Allik, J., Realo, A., Pullmann, H., Rossier, J., Zecca, G., … Tseung, C. N. (2012). Comparability of self-reported conscientiousness across 21 countries. European Journal of Personality, 26, 303–317. doi: ez1.periodicos.capes.gov.br/10.1002/per.840 First citation in articleCrossref, Google Scholar

  • Mõttus, R., Allik, J., Realo, A., Rossier, J., Zecca, G., Ah-Kion, J., … Johnson, W. (2012). The effect of response style on self-reported conscientiousness across 20 countries. Personality and Social Psychology Bulletin, 38, 1423–1436. doi: 10.1177/0146167212451275 First citation in articleCrossref, Google Scholar

  • Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. RobinsonP. R. ShaverL. S. WrightsmanEds., Measures of social psychological attitudes, Vol. 1. Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press. First citation in articleGoogle Scholar

  • Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological bulletin, 135, 322–338. First citation in articleCrossref, Google Scholar

  • Primi, R., Santos, D., John, O. P. & De Fruyt, F. (2016). The development of a nationwide inventory assessing social and emotional skills in Brazilian youth. European Journal of Psychological Assessment, 32, 5–16. doi: 10.1027/1015-5759/a000343 First citation in articleLink, Google Scholar

  • R Development Core Team (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org First citation in articleGoogle Scholar

  • Revelle, W. (2014). psych: Procedures for personality and psychological research. Evanston, IL, USA: Northwestern University. First citation in articleGoogle Scholar

  • Santos, D. & Primi, R. (2014). Social and emotional development and school learning: a measurement proposal in support of public policy Technical report for Organization for Economic Cooperation and Development (OCDE) Rio de Janeiro State Education Department (SEEDUC) and Ayrton Senna Institute. São Paulo, Brazil: Ayrton Senna Institute. First citation in articleGoogle Scholar

  • Soto, C. J. & John, O. P. (2009). Ten facet scales for the Big Five Inventory: Convergence with NEO PI-R facets, self-peer agreement, and discriminant validity. Journal of Research in Personality, 43, 84–90. First citation in articleCrossref, Google Scholar

  • Soto, C. J., John, O. P., Gosling, S. D. & Potter, J. (2008). The developmental psychometrics of Big Five self-reports: Acquiescence, factor structure, coherence, and differentiation from ages 10 to 20. Journal of Personality and Social Psychology, 94, 718–737. First citation in articleCrossref, Google Scholar

  • Tandon, A., Murray, C. J., Salomon, J. A. & King, G. (2003). Statistical models for enhancing cross-population comparability In Health systems performance assessment: Debates, methods and empiricism (pp. 727–746). Geneva: World Health Organization. Retrieved from http://www.who.int/healthinfo/paper42.pdf First citation in articleGoogle Scholar

  • Trilling, B. & Fadel, C. (2009). 21st century skills: Learning for life in our times. San Francisco, CA: Jossey-Bass. First citation in articleGoogle Scholar

  • Van Herk, H., Poortinga, Y. H. & Verhallen, T. M. (2004). Response styles in rating scales evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35(3), 346–360. doi: 10.1177/0022022104264126 First citation in articleCrossref, Google Scholar

  • Von Stumm, S. & Ackerman, P. L. (2012). Investment and intellect: A review and meta-analysis. Psychological Bulletin, 139, 841–869. First citation in articleCrossref, Google Scholar

  • Wand, J., King, G. & Lau, O. (2011). Anchors: Software for anchoring vignette data. Journal of Statistical Software, Forthcoming, Retrieved from http://www.jstatsoft.org First citation in articleGoogle Scholar

  • Wille, B., De Fruyt, F. & De Clercq, B. (2013). Expanding and reconceptualizing aberrant personality at work: Validity of Five-Factor Model aberrant personality tendencies to predict career outcomes. Personnel Psychology, 66, 173–223. First citation in articleCrossref, Google Scholar