Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

Obtaining Validity Evidence by Cognitive Interviewing to Interpret Psychometric Results

Published Online:January 01, 2013https://doi.org/10.1027/1614-2241/a000073

Abstract

The latest edition of the Standards for Educational and Psychological Testing (APA, 1999) promotes the analysis of respondents’ response processes in order to obtain evidence about the fit between the intended construct and the response process produced. The aim of this paper was twofold. First, we assess whether cognitive interviewing can be used to gather such validity evidence, and secondly, to analyze the usefulness of the evidence provided for interpreting the results from traditional psychometric analysis. The usefulness of the Cognitive Interviewing Reporting Framework (Boeije & Willis, 2013) for reporting the cognitive interviewing findings was also evaluated. As an empirical example, we tested the (Spanish language) APGAR family function scale. A total of 21 pretest cognitive interviews were conducted, and psychometric analyzes were conducted of data from 28,371 respondents who were administered the APGAR scale. Results and utility of CIRF as a reporting framework are discussed.

References

  • American Psychological Association American Educational Research Association National Council on Measurement in Education . (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association. First citation in articleGoogle Scholar

  • Austin, J. K. , Huberty, T. J. (1989). Revision of the family APGAR for use by 8-year-olds. Family Systems Medicine, 7, 323–327. First citation in articleCrossref, Google Scholar

  • Beatty, P. , Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71, 287–311. First citation in articleGoogle Scholar

  • Bellón, J. A. , Delgado, J. , Luna, P. , Lardelli, P. (1996). Validez y fiabilidad del cuestionario de función familiar Apgar-familiar. Atención Primaria, 18, 289–296. First citation in articleGoogle Scholar

  • Boeije, H. R. , Willis, G. (2013). The Cognitive Interviewing Reporting Framework (CIRF): Towards the harmonization of cognitive testing reports. Methodology, 9, 87–95. doi: 10.1027/1614-2241/a000075 First citation in articleGoogle Scholar

  • Carmines, E. G. , Zeller, R. A. (1979). Reliability and validity assessment. London, UK: Sage. First citation in articleCrossref, Google Scholar

  • Castillo, M. , Padilla, J. L. , Gómez-Benito, J. , Andrés, A. (2010). A productivity map of cognitive pre-test methods for improving survey questions. Psicothema, 22, 482–488. First citation in articleGoogle Scholar

  • Conrad, F. G. , Blair, J. (2009). Sources of error in cognitive interviews. Public Opinion Quarterly, 73, 32–55. First citation in articleCrossref, Google Scholar

  • Conrad, F. G. , Blair, J. , & Tracy, E. (1999). Verbal reports are data! A theoretical approach to cognitive interviews. Proceedings of the Federal Committee on Statistical Methodology Research Conference (pp. 11–20). Arlington, VA: FCSM. First citation in articleGoogle Scholar

  • DeWalt, D. A. , Rothrock, N. , Yount, S. , Stone, A. A. (2007). Evaluation of item candidates the PROMIS qualitative item review. Medical Care, 45, S12–S21. First citation in articleCrossref, Google Scholar

  • Feldt, L. (1969). A test of the hypothesis that Cronbach’s alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363–373. First citation in articleGoogle Scholar

  • Hardison, H. G. , Neimeyer, R. A. (2007). Numbers and narratives: Quantitative and qualitative convergence across constructivist assessments. Journal of Constructivist Psychology, 20, 285–308. First citation in articleCrossref, Google Scholar

  • Knafl, K. , Deatrick, J. , Gallo, A. , Holcombe, G. , Bakitas, M. , Dixon, J. , Grey, M. (2007). Focus on research methods: The analysis and interpretation of cognitive interviews for instrument development. Research in Nursing & Health, 30, 224–234. First citation in articleCrossref, Google Scholar

  • Mauricio, J. , Romero, N. , Saa, H. A. , Herrera, J. A. , Reyes-Ortiz, C. A. (2006). Prevalence of religiosity, family function, social support and depressive symptoms in old people. Revista Colombia Médica, 37(2 Suppl. 1), 26–30. First citation in articleGoogle Scholar

  • Miller, K. (2007). Design and analysis of cognitive interviews for cross-national testing. 2007 European Survey Research Association Annual Meeting, Prague, Czechoslovakia. First citation in articleGoogle Scholar

  • Olt, H. , Jirwe, M. , Gustavsson, P. , Azita, E. (2010). Cultural competence among healthcare professionals revised (IAPCC-R). Journal of Transcultural Nursing, 21, 55. First citation in articleCrossref, Google Scholar

  • Poole, H. M. , Murphy, P. , & Nurmikko, T. J. (2009). Development and preliminary validation of the NePIQoL: A quality-of-life measure for neuropathic pain. Journal of Pain and Symptom Management, 37, 233–245. First citation in articleCrossref, Google Scholar

  • Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4, 207–230. First citation in articleCrossref, Google Scholar

  • Sarkisian, C. A. , Hays, R. D. , Berry, S. , & Mangione, C. M. (2002). Development, reliability, and validity of the expectations regarding aging ERA-38 survey. The Gerontologist, 42, 534. First citation in articleCrossref, Google Scholar

  • Smilkstein, G. (1978). The Family APGAR: A proposal for family function test and its use by physicians. Journal of Family Practice, 6, 1231–1239. First citation in articleGoogle Scholar

  • Spanish Ministry of Health and Consume . (2006). Encuesta Nacional de Salud de España 2006. [National Health Survey in Spain 2006] Madrid, Spain: Ministerio de Sanidad y Consumo. First citation in articleGoogle Scholar

  • SPSS, Inc. (2007). SPSS-16 User’s guide. Chicago, IL. First citation in articleGoogle Scholar

  • Tourangeau, R. (2003). Cognitive aspects of survey measurement and mismeasurement. International Journal of Public Opinion Research, 15, 3–7. First citation in articleCrossref, Google Scholar

  • Willis, G. B. (2005). Cognitive interviewing. Thousand Oaks, CA: Sage. First citation in articleCrossref, Google Scholar

  • Wolnitzki, L. , Vargas, N. , Cerón, C. , Errázuriz, G. , Fabres, J. , Morales, J. C. , … Varela, P. (1989). Social net in support of adolescents scholars: Its appraisal by Smilkstein Apgar test. Boletín Hospital San Juan de Dios, 36, 147–155. First citation in articleGoogle Scholar