The Cognitive Interviewing Reporting Framework (CIRF)
Towards the Harmonization of Cognitive Testing Reports
Abstract
Cognitive interviewing is an important qualitative tool for the testing, development, and evaluation of survey questionnaires. Despite the widespread adoption of cognitive testing, there remain large variations in the manner in which specific procedures are implemented, and it is not clear from reports and publications that have utilized cognitive interviewing exactly what procedures have been used, as critical details are often missing. Especially for establishing the effectiveness of procedural variants, it is essential that cognitive interviewing reports contain a comprehensive description of the methods used. One approach to working toward more complete reporting would be to develop and adhere to a common framework for reporting these results. In this article we introduce the Cognitive Interviewing Reporting Framework (CIRF), which applies a checklist approach, and which is based on several existing checklists for reviewing and reporting qualitative research. We propose that researchers apply the CIRF in order to test its usability and to suggest potential adjustments. Over the longer term, the CIRF can be evaluated with respect to its utility in improving the quality of cognitive interviewing reports.
References
(2004). The dynamics of cognitive interviewing. In , Questionnaire development evaluation and testing methods (pp. 45–66). Hoboken, NJ: Wiley.
(2013). Examining the personal experience of Aging Scale with the Three Step Test Interview. Methodology, 9, 96–103.
(2010). Analysis in qualitative research. London, UK: Sage.
(2011). Making a difference: Towards a method for weighing the evidence in a qualitative synthesis. Journal of Evaluation in Clinical Practice, 17, 657–663.
(2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. Annals of Family Medicine, 6, 331–339.
(2007). Analysing and interpreting cognitive interview data: A qualitative approach. In Proceedings of the 6th Questionnaire Evaluation Standard for Testing Conference (pp. 64–73). Ottawa: Statistics Canada.
(2004). Data quality in cognitive interviews: The case for verbal reports. In , Questionnaire development evaluation and testing methods (pp. 67–87). Hoboken, NJ: Wiley.
(1996). Cognitive interviewing techniques: In the lab and in the field. In , Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 175–195). San Francisco, CA: Jossey-Bass.
(1980). Verbal reports as data. Psychological Review, 87, 215–251.
(2011). Identifying sources of error in cross-national questionnaires: Application of an error source typology to cognitive interview data. Journal of Official Statistics, 27, 1–32.
2009 Testing for equivalence using cross-national cognitive interviewing Center for comparative social surveys Working Paper Series, No 01.
(1997). Perspectives on pretesting: Cognition in the cognitive interview? Bulletin de Methodologie Sociologique, 55, 18–39.
(2004). The three-step test-interview (TSTI): An observational instrument for pretesting self-completion questionnaires. ERIM Report ERS-2004-029-ORG. Rotterdam, The Netherlands: Erasmus Research Institute of Management Retrieved from hdl.handle.net/1765/1265
(1996). Basic and applied memory research: Vol. 2. Practical application. Hillsdale, NJ: Erlbaum.
(1984). Cognitive aspects of survey methodology: Building a bridge between disciplines. Washington, DC: National Academy Press.
(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.
(1984). Protocol analysis of responses to survey recall questions. In , Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 61–64). Washington, DC: National Academy Press.
(2003). Conducting cognitive interviews to understand question-response limitations. American Journal of Health Behavior, 27(Suppl. 3), S264–S272.
(2011). Cognitive interviewing. In , Question evaluation methods: Contributing to the science of data quality. New York, NY: Wiley and Sons.
(2010). Results of a cross-national structured cognitive interviewing protocol to test measures of disability. Quality & Quantity, 4, 801–815.
(2001). Use of the CONSORT statement and quality of reports of randomized trials: A comparative before-and-after evaluation. JAMA, 285, 1992–1995.
(2013). Obtaining validity evidence by cognitive interviewing to interpret psychometric results. Methodology, 9, 113–122.
(1984). Cognitive science and survey methods: A cognitive perspective. In , Cognitive aspects of survey design: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.
(2013). Evaluating the Cognitive Interviewing Reporting Framework (CIRF) by rewriting a Dutch pretesting report of a European Health Survey Questionnaire. Methodology, 9, 104–112.
(1999). Cognitive interviewing: A how-to guide. Retrieved from appliedresearch.cancer.gov/areas/cognitive/interview.pdf.
(2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.
(2013). Reflections on the Cognitive Interviewing Reporting Framework: Efficacy, expectations, and promise for the future. Methodology, 9, 123–128.
(2007). Questionnaire design from a cross-cultural perspective: An empirical investigation of Koreans and Non-Koreans. Journal of Health Care for the Poor and Underserved, 18, 197–217.
