Training Effects with the Observer- Rated Cognitive Errors and the Coping Action Patterns Scales
Abstract
The use of observer-rated scales requires that raters be trained until they have become reliable in using the scales. However, few studies properly report how training in using a given rating scale is conducted or indeed how it should be conducted. This study examined progress in interrater reliability over 6 months of training with two observer-rated scales, the Cognitive Errors Rating Scale and the Coping Action Patterns Rating Scale. The evolution of the intraclass correlation coefficients was modeled using hierarchical linear modeling. Results showed an overall training effect as well as effects of the basic training phase and of the rater calibration phase, the latter being smaller than the former. The results are discussed in terms of implications for rater training in psychotherapy research.
References
(1976). Cognitive therapy and the emotional disorders. New York: International Universities Press.
(1995). Cognitive therapy. Basics and beyond. New York: Guilford.
(1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 101, 147–158.
(2006). Rating scales, scales of measurement, issues of reliability. Resolving some critical issues for clinicians and researchers. The Journal of Nervous and Mental Disease, 194, 557–564.
(2007, June). An investigation of the relationship of cognitive errors and coping action patterns with the therapeutic alliance. Paper presented at the Canadian Psychological Association Annual Convention, Ottawa, Canada.
(2009). Cognitive errors, coping patterns, and the therapeutic alliance: A pilot study of in-session process. Counseling and Psychotherapy Research, 9, 108–114.
(2005, June). Observer ratings of coping and cognitive errors: The Cognitive Errors and Coping Action Patterns (CECAP) method. Paper presented at the 36th Annual Meeting of the Society for Psychotherapy Research, Montréal, Canada.
(2007, June). A pilot study of the validity and reliability of the CECAP method. Paper presented at the Canadian Psychological Association Annual Convention, Ottawa, Canada.
(2008). The cognitive errors rating system. Unpublished manuscript, McGill University, Montréal, Canada.
(1996). MIXREG: A computer program for mixed-effects regression analysis with autocorrelated errors. Computer Methods and Programs in Biomedicine, 49, 229–252.
(1979). Longitudinal study of the effects of rater training on psychometric error in ratings. Journal of Applied Psychology, 64, 502–508.
(1998). Data analysis in social psychology. In , The handbook of social psychology (4th ed., pp. 233–265). Boston, MA: McGraw-Hill.
(2009). Cognitive errors assessed by observer-ratings in bipolar affective disorder, symptoms, and therapeutic alliance. The Cognitive Behavior Therapist, 2, 92–105.
(2009). Early change of defense mechanisms and coping in short-term dynamic psychotherapy: Relations with symptoms and alliance. Clinical Psychology and Psychotherapy, 16, 408–417.
(2010). Change in defense mechanisms and coping over the course of short-term dynamic psychotherapy for adjustment disorder. Journal of Clinical Psychology, 66, 1232–1241.
(2009). Cognitive errors and coping action patterns in child molesters as assessed by external observers. The Open Criminology Journal, 2, 24–28.
(in press). Etude de validation de la version française des échelles de codage du coping et des erreurs cognitives (CE-CAP) sur une population nonclinique [
Validation studies of the French version of the rating scales on coping and cognitive errors (CE-CAP) on a nonclinical population ]. Annales Médico-Psychologiques(2009). Coping specificities in bipolar affective disorder: Relation with symptoms, symptom evolution and therapeutic alliance. German Journal of Psychiatry, 12, 19–27.
(1981). Cognitive distortion and cognitive errors in depressed psychiatric and low back pain patients. Journal of Consulting and Clinical Psychology, 49, 517–525.
(2006). Can minimally trained observers provide valid global ratings? Journal of Family Psychology, 20, 335–338.
(1998). Understanding transference: The conflictual relationship theme method (2nd ed.). Washington, DC: American Psychological Association.
(2005). The Coping Action Patterns Rating System. Unpublished manuscript, McGill University, Montréal, Canada.
(2007, June). Changes in cognitive errors and coping action patterns following 20 sessions of either dynamic psychotherapy or CBT of acute recurrent depression. Paper presented at the 38th Annual Meeting of the Society for Psychotherapy Research, Madison, WI.
(2004). Studying defense mechanisms in psychotherapy using the Defense Mechanisms Rating Scales. In , Defense mechanisms: Theoretical, research and clinical perspectives (pp. 165–192). Amsterdam: Elsevier.
(2008, June). The effects of varying amounts of training on the reliability of graduate students’ rating of complex clinical phenomena. Paper presented at the 39th Annual Meeting of the Society for Psychotherapy Research, Barcelona, Spain..
(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
(2003). Searching for the structure of coping: A review and critique of category systems for classifying ways of coping. Psychological Bulletin, 129, 216–269.
(2001). Interobserver disagreements on clinical dementia rating assessment: Interpretation and implications for training. Alzheimer Disease and Associated Disorders, 15, 155–161.
(2002). Beurteilerübereinstimmung und Beurteilerreliabilität [
Interrater agreement and reliability ]. Göttingen: Hogrefe.
