Interactions with Standardized Patients to Evaluate Students’ Psychotherapy-Competencies

Psychotherapy training entails learning about models of psychopathology and the theory of psychological interventions. At the same time, students need to acquire practical skills to become therapists (Vogel & Alpers, 2009). German federal law recently restructured the training of psychotherapists (Der Bundestag, 2019), the most important consequence being that training is now integrated into university-based master’s programs within established psychology departments. While university exams traditionally focus on the assessment of knowledge, this reform now requires the assessment of therapeutic performance as well (Vogel & Alpers, 2009). As a logical consequence, new licensing exams (Approbationsprüfung) require the formal assessment of students’ practical skills. Highly specified licensing exams include a round of interviews of each student with trained standardized patients (SPs) who simulate a realistic scenario for a clinical interview (so-called Parcourprüfung). Future licensing is thus preceded by a reliable assessment of each candidate’s competencies in these clinical interviews. We are unaware of similar practical exams, since previously licensing exams focused either on the acquisition of knowledge or on case formulations. Because the didactic method of SPs is relatively new to the field and formal assessments of competencies in such interviews have yet to be evaluated, we explored whether competencies can be evaluated based on interviews with SPs.

Psychotherapy training in Germany is regulated by federal law, and for the last 20 years, it has been carried out as highly structured on-the-job postgraduate training. Recently, new legislation further prescribed integrated academic training within psychology departments (Der Bundestag, 2019). Importantly, this recent development corresponds to international training models, in particular the so-called scientist-practitioner model, which is well established in the United States (see (Alpers & Hofmann, 2007). By closing the gap between research and practice, this reform also addresses the criticism that current master’s programs in clinical psychology are mostly theoretically oriented (Bergold, 2008; Rief et al., 2012; Schiefele & Jacob-Ebbinghaus, 2006; Wentura et al., 2013).

Federal law now defines the relevant content of the masters’ programs and specifies which competencies are required to pass licensing exams after their completion. This is justified by the need to guarantee compliance with patients’ safety, although graduates obtain their license much earlier than in the previous model of postgraduate training (Bundesministerium für Gesundheit, 2020). Therefore, the new master’s program requires more practical training and consequentially the assessment of each graduate’s therapeutic performance (Rief et al., 2014).

Several didactic approaches have been implemented to include practical skills training in the course of psychology programs and psychotherapy training. Of course, sooner or later students should be exposed to real patients with real mental-health issues – the individuals they treat after graduation. Interacting with patients provides the most realistic impression of practical competencies as demanded to adequately perform psychotherapy. Internships and placements therefore form an integral part of any structured psychotherapy training. However, it is not always feasible to include real patients in classroom teaching because of practicability and didactic hurdles. First, enormous costs are involved for instructors and their patients; second, patients are often hesitant to participate in educational programs (see Alpers & Steiger-White, 2020).

Much easier to implement in teaching are simulations of interactions with patients. Peer-to-peer role-playing has traditionally been used to enact psychotherapy in the classroom. Indeed, role-playing has a strong track record to illustrate relevant aspects of professional interaction; it has proven to prepare the students for real-life interaction with patients with mental-health issues (Bennett-Levy et al., 2009; Fairburn & Cooper, 2011; Mitterhofer et al., 2011). In such role-playing, students alternate between the role of the patient and that of the psychotherapist. Although peer-to-peer role-playing is an important first step in practical training it suffers from specific limitations (Alpers & Steiger-White, 2020), and it is rather challenging to simulate authentic interactions. First, while students concentrate on their therapeutic role, they are typically not trained to authentically portray the patient role. Second, patient roles and their anamnestic history are usually developed ad-hoc. Third, and most importantly, peer-to-peer role-playing takes place in a familiar environment with peers of similar demographic characteristics who may know each other (in their role as a student).

More effective are external actors, who can be instructed to enact the patient role. As their roles and enactment can be well rehearsed and standardized, they are often referred to as SPs. Because they can be selected and trained to portray authentic patient roles, modules that include such SPs can offer an intermediate step between peer-to-peer role-playing and exposure to actual patients. Moreover, thanks to their high degree of standardization, they may be more suitable for exams than role-playing with other students as well as interactions with real patients.

The general approach to using SPs in training and also in testing is well established in medical schools (Barrows, 1993; Köllner et al., 2016; McNaughton et al., 2008), and this didactic method is now an integral part of medical training in Germany (Bundesministerium für Gesundheit, 2017). In medical schools, SPs are usually recruited from the general population; training procedures and role scripts have been established. This can be done so successfully that well-trained actors with medical conditions cannot be differentiated from real patients (e. g., Ortwein et al., 2006).

SPs are much less established in psychology departments and the formal training of psychotherapists. One reason may be that it is much more difficult to enact a patient with a mental disorder than someone with, say, abdominal pain. Although we acknowledge that mental disorders are difficult to portray, we argue that the challenge posed by portraying mental disorders is one of the most important reasons for utilizing well-trained actors. While traditional curricula have heavily leaned on role-playing among peers, trained actors are likely the more realistic option. Indeed, positive experiences with this didactic method are growing, and many studies have started to include it in the current postgraduate training programs for psychotherapists (Eckel, Alpers et al., 2014; Eckel, Merod et al., 2014; Nikendei et al., 2019; Partschefeld et al., 2013). We recently implemented a teaching module with SPs in our university’s curriculum for the master’s in psychology to foster teaching based on the scientist-practitioner model. Our motivation was the anticipation of the revised psychotherapy training (Alpers & Steiger-White, 2020).

From the perspective of a multilevel concept of psychotherapy training, we decided to first focus on practical competencies (Vogel & Alpers, 2009), respectively, on basic therapeutic techniques (Linden et al., 2007). As part of our class on therapeutic interviewing, students were first introduced to the theoretical models and then practiced basic interviewing skills in an anamnestic patient interview. At the end of the course, students were asked to use the knowledge they had acquired and what they had already practiced in peer-to-peer role-playing in a simulated interview with a trained SP. This module was experienced as particularly useful by the instructors and very well received by the students; course evaluations were excellent and revealed great satisfaction. Most importantly, the students’ psychotherapeutic self-efficacy increased over time. Interestingly, this progress was most pronounced on those dimensions of therapeutic self-efficacy that were addressed in the class. These findings show first and promising evidence that the method of SPs is well applicable in a master’s program of clinical psychology and may thus be useful for the new training format at German universities (Alpers & Steiger-White, 2020).

While there is a strong case that SPs are useful as a teaching method, it remains unclear whether interactions with SPs can be used to objectively evaluate therapeutic competencies in exams. A necessary first step in documenting its usefulness would thus be to identify measures that allow a reliable assessment of students’ competencies in simulated interactions.

The proper assessment of therapeutic competencies in psychotherapists has long been a research topic (Shaw & Dobson, 1988). From a research perspective, it is particularly important for treatment evaluation studies, where therapists’ performance is typically examined concerning therapy outcome (Beidas & Kendall 2010; Dobson & Kazantzis, 2003; Kazantzis, 2003; Kuyken &Tsivrikos, 2009; Trepka et al., 2004). Previous research evaluated these skills with several heterogeneous methodological approaches, not all of which are well-validated (see Weck et al., 2010).

One of the more established tools for evaluating therapeutic competency is the Cognitive Therapy Scale (CTS; see Young & Beck, 1980, 1986). The rating scale has been used in many different settings, often in psychotherapy outcome studies, and, most importantly for our paper, it has been used to study the effects of psychotherapy training. Fortunately, there is an adapted and validated German version with very good psychometric properties (Weck et al., 2010). Although there is convincing evidence that the CTS is a useful tool to assess psychotherapeutic competencies in licensed psychotherapists (see Weck et al., 2010), it has not been well established whether the scale is also useful to evaluate performance at an earlier stage of trainees’ university-level master’s studies. Moreover, we are unaware of any application of the scale to evaluate interactions in simulations with SPs. However, we argue that such instruments may bear potential for the assessment in the mandatory licensing exam, which must be established for the new licensing procedure in Germany (Bundesministerium für Gesundheit, 2020).

We therefore evaluated the properties of the scale in this pilot study by coding videotaped material from our master’s students’ program in clinical psychology (Alpers & Steiger-White, 2020). To this end, we used the CTS to evaluate psychotherapeutic competencies in students.

Our first goal was to document the usefulness of an established scale regarding our students’ performance. Only if ratings of competency can be achieved economically (practicability and time costs) would such an evaluation be reasonable for routine evaluations on a large scale. A standardized scale to evaluate therapeutic competency must be adaptable to the current setting and task. Second, we examined whether students’ performance can be reliably measured on master’s students. Third, as an indication of the validity of such ratings, we compared our students’ scores with those of more experienced therapists (see Weck et al., 2016). Only if values within our sample had a certain range, and only if they plausibly differed from those obtained from more experienced psychotherapists, would we conclude that the information we obtain is meaningful for future exams.

Taken together, this information is essential to determine whether simulations of interviews with SPs are applicable in licensing exams as now required nationwide in Germany. In the face of the growing international interest in better-structured training procedures, this may contribute to the dissemination of efficient and effective training and testing procedures for much-needed therapists.

Method

Procedure

The Class

The interview to be evaluated for this study was part of a mandatory small-group seminar conducted in a block of two and a half days as part of the master’s program in clinical psychology (Gesprächsführungsseminar). Students received course credit (two European Credit Transfer and Accumulation System points) for their participation. The general goal of the course was to teach communication and interviewing techniques and to help students to deal with difficult interviewing situations. During the seminar, there were several practical units where students worked in triads to practice their communication and interviewing skills. Figure 1 illustrates the general procedure.

The SP Interviews

On the last day of the course, each student conducted a 20-minute anamnestic interview with one of the SPs, who was trained to portray one of the roles described elsewhere (Steiger-White & Alpers, 2020). While one student interviewed the SP, another student observed the interview silently to learn from observation and later provide informal feedback. Only when both student and the SPs provided informed consent were the sessions videotaped for subsequent evaluation.

The students’ task was to conduct a 20-minute interview with a patient to gain enough information on the problem, the probable diagnosis, and an overview of the case history. Importantly, the SPs were instructed to closely follow their role scripts and also simulate one of four particularly challenging behaviors, such as questioning the interviewer’s therapeutic competence, excessive lamenting, suicidal tendencies, and striving for intimacy. These challenges were chosen based on conceptual considerations designed as representations of common interpersonal difficulties in psychotherapy sessions (Noyon & Heidenreich, 2013) and were categorized by one of the two independent raters.

At the end of the simulated interview, the master’s students received feedback, first from the SPs, then from the observer.

Results

Interrater Reliability of Competency Evaluation

The adapted scales internal consistency still had an acceptable Cronbach’s α-value of .78 for the global score (Field, 2013, Nunnally, 1978). Table 1 shows the agreement of the evaluations of the raters who did their assessments independently from each other. All items had satisfactory to good interrater reliabilities for the ratings of the two raters ranging from .58 to .86. Thus, we were could reliably assess psychotherapeutic competencies in the master’s students.

Table 1 Means of global CTS score, the two subscales and single items – scored from 0 (poor) to 6 (excellent) and their interrater reliabilities based on 104 videotaped sessions, each rated twice

	Interrater reliabilities
	M (SD)	ICC(3,2)	p-value
Total score of the CTS	2.92 (0.64)	.89	< .001**
Subscale 1: Session-structuring competencies	2.66 (0.69)	.90	< .001**
Subscale 2: Global psychotherapeutic competencies	3.23 (0.77)	.85	< .001**
1. Agenda setting	0.98 (1.19)	.86	< .001**
2. Dealing with problems / questions / objections	3.15 (1.35)	.73	< .001**
3. Clarity of communication	4.79 (0.92)	.72	< .001**
4. Pacing and efficient use of time	2.71 (1.05)	.72	< .001**
5. Interpersonal effectiveness	3.87 (0.93)	.58	< .001**
6. Resource activation	1.25 (1.00)	.77	< .001**
7. Using feedback and summaries	3.09 (1.34)	.64	< .001**
8. Guided discovery	1.49 (1.04)	.72	< .001**
9. Focus on central cognitions and behavior	3.76 (1.09)	.75	< .001**
10. Rationale	2.85 (1.45)	.76	< .001**
11. Appropriate implementation of techniques	4.17 (1.00)	.68	< .001**

Note. Each item can vary between 0 and 6. Scores on scales are averages of their items.

Table 1 Means of global CTS score, the two subscales and single items – scored from 0 (poor) to 6 (excellent) and their interrater reliabilities based on 104 videotaped sessions, each rated twice

View as image HTML

Students’ Competency Scores

On a descriptive level, the students’ average competency lay between a moderate to a satisfactory level of psychotherapeutic competencies at an item level (see Table 1); the two subscales and the total score were satisfactory. However, some items revealed poor performance on average, since some items may have addressed aspects to be expected in typical therapy sessions but not in the short interview task we gave our students. Importantly, the students’ scores ranged broadly between 1.68 – 4.55 on the total score. The session-structuring competencies ranged between 1.25 – 4.75, and the global psychotherapeutic competencies ranged between 1.60 – 5.10. Apparently, there were no floor or ceiling effects. Figure 2 depicts the histogram of total scores.

In the ANOVA with the factors Scale and Cohort there was a significant main effect of Scale, F‍(1, 101) = 69.45, p < .001, ηp² = 0.41 , but no significant effects of Cohort, F‍(1, 101) = 1.65, p = .198, ηp² = 0.03, and no interaction between both factors, F‍(1, 101) = 2.35, p = .101, ηp² = 0.04. The posthoc paired-sample t-test indicated that performance was better in global psychotherapeutic than in session-structuring competencies in all cohorts, all t-values ≥ 3.46, all p-values ≤ .002, all d-values ≥ 1.20. The results of the ANOVA are illustrated in Figure 3.

Discussion

Students in psychotherapy training must be instructed appropriately to acquire practical skills, and their competencies must be evaluated as part of any teaching program. This is particularly relevant regarding the implementation of the new psychotherapy training in Germany, which now requires evaluations of interviews with SPs as part of the official licensing procedure.

In agreement with the well-established scientist-practitioner model (Benjamin & Baker, 2000), the university-level curricula of clinical psychology and psychotherapy have started to focus more on combining theoretical and practical aspects (i. e., according to the scientist-practitioner model, see Alpers & Hofmann, 2007). To accomplish this, the field has started to experiment with new didactic methods to simulate realistic patient encounters. While teaching with SPs has been well established in medical schools, only recently was this method included in the curricula of psychology programs (Alpers & Steiger-White, 2020).

Importantly, German legislation now requires that SPs be part of the licensing exams (Bundesministerium für Gesundheit, 2020). However, to date, it remains unclear whether psychotherapeutic competencies can be evaluated in MSc-level university students. We recently implemented the method of SPs in our curriculum of clinical psychology and psychotherapy and evaluated 104 videotaped sessions of students and SPs simulating short anamnestic interviews with an SP.

This pilot study demonstrates that short interviews with SPs may be useful for evaluating students’ therapeutic competencies. First, we identified a scale that appears to be useful for rating purposes (CTS; Weck et al., 2010) and adapted it for our purposes. Its time- and cost-effective scoring makes it possible to evaluate many observations as part of a standardized exam in routine licensing procedures.

Second, the interrater agreements we observed in numerous observations show that measurements can be obtained reliably. Only this can guarantee fair and replicable grading as part of a licensure procedure. The rating procedure is feasible, and evaluators can be trained efficiently.

Third, our data provide preliminary evidence for the validity of such evaluations. There is convincing face validity that the scales developed to evaluate professional psychotherapists can be transferred and adapted to evaluate students’ performance. Moreover, a wide range of differences in students’ performance can be mapped by the evaluators’ ratings. Most importantly, our observation that students’ proficiency appears to plausibly differ from performance observed in much more experienced psychotherapists speaks to the validity of this assessment. At the same time, validity is limited because not all aspects to be expected in typical therapy sessions were part of the short interview task we gave our students. Obviously, the match between task and evaluation can be improved by more rigorous item selection.

Taken together, this information provides the essential first evidence that simulations of interviews with SPs may be applicable in student evaluation. This is particularly timely and relevant because such evaluations are now mandatory nationwide in Germany as part of the revised licensing procedures. Aside from these practical implications, this pilot study also provided us with several noteworthy observations.

Interestingly, our students appeared to fair better on global psychotherapeutic competencies than in arguably more basic session-structuring. This may be because the primary topic of the course focused on such global psychotherapeutic competencies (e. g., dealing with problems and interpersonal effectiveness) and less on session-structuring competencies (e. g., agenda-setting). Although we did not examine this as a formal a priori hypothesis, it suggests that our students performed better on those tasks that were explicitly part of the course, which may further indicate that the evaluation can capture relevant aspects of our instruction. Of course, future research should explicitly examine the sensitivity to improvements in performance, ideally in a pre-post design. Apart from this necessity, we found evidence for such specificity in our previous evaluation of self-reported therapeutic self-efficacy (Alpers & Steiger-White, 2020). In that study, students reported more psychotherapeutic self-efficacy for the specific skills they were actually taught. Importantly, the present data substantiate this finding with a more objective assessment of psychotherapeutic competencies by external raters.

Standardized ratings appear to be able to capture competencies, which were previously identified as important for psychotherapists (Vogel & Alpers, 2009). Future work building on these findings may identify subscales on which students’ competencies are low, so that those areas may be targeted by specific instruction or practice sessions of the curriculum.

While such evaluations are not necessarily bound to the use of a specific scale we chose, we would like to discuss several observations regarding the scale’s properties. It is apparently advantageous that the CTS provides a grade-related evaluation scheme. In our sample, we found a mediocre to satisfactory level of psychotherapeutic competencies in students who we obviously did not expect to perform as well as much more experienced practitioners. This finding may speak to the validity of the CTS to capture and evaluate psychotherapeutic competencies at all levels of proficiency. As an extension to this research, future work should also examine the scale’s sensitivity to change. Could they possibly capture students’ progress from before to after a course?

Of course, there are several limitations to this pilot study that need to be considered. First, although both of our raters had a substantial level of clinical experience, they were not yet licensed themselves, a prerequisite for actual evaluations as they are legally defined as part of the licensing procedures. However, the agreement between the two raters was substantial and compares well with calibrated raters in other studies using the same scale (see Grikscheit et al., 2015; Weck et al., 2014, 2016). For licensing exams, which are legally binding, the scales first need to be evaluated with a larger number of much more extensively trained raters (see Strahl et al., 2019). Moreover, in a formal licensing procedure this would require continuous training and evaluation (see Strahl et al., 2018).

Second, for our purpose, we had to adapt the global scale of the CTS. However, we did take this into account in our comparisons between students and practitioners. Importantly, adapted versions of the CTS appear to be still functional. It is important to select those items (or to add to them according to the specific competencies to be examined according to one’s model; see Linden et al. 2007; Vogel & Alpers, 2009). Also, the videos we rated were certainly not completely comparable to those used in the typical scenarios where the CTS is used. The students’ interviews were not complete therapy sessions; they were obviously shorter and did not comprise all of the typical elements.

Third, in this pilot study, we did not completely control how the SPs acted out the difficult interpersonal challenge we instructed them to portray. Obviously, this resulted in more heterogeneous simulations compared to what would be required for formal licensing exams.

Fourth, the pairing of students and assigned SPs was not completely independent (38 SPs acted in simulated situations with 104 master’s students). This might have resulted in a systematic bias for the evaluation of psychotherapeutic competencies. We did not control for subtle differences in difficulty, in pairings of sex, age, and other individual characteristics, most importantly the acting style of different SPs. Also, we were unable to examine order effects, which may be influenced by observing other models before a student had to perform themselves. While it might be possible to better standardize such boundary conditions at any one site, there will always be limits to the standardization across more than one assessment site.

Regarding the efficiency of such ratings, we acknowledge that there was a considerable cost and effort. Although the videotaped sessions lasted only 20 minutes, the evaluation of the psychotherapeutic competencies of 104 master’s students required considerable time resources. If longer exams were to be evaluated, these costs would increase linearly.

Regarding the generalization of our observations, it is important to note that conceptualizations of what constitutes “proficient” therapeutic behavior need to be considered in any evaluation. Only a limited subsection of therapeutic competencies was evaluated in this study, although there are, of course, many more aspects of professional expertise. Obviously, interviewing skills are important, albeit only one of many technical skills (Vogel & Alpers, 2009). Moreover, we did not study students’ performance concerning equally important disorder-specific competencies (Linden et al., 2007), although adherence to specific manuals or a therapeutic rationale has repeatedly been proven to be relevant to outcome (e. g., Hauke et al., 2013; Weck et al., 2016). Another tradition emphasizes the importance of the therapeutic relationship (Luong et al., 2020). It remains to be evaluated whether students may vary in performance on profiles of different therapeutic skills and whether their performance results from an interaction of practical competencies with theoretical knowledge and interpersonal skills (Vogel & Alpers, 2009).

Although our teaching program was not specific to a particular therapeutic tradition, the instructors and raters were trained in CBT. Moreover, the scale we used was based on a CBT rationale and has not been extensively evaluated with therapists from other traditions. Future research is needed to explore whether similar scales can evaluate therapeutic performance rooted in other methods as well. While some basic techniques and concepts of therapeutic change clearly overlap (O’Donohue et al., 2000; Orlinsky & Howard, 1987), other aspects of therapeutic performance differ profoundly, and the specific skills to carry out basic techniques may require more specific attention (Linden et al. 2007).

Conclusion

Our observations provide a promising perspective for the evaluation of students’ performance in professional interactions with SPs. Established instruments to evaluate psychotherapeutic performance can be adapted and applied efficiently and reliably for valid evaluations of student competencies.