Measuring Distinct Social Skills via Multiple Speed Assessments

Assessment of Social Skills via Assessment Centers

In ACs, assessees (i.e., individuals who are evaluated) face a variety of different exercises (e.g., role-plays, group discussions) and are observed and evaluated by different assessors (i.e., individuals who evaluate) on a variety of (often social) skill dimensions (for a recent overview, see Kleinmann & Ingold, 2019). Given the lack of a parsimonious and consensual overarching framework of social skills, the selection of skills is mostly based on idiosyncratic choices that depend on the work or educational context. This results in a myriad of differently labeled social skills in research and practice (Arthur et al., 2003; Klein et al., 2006). Whereas ACs have repeatedly been shown to reliably capture between-person differences that predict important outcomes (e.g., Sackett et al., 2017), the selected skill dimensions seem to play a negligible role. This has most recently been investigated by using generalizability theory (Brennan, 2001), in which the impact of different AC components (e.g., skill dimensions, assessee main effects, assessor effects) and their interactions are estimated by identifying how much they contribute to the variance in assessor ratings. Here, the amount of variance that can be attributed to differences in skill dimension-specific performance is typically low (Jackson et al., 2016; Putka & Hoffman, 2013). This is due to (a) little differentiation between different skills within an exercise (low discriminant validity) and (b) no convergence between the same skills across exercises (low convergent validity; Bowler & Woehr, 2006). That is, high ratings on persuading others within one exercise would typically covary with high ratings on showing compassion within the same exercise. Across exercises, there would be little consistency in persuasiveness or compassion ratings. Reasons for this phenomenon have been discussed extensively (see Lance, 2008 and associated commentaries) and can be broken down into two main arguments: potential assessor biases and potential inconsistencies in assessee behavior.

Assessor biases refer to the idea that assessors are not sufficiently skilled or have too much cognitive load. Along with the fact that assessor and exercise variance are often confounded (i.e., an assessor typically does not evaluate each assessee on each exercise), this might lead to inaccurate and undifferentiated ratings. And of course, training assessors (e.g., frame of reference) and carefully designing ACs (e.g., including behavioral checklists, reducing the number of skill dimensions) will undoubtedly improve the reliability and validity of assessors’ ratings (Lievens, 2009). However, years of research have shown that assessors are actually quite accurate when it comes to observing and evaluating interpersonal behavior (Back & Nestler, 2016; Lievens, 2009). For example, Lievens (2002) showed that when assessees were specifically instructed to act in a cross-situationally consistent manner, assessors picked up on this pattern and rated the assessees accordingly. Thus, a sole focus on assessors cannot explain the inability to reliably assess distinct skills in ACs.

Some authors (e.g., Lance, 2008) have built on this argument (i.e., that assessors’ ratings are not generally flawed) and suggested that it must be assessees’ behavior that is inherently cross-situationally inconsistent. This reasoning is backed by the general finding that performance differences between assessees depend strongly on the situational demands of the respective exercise (e.g., some assessees just perform better in Exercise A, whereas other assessees perform better in Exercise B; see Lance et al., 2004; Lievens, 2009; Lievens et al., 2009). However, AC research has predominantly analyzed assessors’ performance ratings and has not considered actual behavioral variation. That is, just because ratings on the chosen skills are undifferentiated within exercises and inconsistent across exercises does not mean that the same holds for the actually expressed behaviors. For example, even if the behavior is expressed consistently across exercises (e.g., individuals who make more supportive statements in Exercise A also make more supportive statements in Exercise B), the resulting skill ratings (e.g., on a compassion dimension) may be inconsistent. This could be due to the use of exercise-specific rating sheets (i.e., the resulting skill ratings might not actually be based on the same behaviors) or the case if resulting skill ratings are influenced by other aspects of the exercises (e.g., whether the overall exercise performance was good or bad).

In fact, a recent study (Breil, Lievens et al., 2021) showed that basic behavioral differences (e.g., warm behavior, assertive behavior, interpersonal calm behavior) can be differentiated in interpersonal AC exercises and are actually quite stable across AC exercises. This is supported by decades of research from behavioral personality science that has shown moderate to high behavioral consistency in a variety of behavioral domains across many different tasks and situations (e.g., Borkenau et al., 2004; Furr & Funder, 2004; Leikas et al., 2012; Sherman et al., 2010). These results indicate that (a) the unreliable assessment of distinct skills is not solely rooted in inconsistent assessee behavior, and (b) the variety of behaviors that are evoked by different exercises are not necessarily the behaviors that are being evaluated. Thus, it might rather be the typical way in which ACs are designed (i.e., the interplay of behaviors, skill selection, and exercise creation) that limits the interpretability of skill dimensions.

Interplay of Behaviors, Skills, and Exercises

The skills included in ACs are foremost based on job analysis or competency modeling and reflect bundles of behaviors that are deemed important (Lievens, 2017; Rupp et al., 2015). Even though skills are sometimes organized according to theoretical frameworks (e.g., Arthur et al., 2003; Meriac et al., 2014), the typical way of selecting skills is not without its drawbacks as it does not take the empirical structure of behaviors into account (Lance et al., 2004). On the one hand, the skills that are defined for assessment (e.g., compassion, empathy, social sensitivity) might share most of their underlying behaviors (jangle fallacy). On the other hand, one skill (e.g., communication) might be represented by different behaviors depending on the exercise (jingle fallacy). Thus, there might be little differentiation between different skills because the underlying behaviors are often identical or overlap, and there might be no consistency across the same skills because the underlying behaviors differ (across exercises; see Lievens et al., 2009; Melchers et al., 2012).

These effects will likely be further exaggerated due to the typical selection and creation of exercises. Most AC exercises are designed for their face validity (e.g., tasks related to the job) rather than for their capacity to expose relevant behavioral differences (Brannick, 2008; Lievens & Klimoski, 2001). That is, it is unclear how well the variety of different AC exercises that are used can evoke reliable differences in desired behaviors. For example, if one aims to assess persuasiveness within an exercise in which no one must be persuaded (e.g., because the role-player is instructed to act submissive), there will be little reliable behavioral variance related to persuasiveness (see also research on trait relevance and situational strength: Lievens et al., 2006; Meyer et al., 2010). Furthermore, specific aspects of exercises might actively increase the co-dependence of skill ratings. That is, within ACs it is mainly the exercise outcome that is evaluated (Brannick, 2008; Lance et al., 2004) and if there is only one way to “solve” an exercise, assessees are likely to receive high/low ratings on all skills assessed with this exercise. Also, if good performance on an exercise can only be achieved via exercise-specific behaviors (e.g., the interaction partner will only be persuaded by putting forth a content-specific argument), the cross-situational consistency of skill ratings will decrease.

Implications for Assessment Designs

If one aims to assess distinct social skills, the presented findings have several implications for assessment design. That is, skill selection should not only be based on job analysis but also acknowledge empirical findings concerning the structure of the behavioral expression. Accordingly, one should consider how behaviors that are expressed in interpersonal interactions will cluster. Whereas this (bottom-up) approach has received little attention in selection settings, it has been a key focus of behavioral personality science. Here, results suggest that most interpersonal behavior can be represented by the two underlying factors of agency (i.e., dominant and assertive behavior) and communion (i.e., warm and friendly behavior; Dawood et al., 2018; Wiggins, 1979). These two factors are often supplemented by a third factor pertaining to interpersonal calmness (emotionally stable and relaxed behavior; Leising & Bleidorn, 2011), which seems to be especially profound in stressful situations such as assessment settings (Hirschmüller et al., 2015). All three behavioral domains have been shown to emerge across a variety of interpersonal settings and tend to be quite stable across situations (e.g., Borkenau et al., 2004; Leikas et al., 2012). Thus, we suggest (a) not assessing multiple skills per behavioral domain (e.g., compassion and empathy, both of which pertain primarily to communal behavior) and (b) not assessing skills that overlap across the behavioral domains (e.g., broad concepts such as “leadership” will likely include communal and agentic behaviors; thus, for a more distinct assessment of skills, it would be preferable to assess these leadership facets separately).

When it comes to exercises, it is clear that assessees will vary in how they approach each of these specific situations and solve the implied specific tasks. This will undoubtfully impact ratings on all skills assessed with the respective exercise (Lance, 2008; Lance et al., 2004). Thus, it is important to build exercises in such a way that high performance on an exercise can only be achieved by individuals who express skill-related behaviors. For example, if one aims to assess candidates’ compassion with an exercise that involves delivering unpopular news, it is important that the exercise is designed in a way that a positive outcome can only be achieved through compassionate behaviors (e.g., active listening, supportive statements) and not with other behaviors (e.g., using specific arguments, controlling the interaction). However, doing this for multiple skills within an exercise would likely result in the co-occurrences of related behaviors leading to little differentiation between skills. For this reason, we propose that only one skill in each exercise is assessed and that the exercises be built exclusively around the chosen skill (see Brannick, 2008, for similar reasoning). In line with axioms of trait activation theory (Tett & Guterman, 2000), this would involve using multiple skill-specific environmental cues that directly trigger the expression of skill-relevant behaviors (e.g., having someone ask for help will likely trigger differences in communal behavior). For a more reliable assessment of skills, it would then be beneficial to use multiple exercises per skill (i.e., have exercises nested within skills, e.g., three exercises to assess compassion, three additional exercises to assess persuasiveness). This would not necessarily mean having very similar exercises (which might negatively impact validity; Speer et al., 2014). Instead, it allows for a relatively comprehensive assessment of each skill expressed across different contexts that represent the variety of specific environmental cues that trigger individual differences in the skill of interest.

A procedure that is ideal for such an approach involves multiple speed assessments (Brannick, 2008; Herde & Lievens, 2020). In contrast to classic ACs, multiple speed assessments revolve around a larger number of very short interpersonal simulations that elicit overt behavior in a standardized way. The basic idea behind multiple speed assessments is that (a) even brief behavioral observations provide assessors with enough information to make relatively accurate judgments (Breil, Osterholz, et al., 2021; Funder, 2012; Ingold et al., 2018) and (b) by increasing the number of situations/exercises (i.e., sampling behavior across a variety of contexts), the reliability and validity of the overall rating increase (i.e., principle of aggregation; Epstein, 1983).

Present Research

With the current research, we aimed to conduct a first empirical investigation of a more behavior-focused approach that directly addresses the presented implications. Specifically, we tested whether the selection of skills according to the structure of cross-situationally consistent behavioral differences and the creation of exercises around these specific skills would allow for a more reliable assessment of distinct social skills. To do so, we used a multiple speed assessment procedure to identify the best applicants for a spot in medical school across three samples. We hereby chose to assess the three social skills of warmth, assertiveness, and resilience that were found to be desired skills for future physicians (Hertel-Waszak et al., 2017) and that exclusively revolve around behaviors from the three domains of agency (assertiveness), communion (warmth), and interpersonal calmness (resilience). For each of these skills, we designed two exercises that involved multiple cues to specifically evoke skill-related behaviors.

Here, the basic premise was to create exercises in which specific behavioral responses were needed to perform effectively. For example, within the exercises revolving around assertiveness, agentic behavior was required to achieve a positive outcome. Assessees with a high level of assertiveness skill were expected to (a) know that agentic behavior is needed and (b) act accordingly. By observing interindividual differences in expressed agentic behavior (within the relevant exercises), it was then possible to draw conclusions about assessees’ assertiveness skill levels. Such assessed individual differences in social skills refer to maximum performance (i.e., what someone is capable of doing when it matters) and should not be equated with individual differences in personality traits (i.e., what someone tends to do in general, typical performance; Breil et al., 2017; Soto et al., 2021).

To investigate the distinct assessment of the selected social skills, we analyzed the reliability of the skill ratings, the correlations between the skill ratings, and variance components (i.e., identifying the amount of variance that could be attributed to different AC components; e.g., individual differences depending on skill dimensions). For a more reliable assessment of distinct social skills, one would expect that (a) there would be at least acceptable reliability in the individual skill ratings, (b) the correlations between ratings on exercises belonging to the same skill would be higher than the correlations between ratings on exercises belonging to different skills, and (c) there would be a substantial amount of variance in assessor ratings that could be attributed to individual differences in the different social skills.

Participants

Assessors

In each selection sample, up to 60 professional physicians evaluated the assessees. The assessors were allocated to teams of two, assigned to one exercise and one room (six teams/rooms per exercise per sample), and evaluated up to 40 assessees. That is, each assessee was evaluated by two assessors per exercise, and there were different assessors for each exercise and each sample (see Figure 1 for a visual representation).

Procedures

The multiple speed assessment was developed in a manner similar to other approaches (e.g., Herde & Lievens, 2020; Knorr et al., 2018) and consists of multiple interpersonal role-play exercises. All assessors received assessor training (2 hr; see Rupp et al., 2015), which included a lecture (e.g., rater biases, separating observation and evaluation, establishing a frame-of-reference) and practice (e.g., viewing example videos followed by moderated discussion and feedback). Before each exercise, assessees had 90 s to read the instructions and then took part in the exercises, each of which lasted for 5 min. The general sequence of exercises was identical for all assessees, but the starting exercises were randomized. In the exercises, assessees interacted with role-players who all had multiple years of professional acting experience and were specifically instructed concerning their roles and behavior. At the same time, assessees were observed by assessors through a one-way mirror (assessees knew that they were being observed and evaluated). After each exercise, participants switched to the next exercise, and assessors were asked to rate assessees on the corresponding social skill dimension.

Overall, six different interpersonal exercises were used across the three samples. That is, for each of the three social skills (i.e., assertiveness, warmth, resilience), two exercises were developed. Building on the assumptions of trait activation theory (Tett & Guterman, 2000), exercises were specifically designed to be highly relevant for the desired social skills. For this, we relied on research and findings from behavioral personality science (e.g., interpersonal theory; Dawood et al., 2018). For example, for the exercises related to resilience, we included multiple triggers in the exercises (e.g., social pressure, uncertainty) that have been shown to provoke differences in expressed behaviors related to resilience (e.g., Hirschmüller et al., 2015). Furthermore, emphasis was placed on ensuring that the exercises revolved as exclusively as possible around the respective skills and that skill-related behavior would be associated with good performance within each exercise. For an overview of all exercises and behavioral triggers see Figure 2.¹

Measures

Assessors rated assessees on one social skill per exercise (i.e., the skill that the exercise was designed to assess) via one global rating. For the ratings of the social skill dimensions, we relied on global behaviorally anchored rating scales (e.g., Please rate the candidate’s assertiveness) ranging from 0 to 5. Behavioral anchors were based on previous research concerning the selected social skills (e.g., Hirschmüller et al., 2015; Oliver et al., 2016) and were identical across the respective exercises (e.g., assertiveness: shows self-confident posture, confident flow of speech, leads the interaction).² For all social skills, this included a mixture of nonverbal (e.g., warmth: friendly expression) and verbal (e.g., warmth: statements of support) behaviors. The exact behavioral anchors (in a condensed form) are displayed in Figure 2.

Analytic Strategy

To quantify the extents to which the different components contributed to the amount of reliable variance (i.e., individual differences in overall performance, individual differences in skill dimension-specific performance, individual differences in exercise-specific performance), we estimated the variance components on the basis of the assessors’ ratings. Similar to recent studies (e.g., Breil et al., 2020; Jackson et al., 2016; Putka & Hoffman, 2013), we decomposed the variance by specifying the assessees, assessors, skill dimensions, exercises, samples, and all their possible interactions as crossed random factors in a Bayesian linear random-effects model with the R package brms (Bürkner, 2017). Bayesian approaches have the advantage of being applicable to very complex data structures with a large number of estimated variance components (see Jackson et al., 2016). A description and interpretation of the different variance components are summarized in Table 1. Note that (per design) exercises were nested within dimensions, and assessors were nested within exercises. This resulted in fewer variance components in comparison with studies with more classical AC designs (e.g., designs in which exercises and dimensions were crossed; e.g., Jackson et al., 2016). Data and R code can be found at osf.io/jy5wd.

Table 1 Decomposition of observed variance in assessor ratings

Variance component	Meaning
Note. This table was adapted from Putka and Hoffman (2013).
Reliable variance
σ2 assessee	Individual differences in overall performance: Some assessees perform better than others, irrespective of exercises or skill dimensions.
σ2 assessee × skill dimension	Individual differences in skill dimension-specific performance: Some assessees perform better on some skill dimensions than others, irrespective of exercises.
σ2 assessee × exercise	Individual differences in exercise-specific performance: Some assessees perform better on some exercises (nested in skill dimensions) than others.
Unreliable variance
σ2 assessee × assessor	Assessor disagreement: Some assessees are rated higher by some assessors (nested in exercises and skill dimensions) than others.
σ2 assessor	Assessor main effects: Some assessors (nested in exercises and skill dimensions) tend to give more lenient/severe ratings than others.
Other components
σ2 exercise, skill dimension, sample	Exercise, skill dimension, sample main effects: Some exercises, skill dimensions, or samples (and their interactions) have higher average ratings. Here, this does not influence reliability.

Table 1 Decomposition of observed variance in assessor ratings

View as image HTML

Assessment of Distinct Social Skills

Our results showed that implementing the presented changes improved the assessment of distinct social skills. This was evident from a substantial amount of variance reflecting individual differences in performance that depended on the specific skill dimensions. In fact, this crucial variance was considerably higher than in previous studies that followed a more classical AC design (e.g., Breil et al., 2020; Jackson et al., 2016; Putka & Hoffman, 2013; see Figure 4). That is, assessees showed variability not only in their general performance (some assessees were better than others across all exercises) or in their exercise-related performance (some assessees performed better on specific exercises) but also in their skill-related performance (e.g., some assessees performed better on exercises related to assertiveness, whereas other assessees performed better on exercises related to warmth).

The finding that ratings across different social skills were generally positively related (i.e., assessee main effects) was not unexpected, as some assessee characteristics such as cognitive ability (Collins et al., 2003), or being able to assess situational demands (Jansen et al., 2013) are likely related to effective performance irrespective of the specific social skill dimensions. Of course, assessees’ performance ratings were not perfectly consistent even across exercises that belonged to the same skill dimension. This is also not surprising as the exercises were not designed to be completely parallel (and, thus, also more redundant) measures (i.e., they included different tasks, role-players, and affordances). Still, compared with studies on behavioral consistency in personality science (e.g., Leikas et al., 2012; Sherman et al., 2010), the assessor rating consistencies were a bit lower. Thus, the skill ratings might also have been influenced by exercise-specific aspects not directly related to the desired behaviors. For example, the specific arguments put forth in the “mistake” exercise may have influenced the skill ratings in addition to the actually expressed differences in resilient behaviors. However, different performances within exercises belonging to the same broad skills should not generally be treated as meaningless or unreliable as they likely also reflect context-dependent variability in behavioral expressions. That is, some individuals might be more resilient when it comes to time pressure, whereas other individuals might be more resilient concerning criticism. Aggregating across these different aspects of resilience arguably allows for a more comprehensive – and, thus, more valid – skill assessment compared with an approach that maximizes consistency at the expense of restricting the breadth of the skill construct (Asendorpf, 1988; Clifton, 2020; Speer et al., 2014).

Implications

Generally, the findings and suggestions do not imply that the current best practices should be abandoned when it comes to the development of assessment procedures. ACs show predictive validity, and this is likely driven by the variety of job-related exercises that evoke job-related behaviors. However, if the aim is not only to predict job performance but to assess distinguishable (social) skills (e.g., for development purposes or to give feedback on assessees’ strengths and weaknesses), we put forth the “multiple exercises nested in skills speed assessment” as an alternative procedure. This procedure essentially follows mixed-model ACs that acknowledge the importance of exercises and skill dimensions (see Hoffman et al., 2011; Melchers et al., 2012).

In this study, skill selection was based on a bottom-up analysis of observable and distinguishable interpersonal behavior, which serves as an important addition to top-down approaches of reducing assessment dimensions (e.g., Arthur et al., 2003; Meriac et al., 2014). This does not mean that the same skills (i.e., assertiveness, warmth, resilience) should be chosen in all assessment contexts but rather suggests that more emphasis should be placed on how the proposed behavioral structure of desired skills aligns with the empirical and theoretical structure of actually expressed behaviors. This includes identifying the specific situational triggers that evoke desired behavioral differences. Here, one can build on previous research: For example, interpersonal theory (Dawood et al., 2018) has provided many findings on situations in which differences in agentic or communal behaviors should be especially profound.

Exercise effects (i.e., individual differences in performance that depend on the specific context) will always play an important role in assessment contexts. In the presented assessment approach, exercise effects are channeled by assigning multiple exercises to specific skills. This gives meaning to the still prevalent exercise-related performance differences as they can be interpreted as context-dependent facets of the broader social skills. Accordingly, feedback given to assessees will benefit from acknowledging this nested structure (e.g., “Overall, you showed above-average performance. This was especially true for exercises related to assertiveness and warmth. On the exercises related to resilience, you varied in your performance, showing low resilience when it comes to time pressure”).

Concerning the specific type of assessment procedure, multiple speed assessments have been discussed as an ideal way to assess the broad behavioral repertoires of assessees (Herde & Lievens, 2020). Even though relatively few exercises were used in the current implementation, the results indicate that multiple speed assessments can also be used to assess distinguishable behavioral domains. Generally, as the number of exercises assessing each skill increases, exercise-related performance (i.e., context-dependent skill facets) will have less influence on the overall skill scores, resulting in more reliable skill estimations. Thus, there is always a benefit to including more exercises in the assessment procedure, limited only by cost and time constraints. Here, we identified four exercises per skill dimensions as the sweet spot, leaving individual differences in assessees’ skills as the main source of variance in ratings.

Limitations and Future Research

In this study, we focused on the assessment of social skills as expressed in a specific high-stakes selection context. Naturally, to increase generalizability, the present results should be replicated across different selection contexts. Besides this, the present study highlights several important directions for future research.

First, the situational triggers (e.g., role-players acting sad and insecure to evoke differences in communal behavior) were based on theoretical considerations, and we did not directly test whether the triggers we included actually led to an increase in the desired behavioral variance. Thus, future research might specifically change the situational triggers and investigate how this affects behavioral expressions and subsequent ratings. For example, one could directly rate relevant behavioral expressions in exercises with or without the specific triggers (e.g., rate communal behavior in the exercise revolving around warmth but also in the exercises revolving around assertiveness). Furthermore, the perception and behavioral reaction to situational triggers could be further disentangled. That is, some assessees might not even perceive the situation as demanding a specific behavioral response (see Jansen et al., 2013), whereas other assessees might theoretically know how to behave, but do not manage to react as planned. Such an investigation of how different triggers work and affect behavioral expression will (a) advance the theoretical understanding of behavioral functioning and (b) benefit an effective process of exercise-creation.

Second, based on previous research concerning multiple speed assessments (e.g., Herde & Lievens, 2020; Oliver et al., 2016), the skills we assessed were relatively broad and rated via one global item. Here, it would certainly be possible to further zoom in on the specific skills and more directly assess different aspects (e.g., by differentiating between assertive verbal content, assertive nonverbal behavior, and assertive paraverbal behavior), which might help to explain different behavioral expressions between exercises. Furthermore, one can zoom in on the behaviors from a time perspective. That is, a continuous assessment of behavior via joystick approaches over the course of an exercise (i.e., CAID method; Sadler et al., 2009) would enable a more fine-grained perspective on the underlying behavioral processes. This would not only allow for an analysis of (within situation) behavioral variance and its relationship with performance measures but also help in identifying relevant situational triggers.

Third, we provided an in-depth analysis of the reliability and dimensionality of the different skill ratings but did not investigate predictive validity. Whereas a lot of research has suggested that AC performance ratings predict future performance, this will likely vary depending on the exercises and skills that are included (Speer et al., 2014). Thus, it would be beneficial to investigate the predictive validity of the presented procedure and focus on differences between skills. That is, one might expect different relations depending on the type of skill dimension (e.g., warmth might be more strongly related to short-term customer satisfaction, whereas assertiveness might be related to long-term success; see Wirz et al., 2020).