The Multidimensional Forced-Choice Format as an Alternative for Rating Scales

Rating Scales

Most available questionnaires collect item responses by asking test takers to describe themselves on RS such as strongly disagree – disagree – neutral – agree – strongly agree. Importantly, each item is rated individually and – at least in theory – independently from the other items measuring the same trait. RS overall allow reliable and valid assessments, especially with homogeneous samples. Nevertheless, there are numerous criticisms of them, for example, that they can be faked easily and that participants differ in how they interpret and use the response categories (e.g., Hernández et al., 2004; Holden & Book, 2011; Krosnick, 1999; Wetzel et al., 2016). Test construction for RS instruments is fairly straightforward with guidelines being described for example in Simms (2008).

Multidimensional Forced-Choice Format

In the forced-choice format, two or more items are presented to participants simultaneously in a block. In the case of the multidimensional version of the forced-choice format, which we focus on here, these items measure different traits. Responses are given directly with the items and different instructions exist. For example, test takers could be asked to select the item that is most like them and least like them or they could be asked to rank the items according to how well the items describe them. Figure 1 shows different variants of the MFC format (pairs, triplets, quads) with different instructions. For an excellent and more thorough introduction to the MFC format see Brown and Maydeu-Olivares (2018a). The MFC format eliminates any response biases that are specific to RS such as extreme response style or acquiescence. It also eliminates rater biases such as halo effects and severity/leniency, though other response biases (faking, careless responding) can still occur. Test construction with the MFC format is more complex than with RS because it requires a number of additional considerations. In particular, the decision of how to combine items to blocks is important because it has implications for the recovery of absolute trait levels and for respondents’ ability to fake. To reduce faking, items within blocks can be matched by desirability. In the MFC format, more items or larger item blocks (e.g., quads instead of triplets) are needed to achieve similar levels of reliability as with RS. For test construction guidelines on MFC formats see Brown and Maydeu-Olivares (2018a).

Data Analysis of MFC Data

The process of responding to MFC item blocks is overall similar to that of responding to RS items, but additionally requires participants to weigh the items within a block against each other (Sass et al., 2020). Thus, in the MFC format, respondents provide comparative judgments, whereas with RS they provide absolute judgments. This has important implications for data analysis. Using classical scoring methods that simply assign scores to ratings is appropriate to obtain respondents’ absolute trait levels (i.e., normative data) with RS. In contrast, with MFC data, classical scoring leads to ipsative data, which is only appropriate for intraindividual comparisons and not for interindividual comparisons or any analyses based on correlations (Hicks, 1970). Thus, classically scored MFC data cannot be used for common assessment contexts such as selection decisions or analyses used commonly in research such as factor analysis, correlations, or the computation of reliability indices. Ipsativity/normativity can be seen as a continuum and one way to reduce the degree of ipsativity is to include negatively-keyed items, resulting in partially ipsative (but still not normative) data. Absolute trait levels can be derived from MFC data by applying an item response model such as the Thurstonian item response model (Brown & Maydeu-Olivares, 2011). However, when the testing purpose is to obtain an individual’s profile (e.g., of their levels of interest in different areas), classical scoring is also appropriate with MFC data. For more information on scoring MFC data and the properties of ipsative or partially ipsative data, see Brown and Maydeu-Olivares (2018a) and Hicks (1970). For an overview over different item response theory approaches to analyzing MFC data and their integration into a common framework, see Brown (2016).

Research Comparing RS and MFC

Research comparing RS and MFC has focused mainly on two issues: validity and faking. Recent studies applying normative scoring for both formats mostly found similar criterion-related validities for RS and MFC (Lee et al., 2018; Wetzel & Frick, 2020; Zhang et al., 2019), whereas older studies applying ipsative or partially ipsative scoring for MFC often found higher criterion-related validities for MFC (Bartram, 2007; Salgado & Táuriz, 2014). Besides the differences in scoring methods, possible reasons for the inconsistent findings include that older studies used more objective criteria (e.g., job performance), whereas some newer studies used “criteria” assessed with RS. With respect to construct validity, several studies applying normative scoring for MFC found descriptively similar levels of construct validity between both formats (Brown & Maydeu-Olivares, 2013; Lee et al., 2018; Zhang et al., 2019). One study that tested preregistered hypotheses and included comparisons within formats (i.e., MFC with MFC and RS with RS) found inconclusive results (Wetzel & Frick, 2020). On the one hand, some construct validity coefficients were larger for MFC and others for RS. On the other hand, the correspondence between self-ratings and other-ratings was consistently better for MFC. Thus, the evidence on the validity of MFC versus RS is still inconclusive. Nevertheless, in absolute terms, trait estimates from MFC questionnaires can achieve good construct and criterion-related validity.

The second issue that has stimulated a lot of research is whether MFC questionnaires are fake-proof or at least less susceptible to faking than RS questionnaires. In fact, the idea that MFC would be harder to fake was the main reason it was advocated in the first place. So far, most existing research found that less faking occurs with MFC than RS (e.g., Christiansen et al., 2005; Jackson et al., 2000). Cao and Drasgow (2019) recently meta-analyzed previous research on faking Big Five MFC measures and found an overall effect size of d = 0.06 (range from 0.00 for neuroticism and openness to 0.23 for conscientiousness), which is substantially lower than the effect sizes for faking Big Five RS measures (range 0.11–0.45; Birkeland et al., 2006). Cao and Drasgow (2019) also checked scoring method (ipsative, partially ipsative, normative) as a moderator and found significant faking effects only with ipsative and partially ipsative scoring, but not normative scoring of MFC data. Wetzel et al. (2020) confirmed previous findings with a Thurstonian item response model analysis of MFC data and additionally showed that the degree to which MFC can be faked depends on how well the items within blocks are matched in terms of their desirability.

A few other issues in the comparison of MFC versus RS have been the topic of recent investigation. For example, Sass et al. (2020) did not find any differences in respondents’ test motivation between groups filling out the same items with RS or different versions of the MFC format. Lee et al. (2019) found that the MFC version of a Big Five questionnaire produced more stable personality profile solutions than the RS version.

Preliminary Conclusions on the Feasibility of MFC as an Alternative to RS

When deciding whether to apply MFC or RS, researchers are faced with a cost-benefit trade-off. Test construction is easier with RS than MFC and test scores from RS instruments are generally more reliable than test scores from MFC instruments of the same length (see above). RS items are evaluated independently, whereas the items in MFC blocks interact and this can influence the response process as well as the items’ psychometric properties (Lin & Brown, 2017). To derive absolute trait levels, MFC data have to be analyzed with appropriate item response theory models, whereas for RS data, this is possible even with scoring according to classical test theory. MFC eliminates response styles in self-report questionnaires (e.g., acquiescence) and rater biases (e.g., severity/leniency), whereas these biases occur when questionnaires are administered with RS. MFC is harder to fake than RS, especially when items within blocks have been matched carefully with respect to their desirability (Wetzel et al., 2020).

Which of these aspects is most relevant will depend on the assessment context and the specific goals of the assessment. For example, if the testing purpose is selection, faking will be the most important concern and the ability of MFC to reduce faking will outweigh the costs. If the assessment is a cross-cultural assessment, in which respondents from different countries are expected to differ in their RS use, MFC might also be preferred. On the other hand, if there are severe constraints on testing time, RS might be preferred to achieve adequate reliability. Or, if there are few resources for the development of the instrument, RS might also be preferred.

Open Research Questions on MFC as an Alternative to RS

There are a number of open questions that we need answers to in order to be able to make clearer recommendations on the feasibility of using the MFC format. For example, previous research compared correlations between traits assessed with MFC and criteria or other constructs with those obtained from RS. Response biases such as response styles potentially influenced both previous empirical findings and comparisons with the MFC format. We do not know what the true correlations are and using correlations from RS as a benchmark against which we compare correlations from MFC may be a mistake. Thus, we need stronger study designs that include within-format comparisons for MFC and criteria that are not assessed with RS. In addition, we need study designs that rely on more than just self-reports to reduce the influence of response styles. Research so far has not considered whether there are any response biases that might be unique to the MFC format. Are there any and if yes, what is their influence? Previous research on validity in many cases analyzed homogeneous samples (e.g., students from one country). Therefore, it is unclear whether findings generalize to more heterogeneous samples in which response biases might exert a stronger influence. This is particularly important with respect to samples with lower cognitive abilities or education level who might be overburdened with forced-choice tasks, especially with item blocks larger than triplets. In addition, the generalizability of research findings to other constructs besides personality traits could also be investigated in future research.

Another open research question is how to match items within blocks with respect to their desirability. Previous research mainly used item means from an administration with RS or desirability ratings of the individual items. Both methods may be inadequate because items within MFC blocks can still differ in their relative desirability. Related to this, another open research question is whether it is easier to fake MFC blocks that contain mixed positively and negatively-keyed items compared with MFC blocks that contain only equally-keyed items when all items have been matched for desirability. Previous research on faking in MFC versus RS mostly used instructed faking designs. Thus, studies in naturalistic settings (e.g., with applicant samples) are needed.

A few approaches to combining MFC with RS have been developed. Xiao et al. (2017) suggested simply administering several RS items in addition to MFC blocks. This approach improved model convergence and the recovery of latent trait levels in their simulation study. Brown and Maydeu-Olivares (2018b) proposed the graded preference format, in which respondents are faced with a choice between two items, but have to indicate the strength of their preference for one of the items on an RS. Future research could investigate whether these proposals are able to capitalize on the strengths of both formats without being limited by their difficulties.

Implications for Readers and Authors of EJPA

Research on MFC is vibrant and productive and we hope that some of the open questions noted above will be addressed in the next years – maybe by EJPA authors. We have tried to give readers a balanced synopsis of the current state of the research on MFC as an alternative to RS and to show that more research in this area – an area at the core of EJPA – is needed. It has hopefully become clear that both formats have their strengths and challenges. For test constructors, this implies that thinking about which format to use should be an explicit step during test construction and the costs and benefits of the different formats should be weighed against each other (Wetzel & Greiff, 2018). For test users, this implies that the costs and benefits of the different formats should be taken into account during test selection. Keeping in mind that the instrument should be tailored to the testing purpose (Ziegler, 2014), sometimes RS instruments may be more appropriate while sometimes MFC instruments may be the best choice. We hope to see research submitted to EJPA that targets some of the open questions mentioned above either from a more assessment-related methodological perspective or through direct applications and comparisons of the different formats for the assessment of specific traits and the measures associated with them. Ultimately, this will likely lead to a broader portfolio of methods as well as measures available to the community of psychological assessment.