The Emic-Etic Divide in Test Development and Adaptation

Introduction

Over the last decades, our world has become a place of international exchange that does not know cultural borders. To better understand how culture¹ is associated with human thought and behavior, the field of cross-cultural psychology investigates commonalities and differences across different cultures and how they influence how we act and think. Even after the rapid development of cross-cultural psychology and its seminal findings over the last decades, as of now, little impact can be observed in the field of psychological assessment. That is, the habitual approach to test development has still little awareness of cultural issues, even though the role of culture on behavior and thought has been readily acknowledged throughout the history of cross-cultural psychology (Cheung & Fetvadjiev, 2016). This editorial will spotlight this issue and the potential oversight associated with it.

Modern psychological testing developed in the 19th century in North America and Europe and continues to be tributary to the basic principles adopted then: tests are traditionally developed in a culture, and are tributary to the mores and typicalities of that specific culture; this is in part due to an understandable focus of test authors (whose tests oftentimes need to solve a specific problem in a specific culture), and in part due to the need to increase ecological validity (which necessarily emphasizes cultural aspects typical for the target culture). However, tests are also traditionally used outside the culture in which they were initially developed and in an increasingly globalized world even more so. Issues of cultural comparability come into play in these cases, and have been widely discussed in cross-cultural psychology (van de Vijver & Leung, 1997), and especially in the assessment literature (van de Vijver & Tanzer, 2004).

Two primary approaches to cultural aspects and their consideration in psychological assessment converge and sometimes battle in an alleged etic–emic dilemma (Church, 2009). The etic approach emphasizes universality: the target constructs are supposedly universal and, because of this assumption, can be assessed using the same measures across multiple cultures. The emic approach embraces indigenous constructs that are supposed to be the hallmark of a specific culture, with limited reach beyond this specific culture. They are, thus, inaccessible by means of, and hidden from universal assessment instruments.

This editorial focuses on the emic-etic divide in test development and adaptation and encourages authors to make extra efforts in order to ensure that the tests they develop have a significant probability of being cross-culturally valid outside the direct scope of the development process, and of the data that they use in order to offer proof of validity. This extra effort can either be addressed in separate and dedicated studies or integrated into planned studies through straightforward extensions. At the very least, test authors are encouraged to explicitly address (a) the ways in which, and conditions under which, their tests may misperform in cultures other than the source culture, and (b) the issues towards which cultural adaptations of their tests may especially need to be sensitive to.

The Etic Bias in Test Development

The vast majority of psychological assessments available nowadays have been developed in an etic tradition, and are used in other cultures through an “import and test” approach (Berry, 1969). Usually, they were developed in Western, English-speaking countries, and are then applied to new populations, with an expectation that, given the correct procedures for adaptation, they will be equally valid for the new context (Cheung & Fetvadjiev, 2016). This process of test import usually implies (or should imply) several steps, such as translation (of some or all the components, i.e., instructions, items, scoring rubrics, etc.), cultural adaptation (of some others, oftentimes seen as less important components, such as assessment forms, delivery media, or training materials), and development of some sort of evidence of equivalence between the source and target forms of the test (usually in the form of statistical, post-facto analyses) (Greiff & Iliescu, 2017).

Even if psychometric practice continues to lag significantly behind psychometric theory in these aspects (Elosua & Iliescu, 2012), we nevertheless need to acknowledge that the past three decades have brought with them significant understandings and a consensus of the need for sophisticated approaches to the translation/adaptation of psychological tests. We now have comprehensive guidelines (e.g., International Test Commission, 2005, 2017) and whole monographs (e.g., Iliescu, 2017) dedicated to the intricacies of test adaptation – and through this, we have moved significantly beyond both the disregard for culture-in-assessment and the mostly naïve adaptations typical for research and practice 50 or more years ago. A steep evolution in the habitual approach to test adaptation is also visible in the many sophisticated approaches to psychometrically establish equivalence, by the many studies that look at this issue (many of them published in EJPA), and the fundamental role that this issue now plays in test development, for instance, in international large-scale assessments.

At the same time, the domain is still largely etic in its fundamental flavor: tests continue to rarely be developed having in mind the possibility that they may be used at some point in time in a different culture. Elements of “culture-universal design” should be more visible during the test development process (and are largely lacking), delineating the author’s awareness of the potential for cross-cultural bias in the constructs, methods, and items proposed. Such awareness could be shown by explicit discussions of these elements as part of the evidence offered for test validity, as well as by their presence in the test documentation.

We argue for greater awareness of researchers toward emic constructs, for both theoretical and practical reasons. We know that a rigorous approach to test adaptation is not a panaceum. No matter how thorough a specific adaptation process is, some questions will always remain lingering even over the best-adapted scales: How relevant is the investigated construct to the target population? Are indigenous constructs important to the target population but inadequately addressed or omitted by the test? Had the test been developed directly in the target culture, would it look differently? (Cheung et al., 2011).

Here, we do not argue for a radical emic approach. A radical emic approach focuses only on indigenous constructs that are supposed to be characteristic of, and unique to, a specific culture and denies even the existence of universals in psychology, while imagining “cultural psychologies” – i.e., Western psychology, Eastern psychology, individualistic or collectivistic psychology, etc. (Church, 2009). An emic approach however also promotes a sustained effort toward understanding and explaining the cultural components in psychological constructs – and we argue that test authors should be aware of this cultural variance in their constructs and of the manner in which it may reflect in their tests, if these were to be adapted and used in those other cultures.

A large number of arguments have been brought for emic approaches, both empirical and theoretical in nature. Empirical arguments have usually focused on culture-bound constructs or culture-bound manifestations of constructs. For example, components of culture-specific practical intelligence that were commonly ignored by classical cognitive abilities assessments have been shown to be important indicators of intelligence in some cultures (Sternberg et al., 2001). In personality assessment, concepts with unique labels and unique variance have been identified in many cultures (e.g., Ren Qing in China, Amae in Japan, Chong in Korea, Ubuntu in South Africa etc.) (Valchev et al., 2011), a line of research that has offered us such emic measures as the South-African Personality Inventory (SAPI; Nel et al., 2012) and combined emic-etic assessments as the Chinese Personality Assessment Inventory (CPAI; Cheung et al., 1996). For example, the CPAI (in its second edition) uses a combined etic-emic approach and measures personality traits that are considered universal (e.g., Extraversion, Emotionality, Responsibility), but also measures other personality traits that are specific to the Chinese culture and that are both documented by other studies in China, and emerge from factor analytic studies (e.g., Ren Qing, Ah-Q Mentality, or Veraciousness vs. Slickness), resulting in a four-factor structure that only partially mirrors the (arguably universal) Five-Factor Model (Cheung et al., 2003).

More sophisticated theoretical arguments on behalf of the emic approach can be constructed from such influential modern theories as Trait Activation Theory (TAT; Tett & Burnett, 2003). TAT suggests that different cultures may trigger different traits – the theory states that specific situational factors activate specific traits or factors. In the original TAT framework, these factors referred to work characteristics at different levels (task, group, and organization), but recent findings have extended this framework, suggesting a fourth level: culture (Choi et al., 2015).

Conclusion and Call to Action

In the manuscripts that are published in EJPA, authors often try to emphasize the more generalizable conclusions: while studies always refer to a specific test in a specific linguistic and cultural context, elements of the study benefit the whole testing community by being generalizable. One manner in which this could be achieved is through a more “culture-universal design” of tests and studies.

We, therefore, argue that authors should explicitly refer in their papers to how culture may influence elements of the test and that they do so with the etic-emic divide in mind. For example, authors of new tests could discuss what elements of their tests they expect to be more, respectively less, culturally invariant if these tests were to be adapted to other cultures, and to give reasons for these expectations. Conditions under which their tests may misperform in cultures other than the source culture could be outlined, in terms of reasonable expectations for construct, method, or item bias, and maybe ways in which to mitigate these. We have noted above that cultural adaptations usually entail a number of steps or activities, the most often encountered being (a) translation (of some materials/components), (b) cultural adaptation (of some other materials/components), and (c) some evidence of (psychometric) equivalence. Authors could use these three steps or activities as a structure for their appreciation regarding the cultural variance that they have observed, or that is expected in the future, in similar adaptation efforts. For example, they could outline separately what they expect in future studies to be easy or difficult in terms of translation, and what specific components they expect will need to be adapted more profoundly, with what difficulties.

Authors of test adaptations could underscore what specific components of the adapted test may have been influenced more (or less) by culture, and what they recommend that attention should be given to in future test adaptations. The insights that an author gets while actually working on a test adaptation, in terms of what components of the test have a chance to be universal (and invariant) and what components have a probability to be less invariant and more subject to emic influences, are invaluable. But these considerations are often lost and remain as implicit knowledge with the researcher, without being explicitly communicated to the scientific audience. We would, therefore, encourage authors of test developments and adaptations to explicitly refer in their manuscripts to cultural aspects, as they have been actually experienced or can be reasonably inferred from empirical or theoretical reasonings. This could be done in a short section built into the Discussion part of the manuscript, and we encourage this for manuscripts submitted for publication in the European Journal of Psychological Assessment.