Some Thoughts and Considerations on Accommodations in Testing

This editorial focuses on the concept of accommodation in testing, that is, individualized changes – however small they might be – to one or several components of the test in response to a need to preserve fairness and/or validity in special populations or individualized cases.

Accommodation is defined as “a change in a test format or content, or some aspect of test administration, which makes the test accessible to individuals who might otherwise be unable to complete the measure but does not alter the construct being measured” (American Psychological Association, 2012, p. 53). In an ideal case, accommodations are made in the presence of test-taker disabilities that would disadvantage the performance of these test-takers while not being relevant for the focal construct of the test (Linn, 2002)

A large palette of accommodations is possible. Some may be related to the administration context (e.g., lighting or access to the test location), some to the presentation of the test and its stimuli (e.g., a larger print or through usage of assistive technologies), some to the format of responses, some to the mechanics of the test administration process (e.g., slower pacing, extended time limits), etc. Given the individual nature of accommodations, there are many more examples that one could think of.

Accommodations are critical in assessing persons with disabilities, being a moral and professional, and in many countries, also a legal obligation. The most encompassing international approach to disability is the United Nations Convention on the Rights of Persons with Disabilities (UN-CRPD; UN, 2006). Disability, defined as the outcome of the interaction between a personal condition and a context (Schneidert et al., 2003), can easily also be the outcome of the testing situation, that is, of the interaction between the physical or psychological condition of a person and the requirements of the test. Thus, accommodations to meet the need of test takers with a disability is a core aspect of the ethical and moral use of test administration. The specific form this might take will well depend on the type of definition assumed, and here we suggest adopting a broad definition of disability as outlined above.

Not only from a moral but also from a psychometric point of view, accommodations are critical because they are one of the many forms to enhance the validity of an assessment: they provide an acceptable way in which to measure an individual focal characteristic (an ability or a trait) in less-than-ideal situations, that is, when there is a poor match between the (standardized/prescribed) test conditions and the capacities of test takers to exhibit their standing on that characteristic. At the same time, they are a disputed subject because they go against one of the central mantras of the field: standardization of test-taking conditions for all test takers.

Accommodations target of some of the elements of standardization in a test: the standardized administration procedure, presentation of stimuli, response formats, or mechanics of the test administration process are replaced with ad-hoc and individualized procedures. Can this be done in good faith? The pervasive wisdom is that it can and, in fact, should be done when the highly standardized context that functions for the majority (i.e., any of the elements mentioned above) generates disadvantage based on construct-irrelevant variance for people with disability, thus actually lowering the validity of the assessment for a person or a group.

Studies generally differentiate between the target characteristic (skill, ability, or trait) and access characteristics. The target characteristic is the construct that needs to be measured, while access characteristics are constructs that are not focal to the measurement but may be needed or a prerequisite to taking the test – for example, reading the items or signaling the answers. For example, students with muscular dystrophy experience physical fatigue; sitting for longer tests where they are required to write extensive essays will hinder their opportunity to show their standing on, say, language ability or literature knowledge (which are the target constructs). Accommodations in the form of extended testing time or allowing for another form (e.g., speak-aloud) of recording their answers (i.e., changes that would override their limitations in the access skill) may allow them to show their true standing on the target construct.

At the same time, while this wisdom is embraced by important documents in our field (e.g., the Joint Standards; AERA, APA, NCME, 2014), decisions regarding accommodations (i.e., when accommodations are appropriate and when not, and specifically what and how to accommodate) are vague, and there is little practical guidance or empirical research on this issue. There are reasons for this scarcity of information. Two of the most important ones are (a) the fact that accommodations have long been treated as singular and highly individualized decisions, subject to professional reasoning and case-individualized decision-making, and (b) the fact that the range of disabilities and their intensities is so large that coherent empirical research on when and how to accommodate is likely very difficult to conduct.

The result is a state of scarcity in the literature that is not conducive to good practices. As stated by Geisinger and McCormick (2016):

Current research on this topic stems mainly from the direction of educational testing, where interest has been more vivid than in psychological assessment, with both institutions (e.g., the National Center on Educational Outcomes, NCEO; Cormier et al., 2010; Zenisky & Sireci, 2007) and researchers (e.g., Sireci et al., 2005) taking an interest in the domain of accommodations for high-stake educational testing; studies from these areas already contain both systematic reviews (e.g., Cormier et al., 2010; Thompson et al., 2002) and meta-analyses (e.g., Vanchu-Orosco, 2012). The technical reports published by the National Center on Educational Outcomes systematically review research published in this area every few years; they are an important source of systematic evidence on the types of accommodations, samples, and research designs used to understand the effect of these accommodations. Among others, they uncover a good extant base on which to build consistent studies in this area that also might have a more psychological perspective but also show the diversity of results, with some accommodations for some tests and for some forms of disability showing no effects on scores, whereas others showed positive effects on scores or various forms of bias (e.g., item bias) under accommodated conditions. We suggest that developers of tests may therefore want to drive research on their specific tests in such a way as to study the effects of different types of accommodations for specific forms of disability and suggest those best suited to allow access while not distorting the scores of the targeting ability in a specific test. In our example above, this would mean that the test authors would have already undertaken a study on the influences that an extended testing time and a speak-aloud recording of student answers may have on the scores of the test and use the results of this study to suggest reasonable accommodations for people with muscular dystrophy or may have at least offered convincing (and probably empirical) proof for the fact that other research that has been carried out on these topics is generalizable to their specific case.

Two issues are likely important to be mentioned in this context. First, such research is sometimes undertaken by the developers of educational tests, especially those that have very high usage and that contribute to high-stakes decisions, such as the National Assessment of Educational Progress (NAEP) (e.g., Weston, 2003) or the SAT (e.g., Lindstrom & Gregg, 2007), but very rarely for psychological tests, even though these are also oftentimes used for high-stakes decisions – and even though some of these are sometimes used specifically with audiences that have a condition that would mandate accommodations. Second, research on accommodations would need to be characterized by some level of generalizability that would make the conclusions regarding the consequences of a specific type of accommodation, or in a specific disability context, transportable to other similar contexts.

In light of the importance of this issue and the relative scarcity of systematic research in psychological testing, we do encourage studies in the following directions:

1. (1)
   We encourage research on the impact of accommodations (or other alterations) on validity and fairness, in general, or geared toward specific tests or testing contexts. Studies could investigate, for example, how time constraints or different presentation formats influence the performance of test takers with and without a disability and therefore influence the validity of the test. Other studies could investigate how a specific test works under accommodated conditions (e.g., how is performance on Raven’s Progressive Matrices influenced by the following range of accommodations) or how tests in general function when a context forces a specific accommodation (e.g., how do tests function when items are read aloud by the test administrator, for example, in illiterate populations).
2. (2)
   We encourage test developers to address issues of universal design in their test development and validation efforts. Tests can be designed in such a way as to not need accommodations, at least on some issues; for example, the access by design approach builds accessibility into the very design of the test. For example, a test that contains images as stimuli can be designed for the images to be present with magnification and in high contrast to the standardized “default” stimuli. Of course, adopting a universal design would force test developers to address these issues in their technical reports, and we encourage such dedicated studies or sections in studies submitted to the EJPA.
3. (3)
   We encourage test developers, even when they do not adopt a universal design approach, to suggest accommodations that are possible on the test or at least discuss the most likely accommodations that may need to be made with the test, depending on the most likely audiences or testing contexts in which the test could be used with individuals or populations with disabilities. These possible accommodations will need to be discussed and documented, and their impact should be gauged already from the test development stage, and we encourage discussions and research reports on this topic.
4. (4)
   We encourage research into the impact of different criteria regarding the appropriateness of accommodations. Different frameworks (e.g., Hollenbeck, 2002; Phillips, 1994) propose different criteria on which to decide when accommodations should be granted (and how) and when not (and how not). These criteria balance the potential gains or losses of the different groups with the rights and needs of disabled or otherwise disadvantaged test-takers. But no empirical research has been identified on how different these criteria reflect the tenets of fairness and validity.

Professionals in different countries have different professional awareness of and legal obligations regarding the issue of testing accommodations. But we should never forget that test developers and researchers are, at the same time, educators who should continuously point toward good practices and provide evidence and guidance to educate and support these decisions, irrespective of the legal environment regarding the assessment of people with disabilities.

This is why we at EJPA strongly encourage empirical research into this considerable gap in our literature. Such research would inform professional decisions regarding accommodations taken by specialists in specific assessment contexts. Results could also reflect on our understanding of the competencies of test users (see International Test Commission, 2001) in the context in which the burden for decisions regarding some accommodations is delegated by test developers to test users. Such research would also inform formal policy documents in organizations that use assessment.