Current Issues in Optimal Design

The quality of psychological studies substantially depends on their experimental design. The design has to be specified before starting data collection. Wrong decisions concerning the design can only be corrected with great difficulty later on, if at all. Thus, considerable attention should be devoted to the experimental design.

An experimental design concerns all steps of how to perform the experiment. Therefore, many questions have to be answered, for instance: How many factors, levels per factor, and time points need to be investigated? Should the factors be fixed or random? How many experimental units should be selected for each condition? Which covariates have to be included?

Many criteria to evaluate the goodness of designs have been developed. Designs should provide precise estimates of parameters and allow for sound testing of hypotheses. Thus, small standard errors of the estimates and a high power of the test are preferable. Furthermore, internal and external validity are among other standards which are mainly affected by the experimental design.

Optimal design considers an important issue in experimental design. It specifies the distribution of the values of the independent variables, depending on statistical criteria determined a priori. When comparing the outcomes of two groups by a t-test, for example, a desirable criterion has to be specified first. Obviously, the difference of the means should be estimated as precisely as possible, resulting in a small standard error or variance of this difference. Based on this criterion and given a certain sample size, the proportion of subjects allocated to the experimental or, equivalently, to the control group has to be derived such that the criterion just outlined is best fulfilled.

Optimal design for simple regression implies determining the values of the independent variable and the number of observations at each selected value. For instance, in the mental rotation paradigm (Shepard & Metzler, 1971), the reaction time is analyzed depending on stimulus rotation. Here, the optimal design specifies the degrees of rotation to be selected as, for example, 0°, 45°, 90°, 135°, and 180°, as well as the frequency of stimulus presentation. Often, the stimuli with different degrees of rotation are presented with the same frequency. Whether such a design is a good decision depends on the statistical criterion to be optimized. In simple regression, several criteria may be specified, for example, minimizing the standard error of the intercept, the standard error of the slope, or the average standard error of both. Or the predicted reaction time for certain values of the independent variable should be as precise as possible.

Optimal design has a long history in statistics. It goes back to Smith (1918), who wrote the first explicit paper about an optimal design problem. Many important contributions to optimal design theory were made in the 1950s, for example, by Elfving (1952) and Kiefer and Wolfowitz (1959, 1960). The first monographs on optimal design were published by Fedorov (1972) and Silvey (1980).

In the beginning, optimal design for linear models was dominant. This area is now well developed (e.g., Schwabe, 1996). Optimal design for nonlinear models is more challenging since such designs in general depend on the parameters to be estimated. Several kinds of designs were developed to cope with this problem, such as adaptive, Bayesian, and maximin designs (Holling & Schwabe, in press). The development of such designs for nonlinear mixed models is one current topic of special research interest.

Although optimal design has a long history, it did not receive much attention in psychology for a long time. Several reasons may be responsible for this. First, experiments in psychology are often not as expensive as in other fields, such as engineering or chemistry – high effort and cost motivate to thoroughly plan experiments. Second, textbooks on experimental design usually do not include a chapter on optimal design. Third, much of the literature about optimal design is written rather technically and requires profound statistical knowledge. Finally, software packages frequently used in psychology hardly provide opportunities to generate optimal designs.

Recently, some more introductory books have been published. First, the monograph by Berger and Wong (2009) should be mentioned. This comprehensively presents most of the important topics of optimal design relevant for psychological research. Goos and Jones (2012) introduce optimal design using a case study approach. A more advanced, often cited monograph was written by Atkinson, Donev, and Tobias (2007). A comprehensive overview of optimal design theory which, however, presupposes advanced statistical knowledge, is provided by Pukelsheim (2006). An introductory article emphasizing optimal design in psychology has been published by McClelland (1997), while Holling and Schwabe (in press) provide an overview of optimal design theory stressing issues of item response theory.

Application of statistical methods, in general, requires statistical software. Many popular statistical software packages offer few opportunities for generating optimal designs, if any. Exceptions are the software packages SAS and JMP from SAS Institute Inc., which include several procedures for generating optimal designs. There is a growing number of programs for optimal design written in R (e.g., Grömping, 2011; Rasch, Pilz, Verdooren, & Gebhardt, 2011). Furthermore, many stand-alone programs or programs written in Matlab or Mathematica are available for generating optimal designs for specific statistical tasks.

The demand for optimal designs in psychological research has been growing in recent years. Neuroimaging methods like fMRI have become a standard procedure in psychological research. Since these methods are rather complex and expensive, optimal design may considerably reduce the time and cost involved (Maus & van Breukelen, 2013). Optimal design is extremely valuable for large-scale assessments, for example, the Programme for International Student Assessment (PISA; OECD, 2012) or the Trends in International Mathematics and Science Study (TIMSS; Mullis, Martin, Ruddock, O’Sullivan, & Preuschoff, 2009). Such studies require large sample sizes, which may be kept to a minimum by using optimal design (see Kuhn & Kiefer, 2013). Adaptive testing, which minimizes the number of items, also requires optimal design (see Holling & Schwabe, in press). However, practically every psychological study may benefit from optimal design, since the reduction of observations necessary is often considerable.

This topical issue provides an introduction to optimal design and an overview on new developments and applications of optimal design to several important topics in psychology. These topics consist of multilevel analysis, event history analysis, functional magnetic resonance imaging (fMRI) studies, and optimal test assembly. All authors have paid intense attention to high readability. Thus, these articles only presuppose statistical knowledge as typically provided by the BSc and MSc study of psychology. More technical issues are always deferred to an appendix.

The first article by Holling and Schwabe (2013) provides an introduction to optimal design using some typical psychological research issues. The basics of this subject are presented by means of the t-test and simple regression, two common statistical procedures. Optimal designs are easily derived for both methods and it is shown how much additional sample size is required to achieve the same statistical performance when using nonoptimal designs. Then, optimal design is more generally outlined for linear models. Several optimality criteria that have been proposed in the literature are introduced and it is shown that the selection of an adequate criterion to be optimized depends on the hypotheses of a study. Developing optimal designs for nonlinear models is much more demanding than for linear models. Here, as a basic problem, the design to be created depends upon the unknown parameters. This issue is illustrated by logistic regression with a binary predictor. This simple nonlinear statistical method is also used to fundamentally explain adaptive, Bayesian, and minimax designs.

Multilevel studies are frequently encountered in psychological research. Experimental or observational units are frequently nested in groups or clusters, for example, pupils in school classes, employees in organizations, or repeated measurements within persons. Data have to be sampled at different levels – the unit level and the group level. Thus, optimal design for multilevel studies concerns the levels of the independent variables as well as the sampling of units at the different levels of the study. Since multilevel studies usually require large samples of subjects and clusters, optimal design is especially important for this kind of modeling. Van Breukelen (2013) gives a comprehensive and clearly written introduction to optimal design for multilevel studies. Optimal designs of cluster randomized trials are presented and illustrated by an example regarding stress management in primary school. Then, optimal designs of multisite trials are outlined as well as models that include covariates. Furthermore, the author discusses optimal designs for other multilevel models such as models with dichotomous outcomes or nested experiments with two and three levels. Finally, hints for useful software are given.

In the contribution by Moerbeek and Jóźwiak (2013), optimal designs for event history analysis are derived. Event history analysis, also known as survival analysis, is a statistical method to analyze the occurrence and timing of events. Such events may be job loss, an accident at work, or premature psychotherapy termination. The occurrence and timing of events can be related to covariates and points in time with a high probability of the event occurrence being identified. Therefore, event analysis can be usefully applied in many psychological disciplines, such as clinical or work and organizational psychology. Moerbeek and Jóźwiak (2013) consider event history analysis with discrete time intervals. Here, it is not necessary to know the exact time points at which the events occur. Before deriving optimal designs, the authors give a comprehensible introduction to event history analysis. In optimal design for such models, the number of subjects and time intervals has to be determined. Moreover, the authors analyze the effects of costs and attrition on the optimal designs. Their results are illustrated by an example on drinking onset. Finally, the authors present a user-friendly computer program that facilitates the calculation of optimal design for event history analysis for researchers.

In recent years, functional magnetic resonance imaging (fMRI) has rapidly emerged as a versatile technique in psychology as well as in neuroscience. It is, however, a very elaborate and expensive method. For this very reason, optimal design provides a useful technique to increase the efficiency of fMRI experiments. Maus and van Breukelen (2013) present a comprehensive overview of optimal design for fMRI studies. The authors first introduce the fundamental methodology of such studies. They describe common objectives of two basic types of fMRI experiments, blocked and event-related designs, and outline statistical models used for data analysis of this kind of studies. Then, the most important results for optimal design of blocked and event-related designs concerning different design objectives are explained and it is shown that fMRI studies can considerably benefit from optimal design.

Kuhn and Kiefer (2013) describe the optimization procedures used in generating the design of a large-scale assessment of educational standards in mathematics in Austria. The main purpose of large-scale assessments is to quantify the degree of mastery that a population of interest has obtained in a specific content domain (e.g., mathematics). Parameters at the population level are of key interest here. However, the design of large-scale assessments is a complex endeavor, as several, sometimes competing goals have to be taken into consideration simultaneously (e.g., unbiased estimation of population parameters, longitudinal linking across assessment cycles). Kuhn and Kiefer (2013) describe the two-step procedure used in generating the test design of the educational standards assessment of mathematics in Austria. First, they used a local search heuristic, that is, simulated annealing, to generate a nearly optimal design at the item block level. Second, they used linear optimization to allocate items from a large item pool to the item blocks. In addition to providing an introduction to optimal test design in large-scale assessment, their work carefully outlines decisions and constraints made in the design process that are often not explicitly mentioned in the literature, thus allowing a more detailed grasp of the complexities faced in designing large-scale assessments.