8th Congress of the European Association of Methodology (EAM)

The Fallacy of Facts and Findings

Science is about facts, and findings should be replicable. Almost all of us would tend to subscribe. However, thinking about this statement for a while and looking at the world from a probabilistic point of view – and this is what we do as soon as we apply inferential statistics – that point of view turns out to be a highly questionable. From a probabilistic point of view, facts are not at all what science is about and findings are not necessarily expected to replicate. This statement is certainly provocative, but it might motivate to rethink what science, and in particular, empirical research is about. The first paper in this issue is “The statistics of replication” by Larry Hedges (2019) reminds us of what we have learned in our first course on inferential statistics. Scientific theories and hypotheses do not refer to facts and findings, that is to data and to significant or non-significant results; instead they refer to expectations, conditional expectations and their differences, true variances, true covariances or correlations, distributions and conditional distributions of random variables, or other probabilistic terms. Scientific theories and hypotheses, at least those that can empirically be investigated via inferential statistics, refer to random experiments and the parameters that describe the distributions of the random variables that can be considered in such a random experiment. For example, they do not refer to sample means and their differences, but to the underlying conditional expectations and their differences. These concepts are no facts at all; instead they are of a purely theoretical nature, and they are not observable. Their general definition requires the theory of integrals with respect to a probability measure (see, e.g., chapter 3 in Steyer & Nagel, 2017). Similarly, a finding such as “the sample mean difference is significant at the .05 level” is not really what we are interested in. From a scientific point of view, such a finding is just a random event, which occurs with a certain probability if we assume that a specific hypothesis is true. What is of interest instead is the difference between conditional expectation values. Larry Hedges’ paper explores the consequences of this point of view on the so-called replication crises and the methodology of replication studies.

In the second paper, “Causal effects based on latent variable models,” Axel Mayer amplifies this point (2019). Neither causal effects nor latent variables, nor causal effects on latent variables are entities that refer to something beyond a random experiment. Instead causal effects are parameters that are defined within or with explicit reference to a random experiment and the same applies to the values of latent variables. They are defined as specific functions of conditional expectations. However, their definition refers to probabilistic concepts that require some sophistication, which is not acquired in a single course on probability theory. This does not only apply to the definition of causal effects but also to the definition of latent variables. Nevertheless, Axel Mayer’s program EffectLiteR is a great example how highly sophisticated concepts such as causal effects and latent variables can lead to programs for data analysis that are easy to handle, even for applied researchers with a minimal background in probability theory.

In the third paper, “Continuous-time latent Markov factor analysis to explore longitudinal measurement invariance in experience sampling data,” Leonie Vogelsmeier and her coauthors (2019) lead us to another construction site of research methodology: continuous-time processes of latent variables. Again, we have to rethink our ways of thinking, our constructs, and the way we are doing research. Clearly, depression is not a value on a scale such as the Beck depression inventory. But is it a latent variable? Or is it rather a continuous-time process of latent variables? If the latter is true – and this is what I believe – then things are even more sophisticated and researchers will need years to learn the necessary background including continuous-time stochastic processes.

In the fourth paper, “A comparison of simple structure rotation criteria in temporal exploratory factor analysis for event-related potential data,” Scharf and Nestler (2019) deal with problems of reducing continuous-time data that are assessed in EEG research, comparing different rotation methods of exploratory factor analysis. Developing the methodology in this field seems important, considering the enormous complexity of these data structures.

The papers in this special issue and the complexity of the problems they address clearly indicate that we need to rethink the organization of study programs and of empirical research. The methodology of empirical research is far too complex to be taught in a few courses in substantive sciences such as psychology, sociology, or economy, for instance. Instead, methodology is a discipline of its own requiring new study programs in methodology in order to learn only the essentials. Similarly, in many cases, the methodological background of an empirical study is far too complex for most substantive researchers to fully grasp the meaning and the implications of their findings. This definitely necessitates increased cooperation between substantive scientists and methodologists in order to conduct and communicate empirical research.