Overcoming the Inverse Probability Fallacy
A Comparison of Two Teaching Interventions
Abstract
Many common misinterpretations of Null Hypothesis Significance Testing (NHST) are related to the inverse probability fallacy. The inverse probability fallacy is the mistaken belief that the probability of the data given the null hypothesis, P(D|H0), is equivalent to the probability of the null hypothesis given the data, P(H0|D). We contrasted the effectiveness of two teaching interventions aimed at reducing this fallacy: Instruction in Bayes’ theorem (group B) and instruction in the formal logic of NHST (Modus Tollens, group MT). Both interventions were remarkably effective in reducing fallacy. At pre-test, 82% of students agreed with at least one statement of the inverse probability fallacy. At post-B-intervention this figure was 49% and at post-MT, it was 48%. A smaller, but still substantial, effect remained in both groups at a five-week follow-up. This suggests that the essential ingredient in overcoming the inverse probability fallacy is simply to expose the null ritual as problematic.
References
(2001). The practice of social research, 9th ed. London: Wadsworth.
(1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2, 141–178.
(1978). The case against statistical significance testing. Harvard Educational Review, 48, 378–399.
(1988). Statistical power analysis for the behavioural sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum.
(1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
(2005). Inference by eye: Confidence intervals, and how to read pictures of data. American Psychologist, 60, 170–180.
(1999). Educational design for effective learning: Building and using a computer-based learning environment for statistical concepts. In , Advanced research in computers and communications in education (pp. 558–565). Amsterdam: IOS Press.
(1995). Learning environments for conceptual change: The case of statistics. In , Artificial intelligence in education (pp. 389–396). VA: AACE.
(1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In , Judgment under uncertainty: Heuristics and biases (pp. 249–267). New York: Cambridge University Press.
(1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory and Psychology, 5, 75–98.
(1983). Hypothesis evaluation from a Bayesian perspective. Psychological Review, 90, 239–260.
(1993). The superego, the ego, and the id in statistical reasoning. In , A handbook for data analysis in the behavioral sciences: Methodological issues. Hillsdale, NJ: Lawrence Erlbaum Associates.
(2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In , The Sage handbook of quantitative methodology for the social sciences (pp. 391–408). Thousand Oaks, CA: Sage.
(2002). Misinterpretations of significance: A problem students share with their teachers? Methods of Psychological Research, 7, 1–20.
(1984). Syllogistic inference. Cognition, 16, 1–61.
(2004). Beyond significance testing: Reforming data analysis methods in the behavioural sciences. Washington, DC: American Psychological Association.
(1963). Changes in critical thinking, attitudes and values from freshman to senior years. Journal of Educational Psychology, 54, 305–315.
(2000). Null Hypothesis Significance Testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
(1986). Statistical inference: A commentary for the social and behavioural sciences. Chichester, UK: Wiley.
(1987). On the probability of making Type I errors. Psychological Bulletin, 102, 159–163.
(1985). Mathematical problem solving. Orlando, Fla: Academic Press.
(1993). ThinkerTools: Causal models, conceptual change and science education. Cognition and Instruction, 10, 1–100.
