Abstract
Rituals shape many aspects of our lives, and they are no less common in scientific research than elsewhere. One that figures prominently in hypothesis testing is the null ritual, the pitting of hypotheses against chance. Although known to be problematic, this practice is still widely used. One way to resist the lure of the null ritual is to increase the precision of theories by casting them as formal models. These can be tested against each other, instead of against chance, which in turn enables a researcher to decide between competing theories based on quantitative measures. This article gives an overview of the advantages of modeling, describes research that is based on it, outlines the difficulties associated with model testing, and summarizes some of the solutions for dealing with these difficulties. Pointers to resources for teaching modeling in university classes are provided.
References
1973). Information theory and an extension of the maximum likelihood principle. In , Second International Symposium on Information Theory (pp. 267–281). Budapest, Hungary: Akademiai Kiado.
(2007). Editorial. Journal policies and procedures. Cognition, 102, 1–6.
(1976). Language, memory, and thought. Hillsdale, NJ: Erlbaum.
(1983). The architecture of cognition. Cambridge, MA: Harvard University Press.
(2004). An integrated theory of the mind. Psychological Review, 111, 1036–1060.
(1973). Human associative memory. Washington, DC: Winston and Sons.
(1991). Reflections of the environment in memory. Psychological Science, 2, 396–408.
(2004). The hot hand fallacy and the gambler’s fallacy: Two faces of subjective randomness? Memory and Cognition, 32, 1369–1378.
(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.
(1985). Statistical decision theory and Bayesian analysis (2nd ed.). New York: Springer.
(1979). Robustness in the strategy of scientific model-building. In , Robustness in statistics (pp. 201–236). New York: Academic Press.
(2006). Robust inference with simple cognitive models. In , Between a rock and a hard place: Cognitive science principles meet AI-hard problems. Papers from the AAAI Spring Symposium (AAAI Tech. Rep. No. SS-06–03, pp. 17–22). Menlo Park, CA: AAAI Press.
(2000). Cross-validation methods. Journal of Mathematical Psychology, 44, 108–132.
(1956). Perception and the representative design of psychological experiments (2nd ed.). Berkeley: University of California Press.
(2002). Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). New York: Springer.
(1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
(2009). Confidence intervals: Better answers to better questions. Zeitschrift für Psychologie / Journal of Psychology, 217, 15–26.
(1990). Constructing the subject. Cambridge, UK: Cambridge University Press.
(2003). Psychological models of professional decision making. Psychological Science, 14, 175–180.
(2007). Rituale. Formen, Funktionen, Geschichte. Eine Einführung in die Ritualwissenschaft [
(Rituals. Forms, functions, history. An introduction into the science of rituals ]. Stuttgart: Metzler.1971). Computer simulation of human behavior: A history of an intellectual technology. New York: Wiley.
(1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193–242.
(1993). An introduction to the bootstrap. New York: Chapman and Hall.
(1966). Elements of psychophysics. (H.E. Adler, Trans.). New York: Holt, Rinehart and Winston. (Original work published 1860).
(2000). Key concepts in model selection: Performance and generalizability. Journal of Mathematical Psychology, 44, 205–231.
(1996). The appropriate use of null hypothesis testing. Psychological Methods, 4, 379–390.
(2007). The cognitive modeling of human behavior: Why a model is (sometimes) better than 10,000 words. Cognitive Systems Research, 8, 135–142.
(2008). An ecological perspective to cognitive limits: Modeling environment-mind interactions with ACT-R. Judgment and Decision Making, 3, 278–291.
(2006). Simple predictions fuelled by capacity limitations: When are they successful? Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 966–982.
(1991). From tools to theories: A heuristic of discovery in cognitive psychology. Psychological Review, 98, 254–267.
(1996). On narrow norms and vague heuristics: A reply to Kahneman and Tversky. Psychological Review, 103, 592–596.
(1998a). Surrogates for theories. Theory and Psychology, 8, 195–204.
(1998b). We need statistical thinking, not statistical rituals. Behavioral and Brain Sciences, 21, 199–200.
(2008). Homo heuristicus: Why biased minds make better inferences. Manuscript submitted for publication.
(1996). Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, 104, 650–669.
(2008). Fast and frugal heuristics are plausible models of cognition: Reply to Dougherty, Franco-Watkins, & Thomas. Psychological Review, 115, 230–239.
(2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In , The Sage handbook of quantitative methodology for the social sciences (pp. 391–408). Thousand Oaks, CA: Sage.
(1987). Cognition as intuitive statistics. Hillsdale, NJ: Erlbaum.
(1990). Context effects and their interaction with development: Area judgments. Cognitive Development, 5, 235–264.
(1989). The empire of chance. How probability changed science and everyday life. Cambridge, UK: Cambridge University Press.
(1999). Simple heuristics that make us smart. New York: Oxford University Press.
(1996). Orthographic processing in visual word recognition: A multiple read-out model. Psychological Review, 103, 518–565.
(2007). The minimum description length principle. Cambridge, MA: MIT Press.
(1991). Why are formal models useful in psychology? In , Relating theory and data: Essays on human memory in honor of Bennet B. Murdock (pp. 39–56). Hillsdale, NJ: Erlbaum.
(2007). Heuristics and linear models of judgment: Matching rules and environments. Psychological Review, 114, 733–758.
(1998). Emergence: From chaos to order. Redwood City, CA: Addison-Wesley.
(1994). Models of visual word recognition. Sampling the state of the art. Journal of Experimental Psychology: Human Perception and Performance, 20, 1311–1334.
(2002). Representativeness revisited: Attribute substitution in intuitive judgment. In , Heuristics and biases: The psychology of intuitive judgment (pp. 49–81). New York: Cambridge University Press.
(1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3, 430–454.
(1967). Attribution theory in social psychology. In , Nebraska Symposium on Motivation (Vol. 15). Lincoln: University of Nebraska Press.
(1993). The rewards and hazards of computer simulations. Psychological Science, 4, 236–243.
(1993). Editorial comment. Memory and Cognition, 21, 1–3.
(1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin and Review, 1, 476–490.
(2004). Linear theory, dimensional theory, and the face-inversion effect. Psychological Review, 111, 835–863.
(1999). Where is mathematical modeling in psychology headed? Theory and Psychology, 9, 723–737.
(1982). Vision. San Francisco: W.H. Freeman.
(2003). Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental Psychology, 57, 203–220.
(1982). Evolution and the theory of games. Cambridge, UK: Cambridge University Press.
(1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.
(1962). Editorial. Journal of Experimental Psychology, 64, 553–557.
(1970). The significance test controversy. Chicago: Aldine.
(2000). Counting probability distributions: Differential geometry and model selection. Proceedings of the National Academy of Sciences USA, 97, 11170–11175.
(1997). Applying Occam’s razor in modeling cognition: A Bayesian approach. Psychonomic Bulletin and Review, 4, 79–95.
(2004). Assessing the distinguishability of models and the informativeness of data. Cognitive Psychology, 49, 47–84.
(1944). Theory of games and economic behavior. Princeton: Princeton University Press.
(1973a). You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium. In , Visual information processing (pp. 283–310). New York: Academic Press.
(1973b). Production systems: Models of control structures. In , Visual information processing (pp. 463–526). New York: Academic Press.
(2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
(2005). The cognitive substrate of subjective probability. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 600–620.
(1993). The adaptive decision maker. New York: Cambridge University Press.
(2006). Global model analysis by parameter space partitioning. Psychological Review, 113, 57–83.
(2000). When a good fit can be bad. Trends in Cognitive Sciences, 6, 421–425.
(2002). Toward a method for selecting among computational models for cognition. Psychological Review, 109, 472–491.
(2002). Cognitive modeling. Cambridge, MA: MIT Press.
(1981). Search of associative memory. Psychological Review, 88, 93–134.
(1999). Connectionist and diffusion models of reaction time. Psychological Review, 106, 261–300.
(2006). SSL: A theory of how people learn to select strategies. Journal of Experimental Psychology: General, 135, 207–236.
(2007). Cognitive tutor: Applied research in mathematics education. Psychonomic Bulletin and Review, 14, 249–255.
(2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107, 358–367.
(2009). Effect sizes: Why, when, and how to use them. Zeitschrift für Psychologie / Journal of Psychology, 217, 6–14.
(1986). Parallel distributed processing: Explorations in the microstructure of cognition (Vol. I). Cambridge, MA: MIT Press.
(2006). Modeling driver behavior in a cognitive architecture. Human Factors, 48, 362–380.
(1969). Models of segregation. American Economic Review, 59, 499–493.
(1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1172–1181.
(1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115–129.
(2005). How forgetting aids heuristic inference. Psychological Review, 112, 610–628.
(1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.
(1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309–316.
(2007). Forschungsmethoden und Statistik in der Psychologie [
(Research methods and statistics in psychology ]. Munich: Pearson Education.2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248–1284.
(1957). On the psychophysical law. Psychological Review, 64, 153–181.
(1974). Cross-validatory choice and assessment of statistical predictions (with discussion). Journal of the Royal Statistical Society, Series B, 36, 111–147.
(1977). Asymptotics for and against cross-validation. Biometrika, 64, 29–35.
(2000). Teaching social simulation with Matlab. Journal of Artificial Societies and Social Simulation, 3. Retrieved August 3, 2008, from http://www.soc.surrey.ac.uk/JASSS/3/1forum/1.html.
(1975). The mind-body problem revisited. In , Philosophical aspects of the mind-body problem (pp. 200–218). Honolulu, HI: Honolulu University Press.
(1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
(2003). How many parameters does it take to fit an elephant? Journal of Mathematical Psychology, 47, 580–586.
(2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804.
(1999). Statistical methods in psychology journals. Guidelines and explanations. American Psychologist, 54, 594–604.
(1938). Experimental psychology. New York: Holt.
(