Abstract
Observed power analysis is recommended by many scholarly journal editors and reviewers, especially for studies with statistically nonsignificant test results. However, researchers may not fully realize that blind observance of this recommendation could lead to an unfruitful effort, despite the repeated warnings from methodologists. Through both a review of 14 published empirical studies and a Monte Carlo simulation study, the present study demonstrates that observed power is usually not as informative or helpful as we think because (a) observed power for a nonsignificant test is generally low and, therefore, does not provide additional information to the test; and (b) a low observed power does not always indicate that the test is underpowered. Implications and suggestions of statistical power analysis for quantitative researchers are discussed.
References
. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: American Psychological Association.
. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.
(2007). History of childhood sexual abuse and HIV risk behaviors in homosexual and bisexual men. American Journal of Public Health, 97, 1107–1112.
(2007). The effect of web-based question prompts on scaffolding knowledge integration and ill-structured problem solving. Journal of Research on Technology in Education, 39, 359–375.
(2007). Age and gender differences in online behavior, self-efficacy, and academic performance. The Quarterly Review of Distance Education, 8, 213–222.
(1962). The statistical power of abnormal social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.
(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
(1992). Statistical power analysis. Current Directions in Psychological Science, 1, 98–101.
(2003). Confidence intervals are a more useful complement to nonsignificant tests than are power calculations. Behavioral Ecology, 14, 446–447.
(2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 1, 532–574.
(2007). Beyond pragmatics: Morphosyntactic development in autism. Journal of Autism and Development Disorder, 37, 1007–1023.
(2004). Replicability reconsidered: An excessive range of possibilities. Understanding Statistics, 3, 365–373.
(1998). Limits of retrospective power analysis. The Journal of Wildlife Management, 62, 801–807.
(1994). Post hoc power analysis. Journal of Applied Psychology, 79, 783–785.
(1994). The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Annals of Internal Medicine, 121, 200–206.
(1997). Statistical power analysis and amphibian population trend. Conservation Biology, 11, 273–275.
(2008). An examination of nonsuicidal self-injury among college students. Journal of Mental Health Counseling, 30, 137–156.
(1987). How hard is hard science, how soft is soft science?. American Psychologist, 42, 443–455.
(2006). Effect size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist, 34, 601–629.
(2006). Keen to help? Managers’ implicit person theories and their subsequent employee coaching. Personnel Psychology, 59, 871–902.
(2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55, 19–24.
(1995). Continuous univariate distributions (2nd ed.). New York, NY: Wiley.
(2007). Searching under cups for clues about memory – An online demonstration. Teaching of Psychology, 34, 124–128.
(2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
(2001). Post hoc power analysis: An idea whose time has passed? Pharmacotherapy, 21, 405–409.
(2007). Breaking down automaticity: Case ambiguity and the shift to reflective approaches in clinical reasoning. Medical Education, 41, 1185–1192.
(2007). Emotion perception in Asperger’s syndrome and high-functioning autism: The importance of diagnostic criteria and cue intensity. Journal of Autism and Development Disorder, 37, 1086–1095.
(2007). Post hoc power, observed power, a priori power, retrospective power, prospective power, achieved power: Sorting out appropriate uses of statistical power analyses. Communication Methods and Measures, 1, 291–299.
(2004). Post hoc power: A concept whose time has come. Understanding Statistics, 3, 201–230.
(1995). Assessment of nondeclining amphibian population using power analysis. Conservation Biology, 9, 1299–1300.
(2000). Multivariate models of mixed assortment: Phenotypic assortment and social homogamy for education and fluid ability. Behavior Genetics, 30, 455–76.
(2006). The effects of metacognitive training on the academic achievement and happiness of Esfahan University conditional students. Counselling Psychology Quarterly, 19, 415–428.
(2006). Early cochlear implant experience and emotional functioning during childhood: Loneliness in middle and late childhood. The Volta Review, 106, 365–379.
(1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309–316.
(2000). Testing for robustness in Monte Carlo studies. Psychological Methods, 5, 230–240.
(1992). Confidence limit analyses should replace power calculations in the interpretation of epidemiologic studies. Epidemiology, 3, 449–452.
(2006). Client similarities and differences in two childhood anxiety disorders research clinics. Journal of Clinical Child and Adolescent Psychology, 35, 528–538.
(2007). The influence of the discussion leader procedure on the quality of arguments in online discussion. Journal of Educational Computing Research, 37, 83–103.
(1997). Statistical power analysis in wildlife research. The Journal of Wildlife Management, 61, 270–279.
(1997). Retrospective power analysis. Conservation Biology, 11, 276–280.
(2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 25–32.
(2005). On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics, 30, 141–167.
(1998). A note on misconceptions concerning prospective and retrospective power. The Statistician, 47, 385–388.
