Are Difficult-To-Study Populations too Difficult to Study in a Reliable Way?
Lessons Learned From Meta-Analyses in Clinical Neuropsychology
Abstract
Abstract. Replication studies, pre-registration, and increases in statistical power will likely improve the reliability of scientific evidence. However, these measures face critical limitations in populations that are inherently difficult to study. Members of difficult-to-study populations (e.g., patients, children, non-human animals) are less accessible to researchers, which typically results in small-sample studies that are infeasible to replicate. Nevertheless, meta-analyses on clinical neuropsychological data suggest that difficult-to-study populations can be studied in a reliable way. These analyses often produce unbiased effect-size estimates despite aggregating across severely underpowered original studies. This finding can be attributed to a neuropsychological research culture involving the non-selective reporting of results from standardized and validated test procedures. Consensus guidelines, test manuals, and psychometric evidence constrain the methodological choices made by neuropsychologists, who regularly report the results from neuropsychological test batteries irrespective of their statistical significance or novelty. Comparable shifts toward more standardization and validation, complete result reports, and between-lab collaborations can allow for a meaningful and reliable study of psychological phenomena in other difficult-to-study populations.
References
1996). Frontal lobe dysfunction in amyotrophic lateral sclerosis A PET study. Brain, 119, 2105–2120. https://doi.org/10.1093/brain/119.6.2105
(2010). Clinical Neuropsychology. Retrieved from http://www.apa.org/ed/graduate/specialize/neuro.aspx
. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119. https://doi.org/10.1002/per.1919
(2017). Improving the rigor of psychophysiology research. International Journal of Psychophysiology, 111, 5–16. https://doi.org/10.1016/j.ijpsycho.2016.04.006
(2016). Misguided effort with elusive implications. Perspectives on Psychological Science, 11, 574–575. https://doi.org/10.1177/1745691616652878
(1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101. https://doi.org/10.2307/2533446
(1948). A simple objective technique for measuring flexibility in thinking. The Journal of General Psychology, 39, 15–22. https://doi.org/10.1080/00221309.1948.9918159
(2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144, 796–815. https://doi.org/10.1037/xge0000083
(2013). Is ego depletion too incredible? Evidence for the overestimation of the depletion effect. Behavioral and Brain Sciences, 36, 683–684. https://doi.org/10.1017/S0140525X13000952
(2014). Publication bias and the limited strength model of self-control: Has the evidence for ego depletion been overestimated? Frontiers in Psychology, 5, 823. https://doi.org/10.3389/fpsyg.2014.00823
(2013). Global epidemiology of amyotrophic lateral sclerosis: A systematic review of the published literature. Neuroepidemiology, 41, 118–130. https://doi.org/10.1159/000351153
(2004).
(Psychometric foundations of neuropsychological assessment . In L. H. GoldsteinJ. E. McNeilEds., Clinical neuropsychology: A practical guide to assessment and management for clinicians (pp. 121–140). Chichester, UK: Wiley.2003). A meta-analytic review of the sensitivity of the Wisconsin Card Sorting Test to frontal and lateralized frontal brain damage. Neuropsychology, 17, 255–264. https://doi.org/10.1037/0894-4105.17.2.255
(2006). Meta-analysis in neuropsychology: An introduction. The Clinical Neuropsychologist, 20, 5–9. https://doi.org/10.1080/13854040500203266
(2016). Everything is crumbling. Slate. Retrieved from http://www.slate.com/articles/health_and_science/cover_story/2016/03/ego_depletion_an_influential_theory_in_psychology_may_have_just_been_debunked.html
(1978). An exercise in mega-silliness. American Psychologist, 33, 517. https://doi.org/10.1037/0003-066X.33.5.517.a
(2010). How to do a meta‐analysis. British Journal of Mathematical and Statistical Psychology, 63, 665–694. https://doi.org/10.1348/000711010X502733
(2016). Awaji criteria improves the diagnostic sensitivity in amyotrophic lateral sclerosis: A systematic review using individual patient data. Clinical Neurophysiology, 127, 2684–2691. https://doi.org/10.1016/j.clinph.2016.04.005
(2017). The statistical crisis in science: How is it relevant to clinical neuropsychology? The Clinical Neuropsychologist, 31, 1000–1014. https://doi.org/10.1080/13854046.2016.1277557
(2014). The statistical crisis in science. American Scientist, 102, 460–465. https://doi.org/10.1511/2014.111.460
(1994). Separate neural pathways for the visual analysis of object shape in perception and prehension. Current Biology, 4, 604–610. https://doi.org/10.1016/S0960-9822(00)00132-9
(1948). A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of Experimental Psychology, 38, 404–411. https://doi.org/10.1037/h0059831
(2016). Commentary: Misguided effort with elusive implications, and sifting signal from noise with replication science. Frontiers in Psychology, 7, 621. https://doi.org/10.3389/fpsyg.2016.00621
(2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11, 546–573. https://doi.org/10.1177/1745691616652873
(2010). Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin, 136, 495–525. https://doi.org/10.1037/a0019486
(2011). A neurophysiological analysis of working memory in amyotrophic lateral sclerosis. Brain Research, 1421, 90–99. https://doi.org/10.1016/j.brainres.2011.09.010
(2015). Bias-correction techniques alone cannot determine whether ego depletion is different from zero: Commentary on Carter, Kofler, Forster, & McCullough, 2015. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2659409
(2005). Why most published research findings are false. PLoS Medicine, 2, e124. https://doi.org/10.1371/journal.pmed.0020124
(1995). Double dissociation of memory capacities after bilateral occipital-lobe or medial temporal-lobe lesions. Brain, 118, 1129–1148. https://doi.org/10.1093/brain/118.5.1129
(2004). A longitudinal study of the evolution of cognitive function and affective state in patients with amyotrophic lateral sclerosis. Amyotrophic Lateral Sclerosis, 5, 46–54. https://doi.org/10.1080/14660820310017560
(2014). Investigating variation in replicability. Social Psychology, 45, 142–152. https://doi.org/10.1027/1864-9335/a000178
(2019). The reliability of the Wisconsin Card Sorting Test in clinical practice. Assessment. Advance online publication. https://doi.org/10.1177/1073191119866257
(2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One, 9, e105825. https://doi.org/10.1371/journal.pone.0105825
(2018). Executive dysfunction in Parkinson’s disease: A meta-analysis on the Wisconsin Card Sorting Test literature. Neuroscience & Biobehavioral Reviews, 93, 38–56. https://doi.org/10.1016/j.neubiorev.2018.06.014
(2014). Sweet delusion. Glucose drinks fail to counteract ego depletion. Appetite, 75, 54–63. https://doi.org/10.1016/j.appet.2013.12.020
(2017). Cognitive flexibility and its electrophysiological correlates in Gilles de la Tourette syndrome. Developmental Cognitive Neuroscience, 27, 78–90. https://doi.org/10.1016/j.dcn.2017.08.008
(2016). Meta-analytical and electrophysiological evidence for executive dysfunction in primary dystonia. Cortex, 82, 133–146. https://doi.org/10.1016/j.cortex.2016.05.018
(2016). Impaired set-shifting in amyotrophic lateral sclerosis: An event-related potential study of executive function. Neuropsychology, 30, 120–134. https://doi.org/10.1037/neu0000218
(2011). Sexy but often unreliable: The impact of unreliability on the replicability of experimental findings with implicit measures. Personality and Social Psychology Bulletin, 37, 570–583. https://doi.org/10.1177/0146167211400619
(2004). Do schizophrenia patients make more perseverative than non-perseverative errors on the Wisconsin Card Sorting Test? A meta-analytic study. Psychiatry Research, 129, 179–190. https://doi.org/10.1016/j.psychres.2004.06.016
(2015). Validation of mismatch negativity and P3a for use in multi-site studies of schizophrenia: Characterization of demographic, clinical, cognitive, and functional correlates in COGS-2. Schizophrenia Research, 163, 63–72. https://doi.org/10.1016/j.schres.2014.09.042
(2016). No evidence of the ego-depletion effect across task characteristics and individual differences: A pre-registered study. PLoS One, 11, e0147770. https://doi.org/10.1371/journal.pone.0147770
(2017). Challenges to ego-depletion research go beyond the replication crisis: A need for tackling the conceptual crisis. Frontiers in Psychology, 8, 568. https://doi.org/10.3389/fpsyg.2017.00568
(2014). The evolution of self-control. Proceedings of the National Academy of Sciences, 111, E2140–E2148. https://doi.org/10.1073/pnas.1323533111
(1991). Half a century of research on the Stroop effect: An integrative review. Psychological Bulletin, 109, 163–203. https://doi.org/10.1037/0033-2909.109.2.163
(2019). Large scale replication projects in contemporary psychological research. The American Statistician, 73, 99–105. https://doi.org/10.1080/00031305.2018.1505655
(2016). The Peer Reviewers’ Openness Initiative: Incentivizing open research practices through peer review. Royal Society Open Science, 3, 150547. https://doi.org/10.1098/rsos.150547
(2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
(2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. https://doi.org/10.1126/science.aac4716
. (2009). Emotional lability in MND: Relationship to cognition and psychopathology and impact on caregivers. Journal of the Neurological Sciences, 278, 16–20. https://doi.org/10.1016/j.jns.2008.10.025
(2010). Cognitive dysfunction in lower motor neuron disease: Executive and memory deficits in progressive muscular atrophy. Journal of Neurology, Neurosurgery & Psychiatry, 82, 170–175. https://doi.org/10.1136/jnnp.2009.204446
(1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641. https://doi.org/10.1037/0033-2909.86.3.638
(2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82. https://doi.org/10.1146/annurev.psych.52.1.59
(2016). Event-related potentials and cognition in Parkinson’s disease: An integrative review. Neuroscience & Biobehavioral Reviews, 71, 691–714. https://doi.org/10.1016/j.neubiorev.2016.08.003
(1997). Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away. Clinical Psychology Review, 17, 881–901. https://doi.org/10.1016/S0272-7358(97)00056-1
(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. https://doi.org/10.1177/0956797611417632
(2012). The non-motor syndrome of primary dystonia: Clinical and pathophysiological implications. Brain, 135, 1668–1681. https://doi.org/10.1093/brain/awr224
(2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11, 702–712. https://doi.org/10.1177/1745691616658637
(1959). Publication decisions and their possible effects on inferences drawn from tests of significance – or vice versa. Journal of the American Statistical Association, 54, 30–34. https://doi.org/10.2307/2282137
(2017). Replicability and reproducibility in comparative psychology. Frontiers in Psychology, 8, 862. https://doi.org/10.3389/fpsyg.2017.00862
(2006). A compendium of neuropsychological tests (3rd ed.). New York, NY: Oxford University Press.
(2002). Adult clinical neuropsychology: Lessons from studies of the frontal lobes. Annual Review of Psychology, 53, 401–433. https://doi.org/10.1146/annurev.psych.53.100901.135220
(2015). The utility of P300 as a schizophrenia endophenotype and predictive biomarker: Clinical and socio-demographic modulators in COGS-2. Schizophrenia Research, 163, 53–62. https://doi.org/10.1016/j.schres.2014.09.024
(2015). Meta-analyses are no substitute for registered replications: A skeptical perspective on religious priming. Frontiers in Psychology, 6, 1365. https://doi.org/10.3389/fpsyg.2015.01365
(2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13, 411–417. https://doi.org/10.1177/1745691617751884
(2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. https://doi.org/10.1177/1745691612463078
(2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. https://doi.org/10.1017/S0140525X17001972
(