Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Commentary

Correcting Smith et al.’s (2018) Criticisms of All Rorschach Studies in Mihura, Meyer, Dumitrascu, and Bombel’s (2013) Meta-Analyses

Published Online:December 03, 2019https://doi.org/10.1027/1192-5604/a000118

Abstract

Abstract.Smith et al. (2018) describe their article as “an evaluation as to the extent that individual studies have conformed to [Exner’s (1995a)] proposed methodological criteria” (Abstract). However, the authors did not conduct analyses to compare research before and after Exner (1995a) in order to assess its impact nor were the set of criteria they used Exner’s. Instead, they critiqued the individual studies in Mihura and colleagues’ (2013) meta-analyses, declaring all methodologically unsound (including Exner’s). They conjectured that Mihura et al. omitted studies with less “methodological bias” that would have provided more support for Rorschach validity. I explain why most of the criteria they use to criticize the studies’ methodology are not sound. But to directly test their hypotheses, I requested their ratings of study methodology. Findings from studies they rated as having more methodological “issues” (e.g., not reporting IQ or Lambda range) or as being “application studies” – which they said should be excluded – were not less supportive of Rorschach validity as they assumed would be the case. The small effect size associations (r < |.10|) were also in the opposite direction of which Smith et al. argued to be true, indicating that the criteria by which they evaluated other researchers’ studies were not sound. Our findings do indicate that researchers are responding to the one criterion that is clearly stated in Exner (1995a), which is Weiner’s (1991) recommendation to report interrater reliability; before 1991, 12% of studies reported interrater reliability, which afterward jumped to 78.4%. Other claims in the article by Smith et al. are also addressed.

References

  • Andronikof, A., Erdberg, S. P., Weiner, I. B., Meyer, G. J., & Smith, B. L. (2010, March). Wither the Rorschach: Revolution, evolution, or status quo? Roundtable discussion at the annual meeting of the Society for Personality Assessment, San Jose, CA. First citation in articleGoogle Scholar

  • Ball, J. D., Archer, R. P., Gordon, R. A., & French, J. (1991). Rorschach Depression indices with children and adolescents: Concurrent validity findings. Journal of Personality Assessment, 57, 465–476. https://doi.org/10.1207/s15327752jpa5703_6 First citation in articleCrossref, Google Scholar

  • Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061 First citation in articleCrossref, Google Scholar

  • Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376. https://doi.org/10.1038/nrn3475 First citation in articleCrossref, Google Scholar

  • Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). “Power failure: Why small sample size undermines the reliability of neuroscience”: Erratum. Nature Reviews Neuroscience, 14, 442. https://doi.org/10.1038/nrn3502 First citation in articleGoogle Scholar

  • Dao, T. K., & Prevatt, F. (2006). A psychometric evaluation of the Rorschach Comprehensive System’s Perceptual Thinking Index. Journal of Personality Assessment, 86, 180–189. https://doi.org/10.1207/s15327752jpa8602_07 First citation in articleCrossref, Google Scholar

  • Exner, J. E. (1974). The Rorschach: A comprehensive system. New York, NY: Wiley. First citation in articleGoogle Scholar

  • Exner, J. E. (1986). Some Rorschach data comparing schizophrenics with borderline and schizotypal personality disorders. Journal of Personality Assessment, 50, 455–471. https://doi.org/10.1207/s15327752jpa5003_14 First citation in articleCrossref, Google Scholar

  • Exner, J. E. (1990). A Rorschach workbook for the Comprehensive System (3rd ed.). Asheville, NC: Rorschach Workshops. First citation in articleGoogle Scholar

  • Exner, J. E. (1992). R in Rorschach research: A ghost revisited. Journal of Personality Assessment, 58, 245–251. https://doi.org/10.1207/s15327752jpa5802_3 First citation in articleCrossref, Google Scholar

  • Exner, J. E. (Ed.). (1995a). Issues and methods in Rorschach research, Mahwah, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Exner, J. E. (1995b). Introduction. In J. E. Exner (Ed.), Issues and methods in Rorschach research (pp. 1–24). Mahwah, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Exner, J. E. (2000). 2000 alumni newsletter. Asheville, NC: Rorschach Workshops. First citation in articleGoogle Scholar

  • Exner, J. E. (2003). The Rorschach: A comprehensive system (4th ed.). Hoboken, NJ: Wiley. First citation in articleGoogle Scholar

  • Exner, J. E., Colligan, S. C., Boll, T. J., Stischer, B., & Hillman, L. (1996). Rorschach findings concerning closed head injury patients. Assessment, 3, 317–326. https://doi.org/10.1177/1073191196003003011 First citation in articleCrossref, Google Scholar

  • Exner, J. E., & Erdberg, P. (2005). The Rorschach: A comprehensive system: Advanced interpretation (3rd ed., Vol. 2). Hoboken, NJ: Wiley. First citation in articleGoogle Scholar

  • Exner, J. E., Wylie, J., Leura, A., & Parrill, T. (1977). Some psychological characteristics of prostitutes. Journal of Personality Assessment, 41, 474–485. https://doi.org/10.1207/s15327752jpa4105_3 First citation in articleCrossref, Google Scholar

  • Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. https://doi.org/10.3758/BF03193146 First citation in articleCrossref, Google Scholar

  • Fontan, P., & Andronikof, A. (2018, March). Complexity, falsifiability and R-PAS. In P. FontanChair, A critical review of R-PAS. Symposium at the annual meeting of the Society for Personality Assessment, Washington, DC. First citation in articleGoogle Scholar

  • Hart, L. R. (1991). The Egocentricity Index as a measure of self-esteem and egocentric personality style for inpatient adolescents. Perceptual and Motor Skills, 73 (pp. 907–914). https://doi.org/10.2466/PMS.73.7.907-914 First citation in articleCrossref, Google Scholar

  • Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage. First citation in articleCrossref, Google Scholar

  • Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B. Jr., Alper, S., … Nosek, B. A. (2019). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443–490. https://doi.org/10.1177/2515245918810225 First citation in articleGoogle Scholar

  • Meyer, G. J., Giromini, L., Viglione, D. J., Reese, J. B., & Mihura, J. L. (2015). The association of gender, ethnicity, age, and education with Rorschach scores. Assessment, 22, 46–64. https://doi.org/10.1177/1073191114544358 First citation in articleCrossref, Google Scholar

  • Meyer, G. J., Hilsenroth, M. J., Baxter, D., Exner, J. E. Jr., Fowler, J. C., Piers, C. C., & Resnick, J. (2002). An examination of interrater reliability for scoring the Rorschach comprehensive system in eight data sets. Journal of Personality Assessment, 78, 219–274. https://doi.org/10.1207/S15327752JPA7802_03 First citation in articleCrossref, Google Scholar

  • Meyer, G. J., Viglione, D. J., & Mihura, J. L. (2017). Psychometric foundations of the Rorschach Performance Assessment System© (R-PAS©). In R. E. ErardB. F. EvansEds., The Rorschach in multimethod forensic practice: Conceptual foundations and practical applications (pp. 23–91). New York, NY: Routledge. First citation in articleGoogle Scholar

  • Meyer, G. J., Viglione, D. J., Mihura, J. L., Erard, R. E., & Erdberg, P. (2011). Rorschach Performance Assessment System: Administration, coding, interpretation, and technical manual. Toledo, OH: Rorschach Performance Assessment System. First citation in articleGoogle Scholar

  • Mihura, J. L., Meyer, G. J., Bombel, G., & Dumitrascu, N. (2015). Standards, accuracy, and questions of bias in Rorschach meta-analyses: Reply to Wood, Garb, Nezworski, Lilienfeld, and Duke (2015). Psychological Bulletin, 141, 250–260. https://doi.org/10.1037/a0038445 First citation in articleCrossref, Google Scholar

  • Mihura, J. L., Meyer, G. J., Dumitrascu, N., & Bombel, G. (2013). The validity of individual Rorschach variables: Systematic reviews and meta-analyses of the comprehensive system. Psychological Bulletin, 139, 548–605. https://doi.org/10.1037/a0029406 First citation in articleCrossref, Google Scholar

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 1–8. https://doi.org/10.1126/science.aac4716 First citation in articleCrossref, Google Scholar

  • Popper, K. R. (1934/1959). The logic of scientific discovery. New York, NY: Basic Books. First citation in articleGoogle Scholar

  • Nygren, M. (2004). Rorschach Comprehensive System variables in relation to assessing dynamic capacity and ego strength for psychodynamic psychotherapy. Journal of Personality Assessment, 83, 277–292. https://doi.org/10.1207/s15327752jpa8303_10 First citation in articleCrossref, Google Scholar

  • Pineda, J. A., Giromini, L., Porcelli, P., Parolin, L., & Viglione, D. J. (2011). Mu suppression and human movement responses to the Rorschach test. NeuroReport, 22, 223–226. https://doi.org/10.1097/WNR.0b013e328344f45c First citation in articleCrossref, Google Scholar

  • Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage. First citation in articleGoogle Scholar

  • Smith, J. M. (2018, March). The validity and generalizability of CS Rorschach research. In P. FontanChair, A critical review of R-PAS. Symposium at the annual meeting of the Society for Personality Assessment, Washington, DC. First citation in articleGoogle Scholar

  • Smith, J. M., Gacono, C. B., Cunliffe, T. B., Kivisto, A. J., & Taylor, E. E. (2014). Psychodynamics in the female psychopath: A PCL-R/Rorschach investigation. Violence and Gender, 1, 176–187. https://doi.org/10.1089/vio.2014.0023 First citation in articleCrossref, Google Scholar

  • Smith, J. M., Gacono, C. B., Fontan, P., Taylor, E. E., Cunliffe, T. B., & Andronikof, A. (2018). A scientific critique of Rorschach research: Revisiting Exner’s Issues and Methods in Rorschach Research (1995). Rorschachiana, 39, 180–203. https://doi.org/10.1027/1192-5604/a000102 First citation in articleLink, Google Scholar

  • Smith, J. M., & Taylor, E. E. (2017, July). The validity and generalizability of CS Rorschach research. In P. FontanChair, The science of the Rorschach. Symposium at the XXII Congress of the International Society of the Rorschach and Projective Methods, Paris, France. First citation in articleGoogle Scholar

  • Weiner, I. B. (1991). Editor’s note: Interscorer agreement in Rorschach research. Journal of Personality Assessment, 56, 1. First citation in articleGoogle Scholar

  • Weiner, I. B. (1995). Variable selection in Rorschach research. In J. E. ExnerEd., Issues and methods in Rorschach research (pp. 73–97). Mahwah, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Wood, J. M., Garb, H. N., Nezworski, M. T., Lilienfeld, S. O., & Duke, M. C. (2015). A second look at the validity of widely used Rorschach indices: Comment on Mihura, Meyer, Dumitrascu, and Bombel (2013). Psychological Bulletin, 141, 236–249. https://doi.org/10.1037/a0036005 First citation in articleCrossref, Google Scholar