Assessing the Alternate-Form Reliability of Interview-Based and Web-Based Rorschach Responses Measuring Body Boundary Imagery and Regressive Imagery

Introduction

The World Wide Web appears to be an ideal platform that can be exploited to support psychological research and psychometric assessments of personality (Buchanan, 2000). Web-based surveys, however, require the administration of reliable and valid measurements to obtain meaningful data. Thus, this study aimed to assess the alternate-form reliability of lexical frequencies (i.e., body boundary imagery and regressive language) between interview-based and web-based Rorschach responses as measured using the Body Type Dictionary (BTD; Wilson, 2006), which is conceptually based on Fisher and Cleveland’s (1956, 1958) manual scoring system of body boundary imagery, whereas Martindale’s (1975, 1990) Regressive Imagery Dictionary gauged lexis classified as primordial and conceptual thought language. Alternate reliability refers to the degree of consistency between results obtained using different forms of the same test (Rosenthal & Rosnow, 1984).

Method

Content Analysis

The BTD and RID were applied to the interview- and web-based Rorschach responses using PROTAN computerized content-analysis software (Hogenraad, Daubies, Bestgen, & Mahau, 2003) to measure the frequencies of the lexical items in the dictionary categories. A lemmatization process reduced the words to their base forms. For example, agrees, agreed, and agreeing were all reduced to agree. Subsequently, the lexical content of the responses was matched against the predefined lexical categories of the BTD and RID.

To control for text lengths, the PROTAN content-analysis software computes two counts of lexical occurrences, that is, the density rate and the frequency rate. The density rate indicates the frequency with which distinct lexical items – i.e., types – match each dictionary category, whereas the frequency count shows how many lexical items in total – i.e., tokens – match the lexical categories (Wilson, 2008). For the purpose of this study, the density rate was used to control for verbosity and repetitive mentioning of descriptive content in the spoken Rorschach responses. The density rates for the interview-based and web-based Rorschach responses were calculated using the formula:

Results

The descriptive statistics indicated that total text lengths, t (42) = 6.616, p < .001, and variety in the vocabulary, t (42) = 6.815, p < .001, were significantly higher in the interview-based Rorschach responses compared with the web-based responses. In relation to the linguistic variables, primordial and conceptual thought language, barrier imagery, and penetration imagery were higher in the web-based Rorschach responses compared with the interview-based Rorschach responses (see Table 1 ). A paired t test showed that the frequencies of primordial thought language, t (42) = –15.589, p < .001, barrier imagery, t (42) = –4.834, p < .001, and penetration imagery, t (42) = –3.546, p < .001, were significantly higher in the web-based Rorschach responses compared with the interview-based Rorschach responses, but conceptual thought language, t (42) = 3.038, p < .01, was higher in the interview-based Rorschach responses.

Table 1. Descriptive statistic of linguistic variables in interview-based and web-based Rorschach responses

	Interview responses (N = 43)		Web responses (N = 43)
	Mean	SD	Mean	SD
Primordial thought	8.27	1.17	12.80	1.92
Conceptual thought	5.45	.89	4.43	2.37
Barrier imagery	3.09	1.22	4.45	2.35
Penetration imagery	1.43	1.23	2.34	1.97

Table 1. Descriptive statistic of linguistic variables in interview-based and web-based Rorschach responses

View as image HTML

The ICCs for the interview-based and web-based results for body boundary imagery and regressive language ranged between .36 and .72 (see Table 1 & Table 2 ). Consistent with the first hypothesis (H1), barrier and penetration imagery showed fair to good reliability. Primordial thought language showed fair reliability, but the reliability for conceptual thought language was poor, and thus the second hypothesis (H2) was partly maintained.

Table 2. ICCs and confidence intervals of linguistic variables between interview-based and web-based Rorschach responses

	ICC	95% CI
Note. ICC = Intraclass Correlation Coefficient
Barrier imagery	.716	.475, .846
Penetration imagery	.554	.176, .758
Primordial thought	.434	−.045, .693
Conceptual thought	.359	−.184, .653

Table 2. ICCs and confidence intervals of linguistic variables between interview-based and web-based Rorschach responses

View as image HTML

Discussion

This study compared the consistency of lexical frequencies – i.e., body boundary imagery and regressive language – between interview-based and web-based Rorschach responses. The obtained effect size of the ICCs indicated acceptable alternate-form reliability for barrier imagery and penetration imagery between interview-based and web-based Rorschach responses. In this sense, the web-based administration of the Rorschach inkblot test represents an acceptable alternative for measuring body boundary imagery to the traditional Rorschach interview assessment. By contrast, primordial thought language indicated fair agreement, but conceptual thought language had poor agreement between the experimental conditions. Although the reliability coefficients were below .75 and thus not acceptable for clinical administration, the linguistic variables with coefficients indicating fair to good reliability levels – i.e., barrier and penetration imagery and primordial thought language – demonstrated that the web-based version of the Rorschach inkblot test did not compromise the Rorschach measurement within scientific settings. Consistent with the contentious problems of reliability and validity associated with a projective test, the identified interpretative consistency between the interview-based and web-based conditions could be a reflection of the participants’ working memories, considering that they were exposed to the same sets of Rorschach stimuli in the interview-based and web-based test conditions. However, this consistency in interpretative responses might also be indicative of the stability of a barrier personality. In this sense, the results would provide some evidence of the stability of a barrier personality (Fisher & Cleveland, 1956, 1958).

Apart from the reliability assessment, the results of this study were consistent with research outlining the influence of the mode of questionnaire administration on the quality of participants’ responses. For example, the interview-based Rorschach responses had significantly higher text lengths than did the web-based responses. This increased text length might have been related to differences in the cognitive and social demands associated with the experimental conditions. For example, Rorschach responses with greater text lengths might be perceived as more acceptable than those with shorter responses in relation to quasiconversational interview settings (Bowling, 2005). Verbal Rorschach responses also reflect a different discourse structure that appears to include the interviewer in the participants’ underlying thought processes related to the inkblot interpretations. When comparing verbal with written Rorschach responses, it becomes apparent that verbal responses tend to include an initial interpretative and exploration phase of the presented inkblot in which the stimuli are related to real-life objects or events, followed by a more detailed description and interpretation of the stimuli and a final statement that provides a general synopsis of the interpretation or an emphasis on a specific interpretation. In particular, we recorded the repetition of words between the spoken and written Rorschach responses. The interview responses had a lexical diversity of 28.42%, whereas the web-based responses had up to 53.84% lexical diversity in relation to the overall word count. The written responses also included linguistic features associated with spoken language, such as the fillers and discursive hedges that do not typically occur in written responses – e.g., (interview-based response to inkblot no. I, participant no. 56):

Written interpretations of the web-based Rorschach responses, on the other hand, tended to focus predominately on the interpretation phase without an extensive exploration of the stimuli – e.g., (web-based response to inkblot no. I, participant no. 56):

In addition, the results showed that the web-based Rorschach responses had higher frequencies of body boundary imagery and primordial thought language but lower frequencies of conceptual thought language compared with the interview-based responses. Previous empirical research has shown that the frequencies of barrier and penetration imagery increase with the levels of primordial regression. On the basis of this finding, it can be deduced that the frequencies of primordial thought language and body boundary are context-dependent. The web-based Rorschach condition had a higher level of regression compared with the interview-based condition. Drawing on Freudian psychoanalytic theory (1900/2001a, 1926/2001b), experiences of external and internal frustration typically increase a sense of repression – so-called anticathexis – that maintains a higher level of conceptual thought and simultaneously prevents an increase in primordial regression. The reduced frequency of primordial thought within the interview-based Rorschach condition might have been related to two causes. Firstly, the use of the Internet has been associated with an increase in regressive functioning to the extent that the computer might have a functioning transference that mirrors early parent–child relations and wishful thinking as well as the tendency to resort to familiar sense-impressions when confronted with ambiguous visual cues (Laszlo, Esterman, & Zabko, 1999). The Internet might lower an individual’s unconscious defense mechanisms, resulting in higher levels of self-disclosure (Uecker, 1997). Interview-based assessments, on the other hand, might imply an increased sense of discomfort and anxiety, which could impede the ability to freely associate in relation to the presented inkblot compared with the web-based administration of the Rorschach test. In this sense, the testing situation is sensitive to the interpersonal dynamics, such as transference, that exist between the interviewer’s and the participant’s personalities and needs that inevitably influence the context of the testing situation (Masling, 1992), such as the social desirability bias in which participants might be inclined to make a good impression in the testing situation by providing socially acceptable responses (Barak & Hen, 2008; Bowling, 2005; Joinson, 1999; Rhodes et al., 2003). Thus, participants might be inclined to censor their spontaneous inkblot interpretations and produce responses that are perceived to be socially acceptable instead (Masling, 1992). The administration of a psychiatric assessment tool, in particular, might imply an interviewer bias, and the participants might feel concerned about their interpretations being judged and “psychoanalyzed” by the interviewer. This would increase the participants’ anxiety and vulnerability and thus negatively impact their engagement with the task as well as the articulation and disclosure of the inkblot interpretations. In such instances, the participants may try to produce responses that are perceived as normative and conservative rather than creative and freely associated responses (Masling, 1992).

Although the social desirability and interviewer biases would be considerably reduced in the online administration of the Rorschach, empirical research has shown that the absence of the researcher could increase the frequency of responses featuring sad content themes (Masling, 1992). This negative influence on the participants’ affect could then interact with those who are feeling depressed. In this sense, the web-based administration of the Rorschach would require ethical and clinical guidelines. In particular, an increase in responses related to body boundary and primordial regression in the web-based Rorschach responses echoes the implications associated with the advantage of online assessment. Thus, empirical research has identified elevated web-based questionnaire scores compared with offline questionnaires scores, which alludes to the question of whether Internet-mediated assessment might more accurately reflect the personalities and dispositions of participants compared with offline tests (Barak & Hen, 2008; Buchanan, 2003). To address this question, more research needs to be conducted. In particular, the results of this study are limited to the extent that there are no established norms of body boundary imagery or regressive imagery in Rorschach responses with which the scores from this study could be compared. Given that participants attended the face-to-face Rorschach interviews before responding to the web-based Rorschach tests, future research should provide a counterbalanced research design to control for a possible carry-over effect between the experimental conditions.

The web-based application of the Rorschach inkblot test represents possible advantages, such as its employment in quantitative empirical research to reach a wide and varied general population and to gather data for cross-cultural research purposes, as well as the possibility of compiling an extensive database of Rorschach responses for research and training purposes; however, there are several methodological and clinical limitations of a web-based application. In contrast to research-based face-to-face interviews that use the Rorschach inkblot test with the primary aim of producing quantifiable knowledge, such as the frequency of body boundary responses, clinical interviews that assume an ideographic framework focus on gathering knowledge that will inform the emotional relationship between the patient and the clinician. In this clinician–patient relationship, the patient is assumed to project early attachments and feelings toward a primary caregiver upon the therapist, referred to as transference, that represent the basis for uncovering unmet needs and trauma as well as other painful experiences of losses and helplessness that inform dysfunctional forms of self and other relating (Kvale & Brinkman, 2009, p. 43). In these therapeutic situations, the clinician is able to provide reassurance to the individual performing the task in addition to responding empathically to the patient’s free associations and memories that might have been triggered by the exposure to the Rorschach inkblot cards. These reassuring and empathic responses are assumed to help individuals to regulate their affective functioning (Deschenaux, Lecours, Doyon, & Briand-Malenfant, 2012) and to facilitate transformative processes, thus representing healing elements in the patient–clinician interaction (Finn, 2009; Lerner, 2005). In this sense, the Rorschach inkblot test does not represent an assessment tool only but equally represents a therapeutic intervention. Because of the mechanical and nonhuman interface of the Internet, individuals conducting a web-based Rorschach test could not be reassured that they were following the test instructions correctly, for which individuals with difficulties in functionally regulating their affective responses might experience increased anxieties and frustration (Deschenaux et al., 2012). Similarly, web-based administration of the Rorschach inkblot test would not facilitate offering the patient immediate emotional support and empathic responses to hurtful associations and traumatic memories, as would a clinician in a traditional Rorschach interview setting. In these events, individuals would feel lonely and helpless as well as experience heightened feelings of anxiety, which could trigger regressions to emotional states reminiscent of earlier traumatic experiences. Specifically, web-based Rorschach administration would also imply the replacement of the clinician as an emotionally responsive and comforting secure base with an online interface application might generate in the patient feelings of rejection, hurt, and anger (Faber & Metzger, 2009). Conversely, it has been shown that interactive online psychotherapy would produce strong therapeutic relationships and effective outcomes in relation to a wide range of psychiatric disorders, such as depression, eating disorders, anxiety disorders, and posttraumatic stress disorder, to mention a few (Knaevelsrud & Maercker, 2006, 2007). Given these positive outcomes, the administration of a web-based Rorschach inkblot test could be used within the framework of telemedicine by combining face-to-face psychotherapy sessions with an online-based service. Such a service would be useful for patients who have access to the Internet and feel comfortable to participate in e-therapy; however, as pointed out by Wade (2010), patients with literary problems and insecurities would be excluded from an online-based mental health service. The Rorschach inkblot test has been also employed to assess changes following psychotherapy (Campo, 2009; Grønnerød, 2004) and such a follow-up application could be, for example, applied as an ongoing assessment in relation to individuals attending alternative psychotherapy forms to explore changes in body boundary awareness (Quartier, Antonietti, Frank, & Iglesias, 2013), or as an optional supportive assessment tool for patients who are reintegrating into their social community.

Given the vast availability of information related to the Rorschach inkblot test that can be accessed easily and effortlessly, including the inkblot cards on the online encyclopedia Wikipedia (Nashat, 2010; Schulz and Loving, 2012), it has to be taken into consideration that web-based administration of the Rorschach inkblot test might increase the possibility of malingering and providing fake responses. For example, a patient taking the Rorschach inkblot test at home on a personal computer could request the help of a friend or family member to provide responses, and there is also the risk of sourcing responses that are classified as “normal.” Although there are disputes regarding whether computer software would be able to detect fake responses (Cohen, 1990; Kahn, Fox, & Rhode, 1988), web-based administration of the Rorschach inkblot test that facilitates the straightforward processing of data into a statistical software program would certainly enable the comparison of repeated responses provided by the same patient in web-based test administration to detect unusual protocols, as well as compared with the responses that were provided in initial face-to-face interviews. Taking specifically into consideration that this study did not make use of an inquiry phase that would have allowed participants to produce localized responses, the employment of customized web-based Rorschach administration software that includes a comment box or a grid superimposed on the inkblot cards would also enable patients to provide localized and detailed responses that offer more information about the formal quality compared with generalized whole or vague responses. Such localized responses might also relate to greater regression along the primordial thought continuum and thus provide insight into repressed tendencies that inform the defense mechanisms (Schafer, 1954). The exploration of localized responses and the formal quality would also make researchers and clinicians aware of an individual’s personality tendencies and cognitive processes that have been associated with the specific types of responses in previous research (Exner, 2003). Furthermore, based on empirical evidence that has demonstrated that the inclusion of details and specific information that require increased cognitive effort, and thus increase the chance of self-contradiction in verbal behavior, has been typically associated with truthfulness (Hancock, Curry, Goorha, & Woodworth, 2008, p. 8; see also Porter & Yuille, 1996), the occurrence of contradictory localized and detailed responses could be interpreted as a sign of deception and faking. Given the importance of reaction time in providing a response upon exposure to a card, in particular, customized Rorschach web-based administration software could also include an application to record response time as an extra feature to measure patients’ cognitive processing effort in addition to identifying the defense mechanisms associated with short or delayed reaction times; this software could thus provide a testing situation that assumes some equivalence to the advantages that are only accessible to the clinician in face-to-face Rorschach interviews.

With regard to this study, however, these initial and tentative results were satisfactory within the constraints of its limitations, indicating an acceptable level of equivalence between barrier and, to a limited extent, penetration imagery and primordial thought language among the web-based and interview-based Rorschach inkblot responses.