Detecting Random Responses in a Personality Scale Using IRT-Based Person-Fit Indices
Abstract
Abstract. Random response is a very common aberrant response behavior in personality tests and may negatively affect the reliability, validity, or other analytical aspects of psychological assessment. Typically, researchers use a single person-fit index to identify random responses. This study recommends a three-step person-fit analysis procedure. Unlike the typical single person-fit methods, the three-step procedure identifies both global misfit and local misfit individuals using different person-fit indices. This procedure was able to identify more local misfit individuals than single-index method, and a graphical method was used to visualize those particular items in which random response behaviors appear. This method may be useful to researchers in that it will provide them with more information about response behaviors, allowing better evaluation of scale administration and development of more plausible explanations. Real data were used in this study instead of simulation data. In order to create real random responses, an experimental test administration was designed. Four different random response samples were produced using this experimental system.
References
(2009). A parametric cumulative sum statistic for person fit. Applied Psychological Measurement, 33, 391–410.
(2011). Detection of answer copying based on the structure of a High-Stakes Test. Applied Psychological Measurement, 35, 495–517.
(1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345.
(2003). Detection of back random responding: Effectiveness of MMPI-2 and personality assessment inventory validity indices. Psychological Assessment, 15, 223–234.
(2014). Statistic lz-based person-fit methods for noncognitive multiscale measures. Applied Psychological Measurement, 38, 122–136.
(2010). Screening for atypical suicide risk with person fit statistics among people presenting to alcohol and other drug treatment. Drug and Alcohol Dependence, 106, 92–100.
(2009). Person-fit: Relationship with four personality tests in mathematics. Research Papers in Education, 24, 115–123.
(1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59–79.
(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67–86.
(2008). Nonparametric person-fit analysis of Polytomous Item Scores. Applied Psychological Measurement, 32, 224–247.
(2009). Detection and diagnosis of person misfit from patterns of summed Polytomous Item Scores. Applied Psychological Measurement, 33, 599–619.
(2005). Global, local, and graphical person-fit analysis using person-response functions. Psychological Methods, 10, 101–119.
(2001). The measurement of neuroticism using MMQ, MPI, EPI and EPQ items: A psychometric analysis based on item response theory. Personality and Individual Differences, 30, 641–656.
(2003). The accuracy of the E, N and P trait estimates: An empirical study using the EPQ-R. Personality and Individual Differences, 34, 665–679.
(2012). Assessing inconsistent responding in E and N measures: An application of person-fit analysis in personality. Personality and Individual Differences, 52, 718–722.
(2015).
Assessing person fit in typical-response measures . In S. P. ReiseD. A. RevickiEds., Handbook of item response theory modeling: Applications to typical performance assessment (pp. 128–155). New York, NY: Routledge/Taylor & Francis Group.(2013). A structural model-based optimal person-fit procedure for identifying faking. Educational and Psychological Measurement, 73, 173–190.
(2001). Detecting dissimulation in personality test scores: A comparison between person-fit indices and detection scales. Educational and Psychological Measurement, 61, 997–1012.
(2010). Acquiescence as a source of bias and model and person misfit: A theoretical and empirical analysis. British Journal of Mathematical and Statistical Psychology, 63, 427–448.
(2016). A comprehensive regression-based approach for identifying sources of person misfit in typical-response measures. Educational and Psychological Measurement, 76, 470–486.
(2000). An item response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology, 78, 350–365.
(2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277–298.
(2009). Investigating faking using a multilevel logistic regression approach to measuring person fit. Organizational Research Methods, 12, 296–319.
(2014). Detecting aberrant responding on unidimensional pairwise preference tests: An application of lz based on the Zinnes-Griggs ideal point IRT model. Applied Psychological Measurement, 38, 391–403.
(1988). Optimal appropriateness measurement. Psychometrika, 53, 161–176.
(2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
(1996). Person fit research: An introduction. Applied Measurement in Education, 9, 3–8.
(2002). Outlier detection in high-stakes certification testing. Journal of Educational Measurement, 39, 219–233.
(1997). Trait level estimation for nonfitting response vectors. Applied Psychological Measurement, 21, 321–336.
(2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135.
(1998). A comparison of the Person response function and the lz person-fit statistic. Applied Psychological Measurement, 22, 53–69.
(1990). A comparison of item- and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement, 14, 127–137.
(1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213–229.
(1996). Assessing Person-Fit on measures of typical performance. Applied Measurement in Education, 9, 9–26.
(2003). A discussion of modern versus traditional psychometrics as applied to personality assessment scales. Journal of Personality Assessment, 81, 93–103.
(1990). Fitting the two-parameter model to personality data. Applied Psychological Measurement, 14, 45–58.
(1993). Traitedness and the assessment of response pattern scalability. Journal of Personality and Social Psychology, 65, 143
(2013). Examining faking on personality inventories using unfolding item response theory models. Journal of Personality Assessment, 95, 207–216.
(1999). Correlates of person fit and effect of person fit on test validity. Applied Psychological Measurement, 23, 41–53.
(2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66, 331–342.
(2011). Accuracy of person-fit statistics: A Monte Carlo study of the influence of aberrance rates. Applied Psychological Measurement, 35, 419–432.
(1984). Caution indices based on Item Response Theory. Psychometrika, 49, 95–110.
(2012). A CUSUM to detect person misfit: A discussion and some alternatives for existing procedures. Applied Psychological Measurement, 36, 420–442.
(2000).
Detection of person misfit in adaptive testing using statistical process control techniques . In W. J. van der LindenC. A. W. GlasEds., Computerized adaptive testing: Theory and practice (pp. 201–219). Boston, MA: Kluwer-Nijhoff.(2001). CUSUM-based person-fit statistics for adaptive testing. Journal of Educational and Behavioral Statistics, 26, 199–217.
(1996). Exploring nonlinear models in personality assessment: Development and preliminary validation of a negative emotionality scale. Journal of Personality, 64, 545–576.
