Do the Readability and Average Item Length of Personality Scales Affect Their Reliability?
Some Meta-Analytic Answers
Abstract
Using meta-analytic data, we investigated three attributes of personality scales – their readability, the average number of words per item, and whether adjectives or statements were used – to determine the effects that these qualities have on scale reliability (Cronbach’s α). From a large pool of studies examined in three prior meta-analyses, we extracted the relevant data for seven personality traits measured in 167 independent samples (N = 55,593), and obtained measures of the traits’ readability, the average number of words per item, and whether adjectives or statements were used as the item format. We found strong and consistent evidence that using fewer words per item produced higher scale reliabilities, and there was also evidence in favor of using adjectives, as opposed to statements, as the item format when measuring broad traits. In contrast, the results for readability were inconsistent and mostly nonsignificant. These results have practical implications for scale developers as well as researchers, and they raise the possibility that shorter and more “de-contextualized” items do a better job of assessing personality traits.
References
(1997). Psychological testing (7th ed.). Upper Saddle River, NJ: Prentice Hall.
(1981). Assessing the readability levels of self-report assertion inventories. Journal of Consulting and Clinical Psychology, 49, 142–144.
(1998). Los Cinco Grandes across cultures and ethnic groups: Multitrait-multimethod analyses of the Big Five in Spanish and English. Journal of Personality and Social Psychology, 75, 729–750.
(1973). Methodologies for determining reading requirements of military occupational specialties. Alexandria, VA: Human Resources Research Organization (Report No. 73–5).
(1983). Readability and prose comprehension: Continuities and discontinuities. In , Understanding reading comprehension: Cognition, language, and the structure of prose (pp. 233–246). Newark, DE: International Reading Association.
(1995). Readability revisited: The new Dale-Chall readability formula. Cambridge, MA: Brookline Books.
(2011). The Five-Factor Model of personality traits and organizational citizenship behaviors: A meta-analysis. Journal of Applied Psychology, 96, 1140–1166.
(1984). Research design effects on the reliability of rating scales: A meta-analysis. Journal of Marketing Research, 21, 360–375.
(1992). Revised NEO Personality Inventory (NEO PI-R) and NEO Five-Factor Inventory (NEO-FFI): Professional Manual. Odessa, FL: Psychological Assessment Resources.
(1980). The optimal number of response alternatives for a scale: A review. Journal of Marketing Research, 17, 407–422.
(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.
(1948). A formula for predicting readability: Instructions. Educational Research Bulletin, 27, 37–54.
(1949). The concept of readability. Elementary English, 26, 19–26.
(2006). Higher-order factors of the Big Five in a multi-informant sample. Journal of Personality and Social Psychology, 91, 1138–1151.
(2004). The principles of readability. Retrieved from www.impactinformation.com/impactinfo/readability02.pdf
(1943). Marks of readability style: A study in adult education. New York, NY: Columbia University Press.
(1948). A new readability yardstick. Journal of Applied Psychology, 32, 221–233.
(1950). How to measure readability: Measuring the level of abstraction. Journal of Applied Psychology, 34, 384–390.
(1968). A readability formula that saves time. Journal of Reading, 11, 513–516.
(1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In , Personality psychology in Europe, Vol. 7, (pp. 7–28). Tilburg, The Netherlands: Tilburg University Press.
(1985). The prediction of semantic consistency in self-descriptions: Characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs. Journal of Personality and Social Psychology, 48, 82–98.
(2006). The reading grade level of common measures in child and adolescent clinical psychology. Psychological Assessment, 18, 346–352.
(1991). The Big Five Inventory – Versions 4a and 54. Berkeley, CA: University of California at Berkeley, Institute of Personality and Social Research.
(2008). Paradigm shift to the integrative Big Five Trait taxonomy: History, measurement, and conceptual issues. In , Handbook of personality: Theory and research (3rd ed.). (pp. 114–158). New York, NY: Guilford Press.
(1999). The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In , Handbook of personality: Theory and research (2nd ed.). (pp. 102–138). New York, NY: Guilford Press.
(2001). Relationship of core self-evaluations traits – self-esteem, generalized self-efficacy, locus of control, and emotional stability – with job satisfaction and job performance: A meta-analysis. Journal of Applied Psychology, 86, 80–92.
(1963). The measurement of readability. Ames, IA: Iowa State University Press.
(1975). Derivation of new readability formula for navy enlisted personnel. Millington, TN: Navy Research Branch.
(1996). The use of readability formulas in health care. Psychology, Health & Medicine, 1, 7–28.
(2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
(1990). Relationships among readability, reliability, and validity in a self-report adjective check list. Psychological Assessment, 2, 256–261.
(2005). The NEO-PI-3: A more readable revised NEO Personality Inventory. Journal of Personality Assessment, 84, 261–270.
(2009). Readability of self-report measures of depression and anxiety. Journal of Consulting and Clinical Psychology, 77, 1100–1112.
(1969). SMOG grading – a new readability formula. Journal of Reading, 22, 639–646.
. (2012). The seven formulas application (Version 7.8) [Computer software]. Dallas, TX: Author.
. (1998). Communicating with patients who have limited literacy skills. Journal of Family Practice, 46, 168–175.
(1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
(2004). Language, reading, and readability formulas: Implications for developing and adapting tests. International Journal of Testing, 4, 239–252.
(1994). A meta-analysis of Cronbach’s coefficient alpha. Journal of Marketing, 21, 381–391.
(1989). Readability of three adult personality tests: Basic Personality Inventory, Jackson Personality Inventory, and Personality Research Form-E. Journal of Personality Assessment, 53, 180–183.
(2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306–322.
(1994). Mini-markers: A brief version of Goldberg’s unipolar Big-Five markers. Journal of Personality Assessment, 63, 506–516.
(1993). Readability of adult psychopathology inventories. Psychological Assessment, 5, 384–386.
(1994). Readability of normal personality inventories. Journal of Personality Assessment, 62, 95–101.
(1999). How the questions shape the answers. American Psychologist, 54, 93–105.
(2003). Low literacy and mental illness in a nationally representative sample. Journal of Nervous & Mental Disease, 191, 549–552.
(1953). A new readability formula for primary-grade reading materials. Elementary School Journal, 53, 410–413.
(1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101.
(2007). Meta-analysis of the Big Five and academic success at university. Zeitschrift für Psychologie, 215, 132–151.
(2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics, 30, 261–293.
(2005). Meta-analysis macros for SAS, SPSS, and Stata. Retrieved, December, 28, 2012, from mason.gmu.edu/~dwilsonb/ma.html
