Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

Construct Validity and Measurement Invariance of the eHEALS in a Diverse US High School Sample

Published Online:December 21, 2023https://doi.org/10.1027/1864-1105/a000399

Abstract

Abstract: Although the eHealth Literacy Scale (eHEALS) is a widely used measure of eHealth literacy (eHL), it has not been validated in a diverse sample of US adolescents. This study assessed the construct validity and item and measurement invariance of the eHEALS using both Rasch analyses and classical test theory (CTT). Data were collected from adolescents at a US Northeastern high school (N = 355, 64.9% were 16–18-years old, 54.8% male, 76.4% from a racially minoritized group, and 50.5% Latinx). Adolescents completed a survey that included demographic variables (i.e., age in years, gender, and ethnicity) and the eHEALS. Using Rasch and CTT methodologies, the dimensionality of the eHEALS and item and measurement invariance across gender, age, and ethnicity were assessed. The unidimensional structure of the eHEALS as well as item and measurement invariance across gender, age, and ethnicity were supported. Rasch analyses also suggested that responses to eHEALS items can be summed to determine eHL abilities, although there was a ceiling effect. The eHEALS is a unidimensional measure of eHL that may provide information in health education settings about adolescents’ eHL skills. It performs similarly across male and female as well as older and younger adolescents. However, given the ceiling effect and changes in adolescents’ Internet use since its development, future studies should revise the eHEALS to better capture adolescents’ eHL abilities. Future studies should also examine item and measurement invariance by age (in years), race, family income, and longitudinally to determine the usefulness of the measure across groups and time.

References

  • Arbuckle, J. L. (2014). Amos (Version 26) [Computer software]. IBM SPSS. First citation in articleGoogle Scholar

  • Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, Article 149. https://doi.org/10.3389/fpubh.2018.00149 First citation in articleCrossref, Google Scholar

  • Bond, T. G. (2010). Rasch model. In I. B. WeinerW. E. CraigheadEds., The Corsini encyclopedia of psychology. John Wiley & Sons. https://doi.org/10.1002/9780470479216.corpsy0773 First citation in articleCrossref, Google Scholar

  • Boone, W. J. (2016). Rasch analysis for instrument development: Why, when, and how? CBE Life Science Education, Bethesda, 15(4). https://doi.org/10.1187/cbe.16-04-0148 First citation in articleCrossref, Google Scholar

  • Byrne, B. M. (2004). Testing for multigroup invariance using AMOS graphics: A road less traveled. Structural Equation Modeling: A Multidisciplinary Journal, 11(2), 272–300. https://doi.org/10.1207/s15328007sem1102_8 First citation in articleCrossref, Google Scholar

  • Chang, F.-C., Chiu, C.-H., Chen, P.-H., Miao, N.-F., Lee, C.-M., Chiang, J.-T., & Pan, Y.-C. (2015). Relationship between parental and adolescent ehealth literacy and online health information seeking in Taiwan. Cyberpsychology, Behavior and Social Networking, 18(10), 618–624. https://doi.org/10.1089/cyber.2015.0110 First citation in articleCrossref, Google Scholar

  • Chesser, A., Burke, A., Reyes, J., & Rohrberg, T. (2016). Navigating the digital divide: A systematic review of eHealth literacy in underserved populations in the United States. Informatics for Health & Social Care, 41(1), 1–19. https://doi.org/10.3109/17538157.2014.948171 First citation in articleCrossref, Google Scholar

  • Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5 First citation in articleCrossref, Google Scholar

  • Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. The Journal of Applied Psychology, 78(1), 98–104. https://doi.org/10.1037/0021-9010.78.1.98 First citation in articleCrossref, Google Scholar

  • DeVellis, R. F. (2006). Classical test theory. Medical Care, 44(11 Suppl 3), S50–S59. https://doi.org/10.1097/01.mlr.0000245426.10853.30 First citation in articleCrossref, Google Scholar

  • Fergie, G., Hunt, K., & Hilton, S. (2013). What young people want from health-related online resources: A focus group study. Journal of Youth Studies, 16(5), 579–596. https://doi.org/10.1080/13676261.2012.744811 First citation in articleCrossref, Google Scholar

  • Fleary, S. A., & Joseph, P. (2020). Adolescents’ health literacy and decision-making: A qualitative study. American Journal of Health Behavior, 44(4), 392–408. https://doi.org/10.5993/AJHB.44.4.3 First citation in articleCrossref, Google Scholar

  • Fleary, S. A., Joseph, P., & Pappagianopoulos, J. E. (2018). Adolescent health literacy and health behaviors: A systematic review. Journal of Adolescence, 62, 116–127. https://doi.org/10.1016/j.adolescence.2017.11.010 First citation in articleCrossref, Google Scholar

  • Furr, M. (2011). Evaluating psychometric properties: Dimensionality and reliability. In M. FurrEd., Scale construction and psychometrics for social and personality psychology (pp. 25–51). SAGE Publications. First citation in articleCrossref, Google Scholar

  • Gazibara, T., Cakic, J., Cakic, M., Pekmezovic, T., & Grgurevic, A. (2019). eHealth and adolescents in Serbia: Psychometric properties of eHEALS questionnaire and contributing factors to better online health literacy. Health Promotion International, 34(4), 770–778. https://doi.org/10.1093/heapro/day028 First citation in articleCrossref, Google Scholar

  • Ghaddar, S. F., Valerio, M. A., Garcia, C. M., & Hansen, L. (2012). Adolescent health literacy: The importance of credible sources for online health information. The Journal of School Health, 82(1), 28–36. https://doi.org/10.1111/j.1746-1561.2011.00664.x First citation in articleCrossref, Google Scholar

  • Giger, J. T., Barnhart, S., Feltner, F., Slone, M., Lawler, M. J., Windsor, L., & Windsor, A. (2021). Validating the eHealth Literacy Scale in rural adolescents. The Journal of Rural Health, 37(3), 504–516. https://doi.org/10.1111/jrh.12509 First citation in articleCrossref, Google Scholar

  • Gulec, H., & Smahel, D. (2022). Individual and parental factors of adolescents’ mhealth app use: Nationally representative cross-sectional study. JMIR MHealth and UHealth, 10(12), Article e40340. https://doi.org/10.2196/40340 First citation in articleCrossref, Google Scholar

  • Holch, P., & Marwood, J. R. (2020). EHealth literacy in UK teenagers and young adults: Exploration of predictors and factor structure of the eHealth Literacy Scale (eHEALS). JMIR Formative Research, 4(9), Article e14450. https://doi.org/10.2196/14450 First citation in articleCrossref, Google Scholar

  • Hsu, W., Chiang, C., & Yang, S. (2014). The effect of individual factors on health behaviors among college students: The mediating effects of eHealth literacy. Journal of Medical Internet Research, 16(12), Article e287. https://doi.org/10.2196/jmir.3542 First citation in articleCrossref, Google Scholar

  • Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118 First citation in articleCrossref, Google Scholar

  • IBM Corp. (2020). SPSS statistics for Windows (Version 27) [Computer software]. IBM. First citation in articleGoogle Scholar

  • Karnoe, A., & Kayser, L. (2015). How is eHealth literacy measured and what do the measurements tell us? A systematic review. Knowledge Management & E-Learning: An International Journal, 7(4), 576–600. https://doi.org/10.34105/j.kmel.2015.07.038 First citation in articleCrossref, Google Scholar

  • Kim, K., Shin, S., Kim, S., & Lee, E. (2023). The relation between eHealth Literacy and health-related behaviors: Systematic review and meta-analysis. Journal of Medical Internet Research, 25, Article e40778. https://doi.org/10.2196/40778 First citation in articleCrossref, Google Scholar

  • Kimberlin, C. L., & Winterstein, A. G. (2008). Validity and reliability of measurement instruments used in research. American Journal of Health-System Pharmacy, 65(23), 2276–2284. https://doi.org/10.2146/ajhp070364 First citation in articleCrossref, Google Scholar

  • Korkmaz Aslan, G., Kartal, A., Turan, T., Taşdemir Yiğitoğlu, G., & Kocakabak, C. (2021). Association of electronic health literacy with health-promoting behaviours in adolescents. International Journal of Nursing Practice, 27(2), Article e12921. https://doi.org/10.1111/ijn.12921 First citation in articleCrossref, Google Scholar

  • Kyriazos, T. A. (2018). Applied psychometrics: Sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology, 09(08), 2207–2230. https://doi.org/10.4236/psych.2018.98126 First citation in articleCrossref, Google Scholar

  • Lee, J., Lee, E.-H., & Chae, D. (2021). eHealth literacy instruments: Systematic review of measurement properties. Journal of Medical Internet Research, 23(11), Article e30644. https://doi.org/10.2196/30644 First citation in articleCrossref, Google Scholar

  • Lehtimaki, S., Martic, J., Wahl, B., Foster, K. T., & Schwalbe, N. (2021). Evidence on digital mental health interventions for adolescents and young people: Systematic overview. JMIR Mental Health, 8(4), Article e25847. https://doi.org/10.2196/25847 First citation in articleCrossref, Google Scholar

  • Linacre, J. M. (1994). Many-facet Rasch measurement (2nd ed.). MESA. First citation in articleGoogle Scholar

  • Linacre, J. M. (2002a). What do infit and outfit, mean-square and standardized mean. Rasch Measurement Transactions, 16(2), Article 878. First citation in articleGoogle Scholar

  • Linacre, J. M. (2002b). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85–106. https://www.ncbi.nlm.nih.gov/pubmed/11997586 First citation in articleGoogle Scholar

  • Linacre, J. M. (2013). DIF sample size nomogram. Rasch Measurement Transactions, 26(4), 1392–1402. https://www.rasch.org/rmt/rmt264.pdf First citation in articleGoogle Scholar

  • Linacre, M. J. (2021). Winsteps® Rasch measurement computer program (Version 5.1.1) [Computer software]. Winsteps.com. First citation in articleGoogle Scholar

  • Magno, C. (2009). Demonstrating the difference between classical test theory and item response theory using derived test data. The International Journal of Educational and Psychological Assessment, 1(1), 1–11. First citation in articleGoogle Scholar

  • Maitz, E., Maitz, K., Sendlhofer, G., Wolfsberger, C., Mautner, S., Kamolz, L. P., & Gasteiger-Klicpera, B. (2020). Internet-based health information-seeking behavior of students aged 12 to 14 years: Mixed methods study. Journal of Medical Internet Research, 22(5), Article e16281. https://doi.org/10.2196/16281 First citation in articleCrossref, Google Scholar

  • Marsall, M., Engelmann, G., Skoda, E.-M., Teufel, M., & Bäuerle, A. (2022). Measuring electronic health literacy: Development, validation, and test of measurement invariance of a revised German version of the eHealth Literacy Scale. Journal of Medical Internet Research, 24(2), Article e28252. https://doi.org/10.2196/28252 First citation in articleCrossref, Google Scholar

  • Massachusetts Department of Elementary and Secondary Education. (n.d.). National School Lunch Program. https://www.doe.mass.edu/cnp/nprograms/nslp.html First citation in articleGoogle Scholar

  • Nguyen, J., Moorhouse, M., Curbow, B., Christie, J., Walsh-Childers, K., & Islam, S. (2016). Construct validity of the eHealth Literacy Scale (eHEALS) among two adult populations: A Rasch analysis. JMIR Public Health and Surveillance, 2(1), Article e24. https://doi.org/10.2196/publichealth.4967 First citation in articleCrossref, Google Scholar

  • Nguyen, T. H., Paasche-Orlow, M. K., & McCormack, L. A. (2017). The state of the science of health literacy measurement. Studies in Health Technology and Informatics, 240, 17–33. https://doi.org/10.1371/journal.pone.0109110 First citation in articleCrossref, Google Scholar

  • Norman, C. (2011). eHealth literacy 2.0: Problems and opportunities with an evolving concept. Journal of Medical Internet Research, 13(4), Article e125. https://doi.org/10.2196/jmir.2035 First citation in articleCrossref, Google Scholar

  • Norman, C. D., & Skinner, H. A. (2006a). eHealth Literacy: Essential skills for consumer health in a networked world. Journal of Medical Internet Research, 8(2), Article e9. https://doi.org/10.2196/jmir.8.2.e9 First citation in articleCrossref, Google Scholar

  • Norman, C. D., & Skinner, H. A. (2006b). eHEALS: The eHealth Literacy Scale. Journal of Medical Internet Research, 8(4), Article e27. https://doi.org/10.2196/jmir.8.4.e27 First citation in articleCrossref, Google Scholar

  • Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw Hill Book Company. First citation in articleGoogle Scholar

  • Okoniewski, A. E., Lee, Y. J., Rodriguez, M., Schnall, R., & Low, A. F. H. (2014). Health information seeking behaviors of ethnically diverse adolescents. Journal of Immigrant and Minority Health, 16(4), 652–660. https://doi.org/10.1007/s10903-013-9803-y First citation in articleCrossref, Google Scholar

  • Paige, S. R., Krieger, J. L., Stellefson, M., & Alber, J. M. (2017). eHealth literacy in chronic disease patients: An item response theory analysis of the eHealth Literacy Scale (eHEALS). Patient Education and Counseling, 100(2), 320–326. https://doi.org/10.1016/j.pec.2016.09.008 First citation in articleCrossref, Google Scholar

  • Paige, S. R., Miller, M. D., Krieger, J. L., Stellefson, M., & Cheong, J. (2018). Electronic health literacy across the lifespan: Measurement invariance study. Journal of Medical Internet Research, 20(7), Article e10434. https://doi.org/10.2196/10434 First citation in articleCrossref, Google Scholar

  • Peterson, C. H., Peterson, N. A., & Powell, K. G. (2017). Cognitive interviewing for item development: Validity evidence based on content and response processes. Measurement and Evaluation in Counseling and Development, 50(4), 217–223. https://doi.org/10.1080/07481756.2017.1339564 First citation in articleCrossref, Google Scholar

  • Prieto, L., Alonso, J., & Lamarca, R. (2003). Classical test theory versus Rasch analysis for Quality of Life Questionnaire reduction. Health and Quality of Life Outcomes, 1, Article 27. https://doi.org/10.1186/1477-7525-1-27 First citation in articleCrossref, Google Scholar

  • Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004 First citation in articleCrossref, Google Scholar

  • Quinn, S., Bond, R., & Nugent, C. (2017). Quantifying health literacy and eHealth literacy using existing instruments and browser-based software for tracking online health information seeking behavior. Computers in Human Behavior, 69, 256–267. https://doi.org/10.1016/j.chb.2016.12.032 First citation in articleCrossref, Google Scholar

  • Richtering, S. S., Morris, R., Soh, S.-E., Barker, A., Bampi, F., Neubeck, L., Coorey, G., Mulley, J., Chalmers, J., Usherwood, T., Peiris, D., Chow, C. K., & Redfern, J. (2017). Examination of an eHealth Literacy Scale and a Health Literacy Scale in a population with moderate to high cardiovascular risk: Rasch analyses. PLoS One, 12(4), Article e0175372. https://doi.org/10.1371/journal.pone.0175372 First citation in articleCrossref, Google Scholar

  • Rikard, R. V., Thompson, M. S., McKinney, J., & Beauchamp, A. (2016). Examining health literacy disparities in the United States: A third look at the National Assessment of Adult Literacy (NAAL). BMC Public Health, 16(1), Article 975. https://doi.org/10.1186/s12889-016-3621-9 First citation in articleCrossref, Google Scholar

  • Rose, T., Barker, M., Maria Jacob, C., Morrison, L., Lawrence, W., Strömmer, S., Vogel, C., Woods-Townsend, K., Farrell, D., Inskip, H., & Baird, J. (2017). A systematic review of digital interventions for improving the diet and physical activity behaviors of adolescents. The Journal of Adolescent Health, 61(6), 669–677. https://doi.org/10.1016/j.jadohealth.2017.05.024 First citation in articleCrossref, Google Scholar

  • Soellner, R., Huber, S., & Reder, M. (2014). The concept of eHealth literacy and its measurement: German translation of the eHEALS. Journal of Media Psychology: Theories, Methods, and Applications, 26(1), 29–38. https://psycnet.apa.org/journals/jmp/26/1/29/ First citation in articleLink, Google Scholar

  • Sudbury-Riley, L., FitzPatrick, M., & Schulz, P. J. (2017). Exploring the measurement properties of the eHealth Literacy Scale (eHEALS) among Baby Boomers: A multinational test of measurement invariance. Journal of Medical Internet Research, 19(2), Article e53. https://doi.org/10.2196/jmir.5998 First citation in articleCrossref, Google Scholar

  • Terwee, C. B., Bot, S. D. M., de Boer, M. R., van der Windt, D. A. W. M., Knol, D. L., Dekker, J., Bouter, L. M., & de Vet, H. C. W. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60(1), 34–42. https://doi.org/10.1016/j.jclinepi.2006.03.012 First citation in articleCrossref, Google Scholar

  • van de Schoot, R., Lugtig, P., & Hox, J. (2012). A checklist for testing measurement invariance. The European Journal of Developmental Psychology, 9(4), 486–492. https://doi.org/10.1080/17405629.2012.686740 First citation in articleCrossref, Google Scholar

  • Wartella, E., Rideout, V., Montague, H., Beaudoin-Ryan, L., & Lauricella, A. (2016). Teens, health and technology: A national survey. Media and Communication, 4(3), 13–23. https://doi.org/10.17645/mac.v4i3.515 First citation in articleCrossref, Google Scholar

  • Wolfe, E. W., & Smith, E. V. Jr. (2007a). Instrument development tools and activities for measure validation using Rasch models: Part 1 – Instrument development tools. Journal of Applied Measurement, 8(1), 97–123. https://psycnet.apa.org/record/2008-02262-006 First citation in articleGoogle Scholar

  • Wolfe, E. W., & Smith, E. V. Jr. (2007b). Instrument development tools and activities for measure validation using Rasch models: Part II–validation activities. Journal of Applied Measurement, 8(2), 204–234. https://www.ncbi.nlm.nih.gov/pubmed/17440262 First citation in articleGoogle Scholar

  • Wright, B., & Panchapakesan, N. (1969). A procedure for sample-free item analysis. Educational and Psychological Measurement, 29(1), 23–48. https://doi.org/10.1177/001316446902900102 First citation in articleCrossref, Google Scholar

  • Yang, S. C., Luo, Y. F., & Chiang, C.-H. (2019). Electronic health literacy and dietary behaviors in Taiwanese college students: Cross-sectional study. Journal of Medical Internet Research, 21(11), Article e13140. https://doi.org/10.2196/13140 First citation in articleCrossref, Google Scholar