Factor Structure of the eHEALS

Models Underlying eHealth Literacy

Four models underlying the concept of eHealth literacy are important to the discussion about the dimensionality of the eHEALS. The Lily Model (Norman & Skinner, 2006b) conceptualizes eHealth literacy as seeking, finding, understanding, and appraising health information as well as applying knowledge and addressing health problems. The model separates six literacies or skills (traditional, computer, information, health, media, and science) whose area of overlap constitutes eHealth literacy. Nevertheless, Norman and Skinner (2006a) derived a unidimensional scale from it. This already points to potential difficulties in determining the factor structure of the eHEALS: Scale and model imply different factor structures. To reflect the conceptual model, the eHEALS would also have to be conceptualized multidimensionally (Lee et al., 2021).

The Model of Health Competence is the second approach and is based on the results of expert concept mapping (Soellner et al., 2010; Soellner et al., 2017). Seven dimensions have been differentiated (communication and cooperation, self-perception, self-regulation, proactive approach to health, healthcare system knowledge and acting, information-seeking, information appraisal). A theoretical overlap with the eHEALS was established for two dimensions: information-seeking and information appraisal (Soellner et al., 2014). Information-seeking allows one to access and use health-related information through different sources, and information appraisal allows adequate interpretation of health-related information (Soellner et al., 2014). Based on this model, we postulated a 2-factor structure of the eHEALS.

Informed choice is another important concept related to health literacy, especially regarding health decisions. Three dimensions are defined: knowledge, attitude, and uptake (Marteau et al., 2001). The knowledge dimension overlaps considerably with information-seeking and information appraisal, while the uptake dimension implies a third dimension of using information for health decisions within the eHEALS. According to the theoretical underpinnings of the informed choice approach, we hypothesized a 3-factor model for the eHEALS in the context of using an online decision aid (Reder et al., 2019).

The Integrated Conceptual Model of Sørensen et al. (2012) is based on a systematic review of 17 definitions of health literacy and 12 frameworks of health literacy. It proposes the following ability dimensions, all of which target the maintenance or improvement of health: (1) access health information, (2) understand health information, (3) appraise health information, and (4) apply (use) the information to make a decision to maintain and improve health. The authors criticize that available measures of health literacy fail to cover all aspects of the concept and thus call for the development of measures that better reflect the conceptual models for health literacy (Sørensen et al., 2012). The fourth dimension of this model approximates making informed decisions (Sørensen et al., 2012) and thus mirrors Marteau et al.’s (2001) concept of informed choice. After connecting the Sørensen et al. (2012) model to the 3-factor model of the eHEALS based on the informed choice approach, Dimension 1 corresponds to information-seeking, Dimension 3 to information appraisal, and Dimension 4 to information use.

Factor Structure of the eHEALS

The theoretical models and the number of postulated dimensions vary, as do the methods applied for data analyses. Some researchers apply only exploratory variable-reduction techniques (exploratory factor analysis, EFA, and principal component analysis, PCA; in the following, we refer to these exploratory techniques as EFA, although EFA and PCA are sometimes distinguished), whereas others combine exploratory and confirmatory techniques or test a hypothesized model through confirmatory factor analysis (CFA). While most studies treat the items as continuous, some treat them as categorical and conduct item factor analysis (Wirth & Edwards, 2007) either as a categorical CFA or an analysis based on item response theory models (IRT). It is important to distinguish whether CFA tests theoretically assumed models or previous empirical findings of EFA. In the latter case, models often have fewer factors because of the criteria used (eigenvalue > 1, buckle in the Screeplot).

Below, we present exemplary results on the structure of the eHEALS categorized by their analysis method. Studies are divided by exploratory vs. confirmatory analysis techniques. Confirmatory analyses are further divided by treating outcomes as continuous or categorical.

A result from EFA is that 1 factor was found in various populations (e. g., Juvalta et al., 2020; Mialhe et al., 2021; Wångdahl et al., 2021). Fewer studies reported 2 factors resulting from EFA (e. g., Dale et al., 2020; Holch & Marwood, 2020; Shiferaw, 2020). Exploratory structural equation modeling (ESEM) found a 3-factor model to be superior to 1- and 2-factor models, though the high correlations of the three factors pointed to a unidimensional second-order factor (the data were modeled as categorical; Stellefson et al., 2017). Paige et al. (2018) also found a 3-factor model by using ESEM (the data were modeled as continuous).

Studies reporting 1, 2, or 3 factors resulting from a (continuous) CFA are exemplarily presented in Electronic Supplement (ESM) 1. Sample populations differed widely. Factor compositions of the 2 and 3 factors were often different, although some replication attempts were made. The studies reporting 2 and 3 factors mainly employed CFA alone, while the studies reporting 1 factor mainly used CFA to confirm previous EFA results. Studies regarding the items of the eHEALS as categorical similarly suggested 1, 2, or 3 factors. One factor was the result of a categorical CFA in chronic disease patients in the USA (Paige et al., 2017). Brørs et al. (2020) found three factors in a categorical CFA in Norwegian coronary patients. One factor from IRT analysis was found by Diviani et al. (2017), Juvalta et al. (2020), and Lin et al. (2020). Juvalta et al. (2020) tested separate bifactor models with 2 and 3 group factors in Swiss-German parents. According to the results, a unidimensional structure proved to be valid. Two factors from IRT analysis were identified by Richtering et al. (2017).

In sum, most research indicates that the eHEALS is unidimensional. While 2- and 3-factor solutions have also received support, they differ in the configuration of their factors. Hence, the factor structure of the eHEALS is still ambiguous. As Hyde et al. (2018) and Sudbury-Riley et al. (2017) pointed out, theoretical arguments for dimensionality have received too little attention, and theoretically unfounded interpretations of factor results do not add substantial content to this controversy. As the theoretical background is heterogeneous, it is not surprising that the theoretical combination of items and factors varies, too. Thus, the discussion is not just about the number of factors but also about their content. As long as it is unclear which dimensions can be assessed separately (Dresch et al., 2022), eHealth literacy and its dimensions cannot be measured and interpreted adequately. Hence, research is needed to clarify the dimensionality of the eHEALS.

Study Purpose

To our knowledge, only two studies on the dimensionality of the German version of the eHEALS have been conducted in Germany (one other study used the German version in Switzerland, Juvalta et al., 2020): (1) The eHEALS was originally translated for a study among 12th-grade students; a 1-factor model was tested against a 2-factor model comprising the subscales information-seeking and information appraisal (Soellner et al., 2014). (2) In a sample of women aged 50 participating in a randomized controlled trial on informed choice in mammography screening, the superior fit of the 2-factor model over the 1-factor model was replicated, but a 3-factor model (information-seeking, information appraisal, and information use), hypothesized regarding the competence requirements of informed choice, fitted best (Reder et al., 2019). We have not tested the 3-factor structure with the 12th-grade student sample (Soellner et al., 2014) because assessing eHealth literacy according to the model of health competence (Soellner et al., 2009) did not implicate the third factor of information use. Much new research has been published since our previous publication in 2014. Back then, no other study had conducted a CFA to test for a multifactorial model of the eHEALS. Only three translations of the eHEALS had been published: a Dutch translation (van der Vaart et al., 2011), a Japanese translation (Mitsutake et al., 2011), and a Chinese translation (Koo et al., 2012). Today, much more research covers many multidimensional models in various languages and populations. Remarkably, a conclusion regarding the factor structure of the eHEALS can still not be drawn.

To allow us to add further evidence on the structure of the eHEALS, we conducted a reanalysis to assess whether the 3-factor model fitted better than the 2- and 1–factor models in data collected using the German version of the eHEALS in a German non-representative sample.

Study Design, Participants, and Procedure

The data were collected as part of a cross-sectional paper-and-pencil survey on health literacy (see Soellner et al., 2010, 2017). The sample comprised 327 students attending the 12th grade of a secondary school preparing for university attendance (“Gymnasium”). The mean age was 18 years. Data were collected during class sessions in four schools in Cologne / Bonn in 2009.

eHEALS

The eHEALS (Norman & Skinner, 2006a) comprises eight items (see ESM 2): (Item 1) knowing how to find information online, (Item 2) knowing how to use the internet to answer questions, (Item 3) knowing what health resources are available, (Item 4) knowing where to find health resources, (Item 5) knowing how to use this health information, (Item 6) having the skills to evaluate health resources, (Item 7) ability to discriminate between high and low-quality resources, (Item 8) having the confidence to use this information to make health decisions. Response options on a 5-point Likert scale range from strongly disagree to strongly agree. The translation procedure was previously described (Soellner et al., 2014).

In previous research, we proposed that the eHEALS has 2 and 3 factors, respectively. In the 2-factor model (Soellner et al., 2014), information-seeking comprises Items 1 to 5 and 8; information appraisal comprises Items 6 and 7. In the 3-factor model (Reder et al., 2019), information-seeking comprises Items 1, 3, and 4; information appraisal comprises Items 6 and 7; information use comprises Items 2, 5, and 8.

Statistical Analysis

We analyzed the data using MPlus Version 8 (Muthén & Muthén, Los Angeles, CA). To allow us (1) to replicate the CFA comparison, (2) to use full information maximum likelihood to deal with missing data, (3) to estimate an absolute model fit, and (4) to account for problems arising from the scale level of the five-point Likert scale, we chose robust maximum likelihood CFA. Cases with missing values (n = 4) were thus included, and data were treated as continuous. Taking into account the controversy regarding the nestedness of models that differ in the number of latent factors (e. g., Brown, 2015) and the possibility of compromised χ²-results following the fixing of parameters to the border of their parameter space (i. e., 1), the following model fit indices were compared: comparative fit index (CFI), Tucker-Lewis index (TLI), root mean squared error of approximation (RMSEA), standardized root mean squared residual (SRMR), Akaike information criterion (AIC), and Bayesian information criterion (BIC). Cut-offs are problematic for model fit because their meaning varies with model characteristics (e. g., McNeish & Wolf, 2022, who propose a simulation-based approach). Different values have been proposed for acceptable model fit: CFI: > .90 (Little, 2013), > .95 (Hu & Bentler, 1999; Schermelleh-Engel et al., 2003); TLI: > .90 (Little, 2013), > .95 (Hu & Bentler, 1999); RMSEA, < .08 (Little, 2013, Schermelleh-Engel et al., 2003), < .06 (Hu & Bentler, 1999); SRMR: < .08 (Hu & Bentler, 1999), < .10 Schermelleh-Engel et al., 2003); AIC and BIC, as small as possible (Schreiber et al., 2006). To interpret our results, we refer to the most conservative values.

To assess scale homogeneity, we calculated the coefficient ω. Following a similar level for acceptability as for Cronbach’s α (Nunnally & Bernstein, 1994), values > .70 indicated acceptable internal consistency.

Weighing Evidence

The number of factors and their item composition are both controversial (Reder, 2019). Lee et al. (2021) conducted a systematic review and a meta-analysis of eHealth literacy instruments up to March 2021. Concerning the eHEALS, this review includes 18 different language versions tested in 26 countries resulting in 7 different 2-factor structures and three different 3-factor structures. Results were obtained in diverse populations: adolescents, adults, and the elderly as well as community samples and clinical settings. The quality of the structural validity of the different eHEALS models was assessed on two dimensions according to the COSMIN guideline (Prinsen et al., 2018): (1) The quality of measurement properties was rated as sufficient, insufficient, or indeterminate. (2) The quality of evidence was rated as low, moderate, or high (Grades of Recommendation, Assessment, Development and Evaluation). The 1-factor structure was rated as insufficient with moderate-quality evidence (Lee et al., 2021). The seven 2-factor models were rated as follows: The 2–factor model tested by Neter et al. (2015) was sufficient with low-quality evidence for structural validity. The 2-factor model tested by Bazm et al. (2016), the 2–factor model tested by Gazibara et al. (2019), the 2-factor model tested by Tomás et al. (2014), and the 2–factor model tested by Holch and Marwood (2020) were all rated as sufficient with moderate quality evidence. The 2-factor model tested by Diviani et al. (2017), Juvalta et al. (2020), and Soellner et al. (2014) as well as the 2-factor model tested by Dale et al. (2020), Efthymiou et al. (2019), Richtering et al. (2017), and Shiferaw (2020) were both rated as insufficient with high-quality evidence (Lee et al., 2021). The three 3-factor models were rated as follows: The three factors tested by Brørs et al. (2020), Gartrell Han et al. (2020), and Sudbury-Riley et al. (2017) and the three factors tested by Paige et al. (2018) were both sufficient with high-quality evidence for structural validity while the three factors tested by Hyde et al. (2018) were rated as sufficient with low quality evidence (Lee et al., 2021).

Thus, two of the three 3-factor structures of the eHEALS were evaluated as superior to all other structures of the instrument (Lee et al., 2021). Unfortunately, this review does not include our 3-factor results (Reder et al., 2019), possibly because the focus of our paper (the moderation effect of the eHEALS) was not within the scope of the search criteria. So, we can add evidence to these results implying a 3-factor structure, but we propose a different factor configuration than those rated in the review (information-seeking, Item 1, Item 3, Item 4; information appraisal, Item 6, Item 7; information use, Item 5, Item 8, and possibly Item 2). This cannot be resolved with the current evidence.

Furthermore, we think it is necessary to focus more intensely on the analysis method when interpreting the different factor results; CFA and EFA are especially likely to arrive at other factor structures if the theoretical assumption in CFA includes multidimensional models.

The items were consistently measured on a 5-point scale; in the analysis, this was sometimes treated as ordinal (e. g., Juvalta et al., 2020) but in the vast majority as continuous. Unfortunately, it is practically impossible to assess to what degree these very different factor structures result from the analysis method (exploratory vs. confirmatory, continuous vs. categorical, etc.), the models tested, the language versions, the country settings, or the heterogeneous populations (Reder, 2019).

The eHEALS has several problems itself. First, it does not mirror any health literacy model in terms of the number of dimensions. It is questionable how valid a scale may be with such weak theoretical underpinnings. Second, the eHEALS has a self-assessment format and is not a performance test entailing the common problems of self-report instruments. Third, the whole discussion about the factor structure is only relevant when the eHEALS is still a valid instrument and not (too) outdated (i. e., after the advancements the online world of health resources and technologies has encountered, including social media and mobile Internet; Lee et al., 2021; Sudbury-Riley et al., 2017; Van der Vaart et al., 2013). Ubiquitous internet access, social media, and the Web 2.0 have changed the electronic environment considerably (Sudbury-Riley et al., 2017). It is questionable how well the eHEALS is equipped to deal with this new environment.

Limitations

The following limitations pertain to our study itself. First, external validity was limited since only students from a particular area in Germany and from a particular school form and grade were included (Soellner et al., 2014). Related to the school form is that participants were more highly educated than the average population (Soellner et al., 2014). Similarly, the previous sample of women aged 50 had a higher education level than the general population (Reder et al., 2019). So, the sample was highly educated in both studies. Second, the data were collected in 2009 and are therefore unlikely to represent contemporary 12th-grade students. This limitation is more severe in this research context since the eHEALS assesses electronic health literacy, and the electronic setting has changed tremendously over the last 10 years (e. g., use of social media and smartphones). Third, it is questionable whether results apply only to the German version or are generalizable to other languages (Reder et al., 2019; Soellner et al., 2014). Fourth, while the replication of findings and extensive use of collected data is desirable in research, running many CFA models on the same data set would capitalize on chance. Since solid evidence exists for a 3–factor model, we deemed a reanalysis with a third CFA permissible. Fifth, we conducted no cross-validation of the exploratory analyses to test whether the bifactor model as well as the 2- and 3-factor models without Item 2 would fit in another sample. Sixth, the exclusion of Item 2 shortened the information use scale to only 2 items. Since the coefficient ω remained high, homogeneity is still given. Allowing a double loading on both factors (information use and information seeking) could have been an additional approach to account for the local misfit of Item 2. Since Item 2 would not add additional value, when applying the eHEALS for intervention planning or evaluation, we opted to leave Item 2 out.

Practical Relevance

The unknown factor structure of the eHEALS means that the construct‍(s) of the eHEALS presently cannot be scored with confidence. A scale that cannot be scored, however, is of little practical help when it comes to assessing the eHealth literacy of a population. Meanwhile, we recommend scoring the eHEALS separately for the three factors – at least when using the German version. Regarding Item 2, the combined evidence is inconclusive. Looking at information seeking, information appraisal, and information use separately may produce valuable insights into eHealth literacy as well as for planning and evaluating health interventions.

Future Research

A more theory-representative eHealth literacy scale would be desirable, which might avoid some of the dimensionality problems currently observed with the eHEALS. Multinational structural measurement invariance of the eHEALS (see Sudbury-Riley et al., 2017, for a comparison of the UK, USA, and New Zealand) should be tested to see how well a factor structure holds in different country settings. Similarly, multilanguage measurement invariance (e. g., French and Italian versions in Switzerland) could be tested to elucidate whether dimensionality problems stem from translation issues. Nevertheless, the question remains whether there is a single factor solution that fits all populations (language, country, setting). The eHEALS is very short to support a 3-factor structure – and even shorter when Item 2 is excluded. Additionally, it is somewhat outdated. It therefore would profit from the development of new items extending the scales we found.