Abstract. In psychology and the social sciences, it is often of interest how complex structural relations among variables are moderated by profiles or combinations of persons’ attributes. Some state-of-the-art methods, such as latent class analysis, are well-suited for this purpose. However, they can lead to methodological problems (e.g., convergence issues) or interpretative difficulties (e.g., due to nondistinctive profiles). For these cases, two other approaches combining structural equation modeling with machine learning have been proposed, namely structural equation model (SEM) trees and SubgroupSEM. These approaches allow for exploration of how parameters of a SEM differ depending on combinations of a person's attributes. This can be useful for generating hypotheses for future research. In this paper, we provide an empirical illustration of SubgroupSEM using an example from research on the development of commitment in interdisciplinary study programs in German higher education and identify combinations of vocational interests related to exceptional development.
In psychology and the social sciences, we are often interested in how complex structural relations among variables are moderated by profiles or combinations of persons’ attributes (i.e., how distinct relationships in an outcome model differ between groups of people). Structural equation modeling is a popular framework for modeling, and estimating complex structural relations and various techniques exist to investigate moderating effects of (observed or latent) group variables. When the group variable is known and observed, a multigroup structural equation model (Jöreskog, 1971) can be applied. There, the outcome model is estimated for each group simultaneously but with potentially varying parameter estimates. If the group variable is a latent class (i.e., unobserved) representing profiles of persons’ attributes, then group membership must be estimated from multiple observed variables, for example, using a latent class analysis (LCA; Arminger et al., 1999; Jedidi et al., 1997). Note that we use LCA as a broader term also including finite mixture models and latent profile analysis. In the following, we will refer to variables used for estimating group membership as covariates and variables being part of the outcome model (i.e., a structural equation model) as model variables.
LCA commonly refers to a latent variable model with a categorical latent variable. That is, we assume that response patterns on the covariates are due to latent categories, which cannot be directly observed, but can be indirectly measured by the covariates. The results of a LCA can be connected to a structural equation model (SEM; Asparouhov & Muthén, 2014; Lanza et al., 2013; Nagin, 2005; Nylund-Gibson et al., 2019), which is often conducted as a two-step procedure: First, a LCA is used to identify groups (or latent classes) with similar response patterns on the covariates. Then, each individual is assigned to one class (e.g., by maximum probability for class membership). Second, the latent class indicator variable is included in the SEM, for instance, as the group variable in a multigroup SEM. In a similar approach, the roles of model variables and covariates are interchanged, that is, latent classes are directly estimated on the model variables identifying heterogeneous structural relations, and, in the second step, these classes are compared with regard to their means and covariance structure on the covariates.
Either way, practical applications of LCA can be challenging and coupled with a number of pitfalls: First, the number of classes is typically unknown beforehand and has to be determined by comparing solutions with differing numbers of classes. This involves methodological challenges, for example, choice of fit indices (Nylund et al., 2007), and substantive challenges, as profiles are commonly desired to be distinct and interpretable (Spurk et al., 2020). Balancing these aspects can be quite complex in a practical application. Second, LCA benefits from a higher number of covariates as well as from higher quality of covariates (Wurpts & Geiser, 2014). In contrast, there is an ongoing development of using short scales in clinical and research settings (Kruyen et al., 2013, 2014). While short scales can provide similar validity as their longer counterparts (Heene et al., 2014; Thalmayer et al., 2011), this might increase the chances of estimation problems in LCA. Third, the estimation of a LCA can be problematic due to a number of local optima, making it difficult for the optimizer to arrive at the global optimum and increasing the likelihood of a faulty result (Shireman et al., 2016). While strategies exist to tackle this issue (cf. Shireman et al., 2017), in some applications, it might remain uncertain whether the found solutions are a global or local optimum.
In practice, there might not always be a solution to these challenges. In these cases, approaches combining machine learning paradigms with confirmatory statistical models can help gaining insights into the relations among model variables and covariates. Two notable examples in the realm of structural equation modeling are SEM trees (Brandmaier et al. 2013), which combine SEM with decision trees (Quinlan, 1986) and SubgroupSEM (Kiefer et al., 2022), which is based on a subgroup discovery paradigm (Klösgen, 1996). These approaches facilitate exploration of complex structural relations with regard to a number of covariates in two ways: (a) The exploration of the covariate space is less formalized, that is, these approaches examine manifest combinations of the covariates instead of generating latent classes via a measurement model. While manifest combinations do not account for measurement uncertainty, this approach has fewer estimation problems; at the same time, the SEM part retains its formal character for modeling structural relations among manifest and latent variables; (b) the clustering of the covariates is directly connected to desired properties of the structural equation model (e.g., exceptional parameter constellations). In other words, these approaches can be targeted to identify combinations of the covariates having a moderating effect on the structural relations among the model variables.
In this paper, we provide an empirical illustration of the SubgroupSEM approach to a real-world research problem from higher education, for which a traditional LCA approach combined with a SEM leads to results that are difficult to interpret. We first try to apply a LCA on variables measuring vocational interests and investigate the resulting interest profiles with regard to their respective development of commitment in interdisciplinary study programs. Then, we apply SubgroupSEM to discover combinations of variables measuring vocational interests (i.e., covariates) that are linked to exceptional development of commitment in interdisciplinary study programs using large-scale assessment data from the National Educational Panel Study (NEPS; NEPS Network, 2021). Thereby, we illustrate how this approach can be used to generate novel hypotheses about the relation of vocational interests and development of commitment.
The remainder of this manuscript is structured as follows: First, we give a brief introduction to LCA and recent combinations of machine learning with structural equation modeling. Second, we introduce our applied example from German higher education research and lay out the substantive background, followed by our substantive research goals. Third, we describe the method of our application, including a presentation of the NEPS sample, our measures, as well as the statistical analysis. Fourth, the results from our study are presented and discussed. Finally, we conclude the manuscript by summarizing the key findings of our application by comparing the results from a LCA with SEM and SubgroupSEM.
LCA and Machine Learning
In LCA (McLachlan & Peel, 2000), we assume that the response patterns observed on the covariates are due to a finite number of latent classes, which are not directly observable. Hence, the observed (multivariate) distribution of the covariates can be approximated as a finite mixture of distributions, for example, multivariate normal distributions. The (finite) number NC of latent classes has to be predetermined, and latent classes are estimated in a way that similar response patterns are likely to come from the same latent class. This is illustrated in the left panel of Figure 1 for a bivariate case with three latent classes. Usually, multiple LCAs with different NC are estimated, and the final number of classes is chosen based on fit indices and interpretability of the solution. Latent classes can be characterized by the means on the observed covariates given a latent class C, but variances and covariances can also be a characteristic of a latent class. As a probabilistic approach, LCA does not provide a direct assignment of each person to one of the classes, but a probability of a response pattern given specific latent classes and, vice versa, the probability of membership in a latent class given a specific response pattern. Hence, LCA accounts for uncertainty of the measurement model by refraining from a direct assignment. However, each person can be assigned post hoc to the latent class which is most probable for this person. This assignment is stored as a latent class indicator variable. Note that it would also be possible to estimate the relations of the latent classes to further variables in a single model without manifest class assignment. However, the latent class variable in these distal-as-indicator models is not comparable to the latent class variable in the aforementioned two-step procedures because the measurement model is not solely based on the covariates but also the model variables (Nylund-Gibson et al., 2019). We will not demonstrate this approach in this paper.
Figure 1 Graphical comparison of approaches to detect heterogeneous subgroups/classes for nine observations. Shapes of marks indicate information of a third (i.e., targeted) variable. Left panel shows clustering as provided by LCA, where clusters are formed by the proximity of observations only, and no information from the third variable is used. Middle panel shows decision tree paradigm, where clusters are formed to maximize predictive accuracy with regard to the third variable (i.e., triangles vs. circles). Right panel shows subgroup discovery paradigm, where clusters are formed balancing predictive accuracy with an easy-to-understand classification. Figure inspired by illustrations in Herrera et al. (2011).
In recent years, two popular machine learning paradigms have been adopted for use with structural equation modeling allowing us to explore SEMs given various combinations of covariates’ values. The first example are SEM trees (Brandmaier et al., 2013), which are based on a decision tree paradigm (Quinlan, 1986). They follow the rationale of recursive partitioning, which means the covariate space is recursively split into smaller portions and each split is designed to maximize the improvement on the model fit as measured, for example, with an LRT. Hence, SEM trees have a strong focus on maximizing predictive accuracy of the resulting tree. This is illustrated in the middle panel of Figure 1, where the algorithm aims at perfectly separating observations with regard to their shape (circles vs. triangles). As a consequence, the separation rule (shape of the filled area) can be quite complex and difficult to interpret from a substantive point of view. For an introduction to recursive partitioning, see Strobl et al. (2009). For further reading on this topic, we recommend Zeileis et al. (2008) for a related approach for statistical models in general and Brandmaier et al. (2016) for a random forest extension of SEM trees.
The second example is SubgroupSEM (Kiefer et al., 2022) which is based on a subgroup discovery paradigm (Klösgen, 1996). Subgroup discovery builds on similar algorithms as the decision tree paradigm but tries to balance predictive accuracy with an easy-to-understand classification. This is illustrated in the right panel of Figure 1. Here, the algorithm does not aim at a perfection separation of the different shapes but at separating as good as possible while maintaining an easily understandable classification rule (illustrated by an ellipse instead of a complex shape). As a consequence, the paradigm accepts classification errors by design. This property makes subgroup discovery less suitable for prediction purposes but more suitable for exploratory research and generation of hypotheses for future research. For an overview of subgroup discovery, see Herrera et al. (2011). Applications of this paradigm to statistical models are often called exceptional model mining (EMM; Duivesteijn et al., 2016; Leman et al., 2008). EMM has been proposed for regression models (Duivesteijn et al., 2012), mediation models (Lemmerich et al., 2020), latent growth curve models (Mayer et al., 2021), and structural equation models in general (SubgroupSEM; Kiefer et al., 2022) to name a few examples.
Both SEM trees and SubgroupSEM differ from LCA in that they can explore a manifest covariate space with regard to parameter constellations in a SEM. Despite the conceptual differences displayed above, both SEM trees and SubgroupSEM can yield similar results in practical application, especially if a low search depth and a likelihood ratio test-based criterion is used. Kiefer et al. (2022) illustrate this in a confirmatory factor analysis example, where depending on the specific search goal, both approaches yield similar results based on a LRT-based criterion and different results with a user-defined criterion for SubgroupSEM. For a more in-depth comparison of LCA, SEM trees, and SubgroupSEM, see Kiefer et al. (2022). While SEM trees are a well-established approach, which is used and described in several practical applications (e.g., Ammerman et al., 2019; Brandmaier et al., 2017), there is a lack of practical demonstrations of the more recent SubgroupSEM approach. Thus, we will apply SubgroupSEM for illustrating the advantages (and disadvantages) of hybrid methods for exploring heterogeneous groups in structural equation models.
Motivating Example
Our illustrative example is motivated by real-world research on interdisciplinary study programs in German higher education. Thus, we now lay out the substantive background on higher education research, followed by our substantive research goals.
Background
In recent years, interdisciplinary study programs (e.g., European studies, business informatics) have gained popularity and importance in the German higher education system (Hachmeister & Grevers, 2019). Interdisciplinary study programs combine different disciplinary perspectives in one integrated program. The program “European Studies,” for example, includes historical, political, geographical, economic, and social perspectives on Europe. Other interdisciplinary programs are limited to two or three different disciplines. Despite their growing popularity, there is little research on the antecedents and trajectories of study success in such programs compared to more traditional study programs.
Study success is a multidimensional construct, including subjective and objective as well as process- and outcome-oriented criteria (Dahm & Kerst, 2016). Here, we chose a subjective, process-oriented one, namely academic commitment, which distinguishes the components' affective involvement and achievement orientation (Dahm & Kerst, 2016). In higher education, the student gets grades, while simultaneously, the student evaluates the study program, adopts its norms, and develops an affective identification (Dahm & Kerst, 2016). Affective involvement addresses the affective connection to the study program, while achievement orientation captures students’ aspirations regarding their academic performance. In interdisciplinary programs, academic commitment might be more challenging due to the diverging academic demands on the achievement side. This diversity of demands, however, might be particularly enjoyable for some students, resulting in higher affective involvement.
An important predictor for study success is the fit of personal characteristics such as interests and abilities and the environment – most prominently theorized by person–environment (PE) fit theory (Edwards, 1996; Kristof-Brown et al., 2005). Congruence between the person and the environment leads to satisfaction with the chosen environment and promoting the retention in this environment. In higher education, therefore, PE fit in terms of both ability-demand fit and interest-vocation fit is associated with study retention and academic success (Le et al., 2014). Interdisciplinary and multidisciplinary study programs are likely to have more diverse demands than monodisciplinary study programs due to the different disciplinary perspectives involved.
A common taxonomy of vocational interests involves six areas of interests (i.e., realistic, investigative, artistic, social, enterprising, conventional; Holland, 1997). Following this model, each person would have a dominant area of interest; however, in practice, interest profiles with more than one peak or even flat profiles (no outstanding area of interest) are observed (Vock et al., 2013). As a consequence, latent profile analysis has repeatedly been applied to model and detect distinctive patterns of vocational interests (e.g., Johnson & Bouchard, 2009; McLarnon et al., 2015). In the context of interdisciplinary study programs, a profile of vocational interests with more than one peak might be beneficial for subjective study success: For instance, a business informatics program might raise enterprising, investigative, and conventional demands. Here, a single dominant area of interest on the student’s side would be disadvantageous regarding PE fit. The experience of inadequate fit of one’s own interests and the study program’s demands are likely to be reflected in less study commitment. Therefore, we raise the question which combination of vocational interests is linked to ongoing study success in the chosen type of study program.
To our knowledge, there is no prior research on how (latent) interest profiles moderate the association between chosen type of study program and study success.
Substantive Research Goals
In this study, we use large-scale panel data from Starting Cohort 5 of the National Educational Panel Study (NEPS) to accomplish two goals: First, we examine the development of commitment to one’s study program over the course of three years and investigate how the development differs for interdisciplinary, monodisciplinary, and multidisciplinary study programs. Second, we take a look at the moderating influence of vocational interests on commitment development within these types of study programs. More precisely, we focus on the question: Can we identify interest profiles (or combinations) for which development of commitment in interdisciplinary study programs is considerably different from other profiles?
While the first substantive research goal can be addressed using a multigroup multistate model (Steyer et al., 2015), the second goal is more difficult to achieve. As we will demonstrate below, LCA leads to a difficult-to-interpret result in this scenario for two reasons: (a) The determination of the number of latent classes is problematic as fit indices keep improving with an increasing number of classes, but the profiles generated are not distinctive beyond a three-class solution; (b) measurement of vocational interests within the NEPS data set contained three items per dimension with each vocational interest score variable containing 13 unique values at most, which is a limitation for latent profile analysis. In comparison, other studies on profiles of vocational interests used more than 10 items per dimension (e.g., Johnson & Bouchard, 2009). Thus, we apply SubgroupSEM to explore the relation of different combinations of vocational interests and development of commitment.
Method
Sample
For our analysis, we use data from the Starting Cohort 5 (first-year university students) from the National Educational Panel Study in Germany (NEPS Network, 2021). From these data, we removed an oversample of student teachers and removed individuals with missing values on covariates and model variables. This resulted in a final sample of N = 11,476 students. The remaining missing values in the data were treated using full-information maximum likelihood (FIML) estimation for the SEM. Note that NEPS data were randomly sampled by study programs across Germany and not on an individual, but a study program, level (i.e., clustered sampling). Therefore, cluster-robust standard errors were used in all models.
Measures
Students’ Commitment to Study Program
The scale students’ commitment to their study program consists of two subscales for achievement orientation and affective involvement each measured with three items:
Y1t: I do not dedicate more time to my studies than absolutely necessary.
Y3t: I pursue high aspirations concerning my academic performances.
Y5t: I invest a great deal of effort in order to be successful in my studies.
Y2t: I enjoy my fields of studies very much.
Y4t: To be honest, my studies don’t thrill me.
Y6t: I can completely identify with my studies.
We used the items Y1 (reversed), Y3, and Y5 as indicators of a common latent state variable η1t called achievement orientation and the items Y2, Y4 (reversed), and Y6 as indicators of a common latent state variable η2t called affective involvement. Each item was measured on a five-point rating scale (coded from 1 = does not apply at all to 5 = does apply completely). Note that all individuals are first assessed in their first semester, that is, the time variable t represents years since the start of the study program and corresponds to a specific wave of the NEPS study.
Types of Study Programs
A classification variable for study programs regarding their interdisciplinary nature is available for NEPS Starting Cohort 5 since Scientific Use File 13-0-0 (Claus et al., in preparation). The classification is based on Hachmeister and Grevers’s (2019) differentiation of monodisciplinary, interdisciplinary, and multidisciplinary study programs. Examples for monodisciplinary study programs are civil engineering, law, and sociology. These programs focus on a single field of study. Interdisciplinary study programs involve several disciplinary perspectives and aim at integrating them. Examples are European Studies, Business Informatics, and Water Science. Multidisciplinary programs also consist of more than one field of study, but the disciplinary perspectives are not integrated. This is, for instance, the case for all teaching degrees (consisting of educational science and the study subjects, e.g., Math and French).
The classification variable K was included with levels 0 = interdisciplinary, 1 = monodisciplinary, and 2 = multidisciplinary. Note that this variable only captures the study program at the first wave and does not account for possible changes of study programs in the following years.
Vocational Interests
Vocational interests are measured according to Holland’s model (1997). Items for the vocational interests scale are sourced from two different instruments (ICA-D, von Maurice, 2006, and AIST-R, Bergmann & Eder, 2005). Each vocational interest dimension is measured with three items on a five-point rating scale (coded from 1 = I am very little interested in that/don’t do it at all to 5 = I am very interested in that/like to do it very much). Example items are to build something (realistic), to observe and analyze something (investigative), draw paintings (artistic), to help sick people (social), to negotiate with other people (entrepreneurial), and to keep records or spreadsheets (conventional). For each interest dimension, a person-specific mean score variable, averaging over the corresponding items, was used.
Statistical Analyses
Multistate Model
As a baseline model, we specified a three-group multistate model, or more specifically, a latent state model with indicator-specific residual factors (Eid et al., 1999) for both achievement orientation η1 and affective involvement η2. The group-specific model is presented in Figure 2. This results in separate latent developments of affective involvement and achievement orientation. Furthermore, we specified this baseline model in a multigroup framework with types of study programs as a categorical variable K. Measurement models of the latent variables were invariant across groups, but expectations E(η1t|K = j), E(η2t|K = j) at time t in group j, variances, and covariances of the latent variables could vary among groups.
Figure 2 Group-specific latent state model with indicator-specific residual factors. η1t represents the latent state variable of achievement orientation and η2t represents the latent state variable of affective involvement at time point t; Yit represent the time-specific (t) measurement of item i of the student’s commitment to their study programs scale. RF denotes the indicator-specific residual factors.
We conduct Wald tests on the latent means of each outcome variable comparing development between interdisciplinary and monodisciplinary studies as well as between interdisciplinary and multidisciplinary studies. The computations were carried out using the statistical programming language R (R Core Team, 2021) and the structural equation modeling R package lavaan (Rosseel, 2012).1
Latent Class Analysis
For the traditional two-step method, we combined our multigroup multistate model with the results from a latent profile analysis (i.e., a special case of LCA). Profiles of vocational interests were extracted via latent profile analysis as implemented in Mplus (B. O. Muthén, 2002; L. K. Muthén & Muthén, 1998–2017), and individuals were assigned to the profile which was most probable for them. Due to the small number of items per dimension, we expect to identify fewer and less nuanced profiles than comparable studies (e.g., Johnson & Bouchard, 2009) using more than 10 items per dimension. Using the class indicator variable, we extend our baseline model to incorporate all combinations of interest profiles and types of study programs (i.e., 3 types × 3 profiles, a total of nine groups).
As this is an exploratory analysis and we do not have any a priori hypothesis, we refrain from conducting and interpreting statistical hypothesis testing in the resulting model but rely on a descriptive presentation of the results. If such a priori hypotheses were under consideration in a specific application, the two-step approach should be carried out with a more sophisticated technique (cf. Nylund-Gibson et al., 2019) accounting for the uncertainty of the post hoc class assignment.
SubgroupSEM
Applying subgroup discovery in structural equation models with SubgroupSEM can be considered a four-fold task:
1.
Specification of the target SEM: We specified the target model as a six-group version (subgroup/complement × 3 study programs) of the multistate model combining the subgroup indicator variable S and the study program variable K; at every iteration of the subgroup discovery algorithm, the subgroup variable S represents a specific combination of categorical covariates and indicates which persons belong to this combination (S = 1) and which do not (i.e., S = 0).
2.
Selection of covariates: We chose the person-specific mean score variables of the six vocational interest dimensions as covariates from which subgroups are generated from. We computed tertiles for the scores, such that the search algorithm differentiates between low, medium, and high interest on all six interest dimensions which yields 36 = 729 possible combinations; for example, in one iteration, the algorithm might compare all persons with high values on the realistic interest dimension (S = 1) versus everyone else (S = 0), and in the next iteration, S = 1 comprises all persons with medium values on the social interest dimension (vs. the complement S = 0) and so on.
3.
Definition of how the interestingness or exceptionality of a subgroup is measured (a so-called interestingness measure or IM): We computed an exploratory IM as the sum of latent mean differences in affective involvement from the first to second year and from the second to third year within the combination of subgroup S = 1 and interdisciplinary study programs K = 0: As the interestingness measure will be sorted descending, positive values that indicate increasing mean values of affective involvement in interdisciplinary study programs will be at the top of the list.
4.
Choice of a search algorithm: We chose an exhaustive depth-first search algorithm which goes through all possible combinations of the covariates. This step also includes the choice of the search depth d = 6 (i.e., 729 combinations of all six interests are inspected) and a minimum subgroup size Nsg = 1,100 (i.e., roughly a tenth of the overall sample to ensure investigated subgroups are of substantial size) that should be considered. For more information on the available algorithms and their differences and advantages, see Kiefer et al. (2022). In the following, we will only present the top 3 subgroups to keep the results compact.
The computations were carried out using the statistical programming language R (R Core Team, 2021) and the R package SubgroupSEM (Kiefer et al., 2022). Again, as this is an exploratory analysis and we do not have any a priori hypothesis, we refrain from conducting and interpreting statistical hypothesis testing in the resulting models.
Results
The model fit statistics for each of the five estimated models are summarized in Table 1. In the following presentation of the results, we will focus on the core aspects of our demonstration but provide more detailed results (i.e., parameter estimates, latent profile analysis results, etc.) at https://doi.org/10.23668/psycharchives.12167 (Kiefer et al., 2023).
For the multistate model, the FIML estimator used N = 2,301 students in interdisciplinary study programs, N = 6,488 in monodisciplinary study programs, and N = 2,687 in multidisciplinary study programs. For both achievement orientation and affective involvement and for all types of study programs, the overall latent means were slowly declining over the three years. On average, achievement orientation is less pronounced than affective involvement. These developments are presented in Figure 3. From visually inspecting, the trajectories seem to be very similar regarding absolute level and form across types of study programs.
Figure 3 Plot of estimated latent means of achievement orientation η1 and affective involvement η2 based on the multistate model and separated by type of study program (i.e., interdisciplinary, monodisciplinary, multidisciplinary). Error bars represent the 95% confidence interval of estimated latent means.
However, Wald tests reveal that trajectories of affective involvement are significantly different between interdisciplinary and monodisciplinary programs (χ2(3) = 8.585, p = .035) as well as between interdisciplinary and multidisciplinary programs (χ2(3) = 8.688, p = .034). These differences in trajectories were not statistically significant for achievement orientation (interdisciplinary vs. monodisciplinary: χ2(3) = 4.128, p = .248; interdisciplinary vs. multidisciplinary: χ2(3) = 1.520, p = .678).
Latent Class Analysis
Determination of the final number NC of latent profiles was difficult due to diverging indications of fit indices and substantive interpretability. That is, up to three (substantively) distinctive profiles could be identified, and adding more profiles to this solution led to profiles that were almost identical to the already identified profiles except for a difference on the social interest dimension. In contrast, fit indices kept improving for up to 10 profiles, where we stopped increasing NC. We finally decided on a three-profile solution to obtain interpretable and distinctive profiles.
The three interest profiles can be described as a flat profile (i.e., medium values on all six dimensions; about 55% of students; N = 7,842 assigned), a profile with peaks on the social as well as enterprising dimensions (about 10%; N = 1,807 assigned), and a profile with peak on social interests (about 35%; N = 5,344 assigned). The profile-specific means of the vocational interest variables are presented in Table 2. As expected, we found fewer and less nuanced profiles than comparable studies in the field using significantly more items per dimension (e.g., Johnson & Bouchard, 2009).
Table 2 Latent profile means of vocational interest scores
Profile
Profile means
R
I
A
S
E
C
Note. Means of manifest vocational interest score variables (R = realistic, I = investigative, A = artistic, S = social, E = enterprising, C = conventional) by latent profiles. Scores range between 1 and 5.
Flat
3.266
3.516
2.746
3.313
3.237
2.682
Social-enterprising
2.158
2.384
2.916
3.514
3.585
2.785
Social
2.671
3.361
3.253
4.904
3.640
2.804
Table 2 Latent profile means of vocational interest scores
The trajectories of achievement orientation and affective involvement separated by cluster assignment and type of study program are presented in Figure 4. For all three profiles, the trajectories of affective involvement are declining regardless of the type of study program.
Figure 4 Plot of estimated latent means of achievement orientation η1 and affective involvement η2 based on the multistate model and separated by type of study program (i.e., interdisciplinary, monodisciplinary, multidisciplinary) and latent profiles of vocational interests. Error bars represent the 95% confidence interval of estimated latent means.
SubgroupSEM
The three most interesting subgroups regarding our IM can be described as (a) persons with medium scores on the realistic and high scores on the social dimension (N = 1,517), (b) persons with high scores on the enterprising and the social dimension (N = 1,633), and (c) persons with high scores on both the artistic dimension and the social dimension (N = 1,633). Table 3 provides a summary of the top 3 subgroups. The estimated means of the latent state variables are presented in Figure 5. The trajectories of affective involvement were positive for the top 3 subgroups as expected. In the first and second subgroups, this trajectory is initialized with a drop in affective involvement after the first year but a strong increase after the second year. In the third subgroup, the trajectory of affective involvement was strictly positive over the three years. While not directly addressed by our exploratory interestingness measure, in all top 3 subgroups, we also found positive trajectories of achievement orientation.
Table 3 Overview of top 3 subgroups
#
Subgroup description
Value
Size
Measure
Note. Subgroup description gives the covariates included in the subgroup description; Value reflects the corresponding value on the covariate; Size reflects the subgroup size; Measure reflects the value of the interestingness measure IM for the corresponding subgroup.
Figure 5 Plot of estimated latent means of achievement orientation η1 and affective involvement η2 based on the multistate model and separated by type of study program (i.e., interdisciplinary, monodisciplinary, multidisciplinary) and top 3 subgroups. Error bars represent the 95% confidence interval of estimated latent means.
In all three subgroups, the trajectory of affective involvement is associated with notably larger standard errors in interdisciplinary study programs than in other types of study programs. This is not surprising, as the exploratory interestingness measure focused on the absolute value in differences without accounting for estimation uncertainty. Thus, rather large effects in small subgroups are usually found that are not necessarily statistically significant.
Discussion of Substantive Results
In general, affective involvement with the study program decreased over time. In interdisciplinary study programs, this trend is more pronounced in the second year of studies than in monodisciplinary study programs. Students start with high expectations and enjoy their studies at the time of the first wave (i.e., during the first semester). Over time, students identify less with their chosen fields. This seems to be particularly true for students in interdisciplinary study programs, while the slope is less steep in more traditional monodisciplinary study programs.
LCA of vocational interests revealed only few and quite similar profile solutions. Students with a flat profile, that is, no dominant vocational interest dimension, show a rather flat trajectory of affective involvement in their interdisciplinary study program, although the general downwards trend is also visible. Students who have two dominant vocational interests in the social and enterprising domain, however, show a steeper decline of their affective involvement with their interdisciplinary program. Apparently, it can be particularly challenging to design an optimally engaging interdisciplinary study program that corresponds to an individual profile with these two main interest domains. Students with a single dominant social area of interest do not differ substantially in their trajectories of affective involvement in interdisciplinary programs from either other profile group. Therefore, the groups discovered by latent profile analysis differ in their decline of affective involvement with their program, while the overall trend is independent from vocational interests. Consequently, using this modeling approach and the limitations that occurred during its estimation (i.e., low number of indicators, diverging indications for number of profiles), only little information regarding PE fit of vocational interests and study programs can be deducted.
In contrast, SubgroupSEM found three subgroups for whom the trajectory of affective involvement in interdisciplinary study programs is increasing over time. Those groups of substantial size demonstrate that there are students whose identification with their chosen interdisciplinary study program grows. While students with high artistic and high social vocational interests show a positive trend of affective involvement during their studies, students in the other two identified subgroups (medium realistic and high social, as well as high enterprising and high social vocational interests) develop a growing affective involvement between Waves 2 and 3, that is, in their third year of higher education. Many interdisciplinary programs require students to take basic introductory courses in the first semesters. Highly integrative courses, that are more likely to meet students’ interests due to their more holistic approach, are often found at the end of the bachelor’s program (Minnis & John-Steiner, 2005). These courses might promote students’ identification with the program, particularly if they align with students’ vocational interests. Future research should test this relation, so interdisciplinary programs might be redesigned by moving those classes into earlier semesters for motivational purposes.
In addition, the subgroups’ positive development of subjective study success is specific to interdisciplinary programs. Students with similar vocational interests who attend other programs show trajectories that represent the baseline curve of steady decline in affective involvement. It is important, however, to stress the exploratory nature of the SubgroupSEM approach. That is, the results may point at promising directions for future research or allow generating hypotheses but should never be interpreted in a confirmatory or even causal way.
Conclusions and Implications
In this paper, we provided an empirical example of how SubgroupSEM can be used in applied psychological research to explore moderation of complex structural relations by a person’s attributes. To showcase how different the conclusions may be that can be drawn from both the compared approaches, we discuss the substantive findings derived from our empirical example.
In an applied example from higher education research, we illustrated how commonly experienced challenges of LCA can be overcome using SubgroupSEM: LCA comes with the advantage of being a model-based approach in which the latent classes are established through a measurement model of the observed covariates. With its probabilistic nature, it accounts for uncertainty of both the estimation as well as the class assignment. However, as our application has illustrated, practical estimation of LCA can be coupled with several challenges. In our application, LCA estimated fewer and less nuanced profiles of vocational interests than comparable studies (e.g., McLarnon et al., 2015). We based our LCA on a rather short scale of the vocational interest dimensions and experienced difficulties in determining the number of latent profiles due to diverging indications in terms of fit indices and substantively interpretable solutions. In addition, neither of the three profiles of vocational interests we identified among students was associated with exceptional development of commitment.
In contrast, SubgroupSEM pursues a less formalized way to explore the covariate space (i.e., manifest combinations instead of latent profiles) which is directly connected to our target SEM. As a result, we identified three subgroups of substantial size, for which the development of affective involvement was increasing over three years of study. This stands in contrast to the overall decreasing development. However, such findings are exploratory in nature and can turn out to be artifacts of the data at hand. As we stated above, predictive accuracy is not the main goal in a subgroup discovery paradigm, and thus, the results from this paradigm should be interpreted with caution and not in a confirmatory way. In contrast, the decision tree paradigm (as used in SEM trees) offers a variety of techniques enabling generalization of the results beyond the sample at hand. Examples include cross-validation (cf. Brandmaier et al., 2013) and random forests (Brandmaier et al., 2016).
In conclusion, the LCA approach offers a model-based, sophisticated approach, which can come with some challenges in practical settings. As an alternative, combinations of machine learning and SEM designed for exploration of groups moderating the structural relations among the model variables can be used. However, researchers ought to be careful interpreting the results before they have replicated them based on a different data set.
References
Ammerman, B. A., Jacobucci, R., & McCloskey, M. S. (2019). Reconsidering important outcomes of the nonsuicidal self-injury disorder diagnostic criterion A. Journal of Clinical Psychology, 75(6), 1084–1097. 10.1002/jclp.22754First citation in articleCrossref, Google Scholar
Arminger, G., Stein, P., & Wittenberg, J. (1999). Mixtures of conditional mean- and covariance-structure models. Psychometrika, 64(4), 475–494. 10.1007/BF02294568First citation in articleCrossref, Google Scholar
Asparouhov, T., & Muthén, B. (2014). Auxiliary variables in mixture modeling: Three-step approaches using Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 21(3), 329–341. 10.1080/10705511.2014.915181First citation in articleCrossref, Google Scholar
Bergmann, C., & Eder, F. (2005). Allgemeiner Interessen-Struktur-Test mit Umwelt-Struktur-Test (UST-R)—Revision. Beltz Test GmbH. First citation in articleGoogle Scholar
Brandmaier, A. M., Prindle, J. J., McArdle, J. J., & Lindenberger, U. (2016). Theory-guided exploration with structural equation model forests. Psychological Methods, 21(4), 566–582. 10.1037/met0000090First citation in articleCrossref, Google Scholar
Brandmaier, A. M., Ram, N., Wagner, G. G., & Gerstorf, D. (2017). Terminal decline in well-being: The role of multi-indicator constellations of physical health and psychosocial correlates. Developmental Psychology, 53(5), 996–1012. 10.1037/dev0000274First citation in articleCrossref, Google Scholar
Brandmaier, A. M., von Oertzen, T., McArdle, J. J., & Lindenberger, U. (2013). Structural equation model trees. Psychological Methods, 18(1), 71–86. 10.1037/a0030001First citation in articleCrossref, Google Scholar
Claus, A. M., Lampel, G., Kiefer, C., Mayer, A., & Wiese, B. S. (in preparation). Interdisziplinäre, multidisziplinäre und monodisziplinäre Studiengänge in Deutschland: Ein Klassifikationsschema. Unpublished report. First citation in articleGoogle Scholar
Dahm, G., & Kerst, C. (2016). Erfolgreich studieren ohne Abi? Ein mehrdimensionaler Vergleich des Studienerfolgs von nicht-traditionellen und traditionellen Studierenden. In A. B. Wolter (Eds.), Ergebnisse der wissenschaftlichen Begleitung des Bund-Länder-Wettbewerbs “Aufstieg durch Bildung: Offene Hochschulen”. Band 1. Zielgruppen Lebenslangen Lernens an Hochschulen (pp. 225–265). Waxmann. First citation in articleGoogle Scholar
Duivesteijn, W., Feelders, A., & Knobbe, A. (2012). Different slopes for different folks: Mining for exceptional regression models with cook’s distance. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining—KDD ’12 (p.868). 10.1145/2339530.2339668First citation in articleCrossref, Google Scholar
Duivesteijn, W., Feelders, A. J., & Knobbe, A. (2016). Exceptional Model Mining: Supervised descriptive local pattern mining with complex target concepts. Data Mining and Knowledge Discovery, 30(1), 47–98. 10.1007/s10618-015-0403-4First citation in articleCrossref, Google Scholar
Edwards, J. R. (1996). An examination of competing versions of the person-environment fit approach to stress. Academy of Management Journal, 39(2), 292–339. 10.5465/256782First citation in articleCrossref, Google Scholar
Eid, M., Schneider, C., & Schwenkmezger,
P. (1999). Do you feel better or worse? The validity of perceived deviations of mood states from mood traits. European Journal of Personality, 13(4), 283–306. 10.1002/(SICI)1099-0984(199907/08)13:4<283::AID-PER341>3.0.CO;2-0First citation in articleCrossref, Google Scholar
Hachmeister, C.-D., & Grevers, J. (2019). Im Blickpunkt: Die Vielfalt der Studiengänge 2019. Entwicklung des Studienangebotes in Deutschland zwischen 2014 und 2019. CHE. First citation in articleGoogle Scholar
Heene, M., Bollmann, S., & Bühner, M. (2014). Much ado about nothing, or much to do about something? Effects of scale shortening on criterion validity and mean differences. Journal of Individual Differences, 35(4), 245–249. 10.1027/1614-0001/a000146First citation in articleLink, Google Scholar
Herrera, F., Carmona, C. J., González, P., & del Jesus, M. J. (2011). An overview on subgroup discovery: Foundations and applications. Knowledge and Information Systems, 29(3), 495–525. 10.1007/s10115-010-0356-2First citation in articleCrossref, Google Scholar
Holland, J. L. (1997). Making vocational choices: A theory of vocational personalities and work environments (3rd ed.). Psychological Assessment Resources. First citation in articleGoogle Scholar
Jedidi, K., Jagpal, H. S., & DeSarbo, W. S. (1997). STEMM: A general finite mixture structural equation model. Journal of Classification, 14(1), 23–50. 10.1007/s003579900002First citation in articleCrossref, Google Scholar
Johnson, W., & Bouchard, T. J. (2009). Linking abilities, interests, and sex via latent class analysis. Journal of Career Assessment, 17(1), 3–38. 10.1177/1069072708325738First citation in articleCrossref, Google Scholar
Kiefer, C., Lemmerich, F., Langenberg, B., & Mayer, A. (2022). Subgroup discovery in structural equation models. Psychological Methods. 10.1037/met0000524First citation in articleCrossref, Google Scholar
Klösgen,
W. (1996). EXPLORA: A multipattern and multistrategy discovery assistant. In U. M. Fayyad (Ed.), Advances in knowledge discovery and data mining. American Association for Artificial Intelligence: CA, USA (pp. 249–271). First citation in articleGoogle Scholar
Kristof-Brown, A. L., Zimmerman, R. D., & Johnson, E. C. (2005). Consequences of individuals’ fit at work: A meta-analysis of person-job, person-organization, person-group, and person-supervisor fit. Personnel Psychology, 58(2), 281–342. 10.1111/j.1744-6570.2005.00672.xFirst citation in articleCrossref, Google Scholar
Kruyen, P. M., Emons, W. H. M., & Sijtsma, K. (2013). On the shortcomings of shortened tests: A literature review. International Journal of Testing, 13(3), 223–248. 10.1080/15305058.2012.703734First citation in articleCrossref, Google Scholar
Kruyen, P. M., Emons, W. H. M., & Sijtsma, K. (2014). Assessing individual change using short tests and questionnaires. Applied Psychological Measurement, 38(3), 201–216. 10.1177/0146621613510061First citation in articleCrossref, Google Scholar
Lanza, S. T., Tan, X., & Bray, B. C. (2013). Latent class analysis with distal outcomes: A flexible model-based approach. Structural Equation Modeling: A Multidisciplinary Journal, 20(1), 1–26. 10.1080/10705511.2013.742377First citation in articleCrossref, Google Scholar
Le, H., Robbins, S. B., & Westrick, P. (2014). Predicting student enrollment and persistence in college STEM fields using an expanded P-E fit framework: A large-scale multilevel study. The Journal of Applied Psychology, 99(5), 915–947. 10.1037/a0035998First citation in articleCrossref, Google Scholar
Leman, D., Feelders, A., & Knobbe, A. (2008). Exceptional model mining. In W. DaelemansB. GoethalsK. Morik (Eds.), Machine learning and knowledge discovery in databases (pp. 1–16). Springer. 10.1007/978-3-540-87481-2_1First citation in articleCrossref, Google Scholar
Lemmerich, F., Kiefer, C., Langenberg, B., Cacho Aboukhalil, J., & Mayer, A. (2020). Mining exceptional mediation models. In D. HelicG. LeitnerM. StettingerA. FelfernigZ. W. Raś (Eds.), Foundations of intelligent systems (pp. 318–328). Springer. 10.1007/978-3-030-59491-6_30First citation in articleCrossref, Google Scholar
Mayer, A., Kiefer, C., Langenberg, B., & Lemmerich, F. (2021). Using subgroup discovery and latent growth curve modeling to identify unusual developmental trajectories. In Handbook of computational social science (1st ed., Vol. 2, pp. 244–268). Routledge. 10.4324/9781003025245-18First citation in articleCrossref, Google Scholar
McLarnon, M. J. W., Carswell, J. J., & Schneider, T. J. (2015). A case of mistaken identity? Latent profiles in vocational interests. Journal of Career Assessment, 23(1), 166–185. 10.1177/1069072714523251First citation in articleCrossref, Google Scholar
Minnis, M., & John-Steiner, V. (2005). The challenge of integration in interdisciplinary education. New Directions for Teaching and Learning, 2005(102), 45–61. 10.1002/tl.196First citation in articleCrossref, Google Scholar
NEPS Network (2021). NEPS starting Cohort 5: First-year students (SC5 15.0.0). LIfBi Leibniz Institute for Educational Trajectories. 10.5157/NEPS:SC5:15.0.0First citation in articleCrossref, Google Scholar
Nylund, K. L., Asparouhov, T., & Muthén, B.
O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling: A Multidisciplinary Journal, 14(4), 535–569. 10.1080/10705510701575396First citation in articleCrossref, Google Scholar
Nylund-Gibson, K., Grimm, R. P., & Masyn, K. E. (2019). Prediction from latent classes: A demonstration of different approaches to include distal outcomes in mixture models. Structural Equation Modeling: A Multidisciplinary Journal, 26(6), 967–985. 10.1080/10705511.2019.1590146First citation in articleCrossref, Google Scholar
Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. 10.18637/jss.v048.i02First citation in articleCrossref, Google Scholar
Shireman, E. M., Steinley, D., & Brusco, M. J. (2016). Local optima in mixture modeling. Multivariate Behavioral Research, 51(4), 466–481. 10.1080/00273171.2016.1160359First citation in articleCrossref, Google Scholar
Shireman, E., Steinley, D., & Brusco, M. J. (2017). Examining the effect of initialization strategies on the performance of Gaussian mixture modeling. Behavior Research Methods, 49(1), 282–293. 10.3758/s13428-015-0697-6First citation in articleCrossref, Google Scholar
Spurk, D., Hirschi, A., Wang, M., Valero, D., & Kauffeld, S. (2020). Latent profile analysis: A review and “how to” guide of its application within vocational behavior research. Journal of Vocational Behavior, 120, 103445. 10.1016/j.jvb.2020.103445First citation in articleCrossref, Google Scholar
Steyer, R., Mayer, A., Geiser, C., & Cole, D. A. (2015). A theory of states and traits—revised. Annual Review of Clinical Psychology, 11(1), 71–98. 10.1146/annurev-clinpsy-032813-153719First citation in articleCrossref, Google Scholar
Strobl, C., Malley, J., & Tutz, G. (2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14(4), 323–348. 10.1037/a0016973First citation in articleCrossref, Google Scholar
Thalmayer, A. G., Saucier, G., & Eigenhuis, A. (2011). Comparative validity of brief to medium-length Big Five and Big Six personality questionnaires. Psychological Assessment, 23(4), 995–1009. 10.1037/a0024165First citation in articleCrossref, Google Scholar
Vock, M., Köller, O., & Nagy, G. (2013). Vocational interests of intellectually gifted and highly achieving young adults: Vocational interests of the gifted. The British Journal of Educational Psychology, 83(2), 305–328. 10.1111/j.2044-8279.2011.02063.xFirst citation in articleCrossref, Google Scholar
von
Maurice, J. (2006). ICA-D. Deutschsprachige Version des Inventory of Children’s Activities—Revised (ICA-R, Tracey & Ward, 1998). Unpublished report. First citation in articleGoogle Scholar
Wurpts, I. C., & Geiser, C. (2014). Is adding more indicators to a latent class analysis beneficial or detrimental? Results of a Monte-Carlo study. Frontiers in Psychology, 5, 920. 10.3389/fpsyg.2014.00920First citation in articleCrossref, Google Scholar
Zeileis, A., Hothorn, T., & Hornik, K. (2008). Model-based recursive partitioning. Journal of Computational and Graphical Statistics, 17(2), 492–514. 10.1198/106186008X319331First citation in articleCrossref, Google Scholar
This research was made possible by a grant from the Federal Ministry of Education and Research of Germany (Grant ID: M527200) to Axel Mayer and Bettina S. Wiese. We gratefully acknowledge this support. Open access publication enabled by Bielefeld University.
