Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Open Access

No Loss, No Gain? COVID-19 school closures and Swiss fifth-graders' competencies and self-concept in mathematics

Published Online:June 21, 2023https://doi.org/10.1024/1010-0652/a000366

Abstract

Abstract: COVID-19-related school closures in spring 2020 interrupted learning routines and posed a great challenge for students' competencies acquisition and self-concepts. Concerns about possible learning losses due to school closures, especially for disadvantaged students, are justified, but the currently available empirical evidence is still scarce, varies greatly with regard to context, and neglects reciprocal effects of competencies and self-concept. To address these shortcomings, this paper first provides insights on the effect that the 8 weeks of school closures had on Swiss primary school students' math competencies. IRT-based math tests were used to cross-sectionally compare data from 1,299 students in Grade 5 in late spring 2020 shortly after the reopening of schools with data from a previous year's cohort (n = 11,314) using propensity score matching. The results revealed no significant differences in math competencies and no evidence of an increase in inequality when children with not German (vs German) as their first language were studied. Second, changes in math self-concepts in the school year 2019/20, when the pandemic first hit, as well as reciprocal effects of math competencies and math self-concept were assessed longitudinally (n = 1,299) using random intercept cross-lagged panel models based on three measurement points. Results showed that higher math self-concept and positive change in math self-concept over the time of school closures were related to higher learning gains. Different development trajectories for children with German (vs not German) first language emerged. The study therefore fosters a better understanding of the effect that pandemic-induced school closures had on learning and relativizes the feared negative effects on math competencies caused by short school closures.

No Loss, No Gain? Mathematische Kompetenzen und Selbstkonzepte von Schweizer Fünftklässler_innen in Zeiten der COVID-19-Schulschliessungen

Zusammenfassung: COVID-19-bedingte Schulschliessungen im Frühjahr 2020 unterbrachen Lernroutinen und stellten eine große Herausforderung für den Erwerb mathematischer Kompetenzen dar. Befürchtungen über mögliche Lernverluste aufgrund von Schulschließungen, insbesondere für benachteiligte Schüler_innen, sind berechtigt, aber die derzeit verfügbaren empirischen Belege sind noch spärlich, variieren stark je nach Kontext und vernachlässigen die Wechselwirkungen zwischen Fachkompetenzen und Selbstkonzept. Um diese Desiderate zu bearbeiten, werden in einem ersten Schritt die Auswirkungen der achtwöchigen Schulschliessung auf die mathematischen Kompetenzen von Schweizer Primarschüler_innen querschnittlich untersucht. Mit Hilfe von IRT-basierten Mathematiktests wurden 1,299 Schülerinnen und Schülern der 5. Klasse im späten Frühjahr 2020, kurz nach der Wiedereröffnung der Schulen, mit einer Vorjahreskohorte (n = 11,314) mittels Propensity Score Matching verglichen. Die Ergebnisse zeigten keine signifikanten Unterschiede in den mathematischen Kompetenzen und keine Hinweise auf eine Zunahme der Ungleichheit beim Vergleich von Kindern mit deutscher (vs. nicht deutscher) Erstsprache. In einem zweiten Schritt wurden im Längsschnitt (n = 1,299) Veränderungen des mathematischen Selbstkonzepts über drei Messzeitpunkte im Schuljahr 2019/20 sowie Wechselwirkungen mit den mathematischen Kompetenzen mittels Random intercept cross-lagged panel Modellen untersucht. Die Ergebnisse zeigten, dass sowohl ein höheres mathematisches Selbstkonzept als auch eine positive Veränderung des mathematischen Selbstkonzepts über die Zeit der Schulschließungen mit höheren Lernzuwächsen verbunden waren. Zusätzlich zeigten sich unterschiedliche Entwicklungsverläufe für Kinder mit deutscher (vs. nicht deutscher) Erstsprache. Die Studie trägt somit zu einem besseren Verständnis der Auswirkungen von pandemiebedingten Schulschliessungen auf das Lernen bei und relativiert für kurze Schulschliessungen die befürchteten negativen Auswirkungen auf die mathematischen Kompetenzen.

Introduction

In spring 2020 the COVID-19 pandemic resulted worldwide in an extraordinary range of government responses, including school closures of very different lengths (Hale et al., 2021). Within days, classroom teaching changed to distance teaching and learning, whereupon researchers, politicians, and educators feared learning losses and an increase in social inequality (Hammerstein, 2021; OECD, 2021; Voss & Wittwer, 2020). In Switzerland school closures lasted 8 weeks (including 2 weeks of spring break), a short period compared to other countries (Hale et al., 2021). Nevertheless, surveys showed that many Swiss students and parents feared learning losses (Helm et al., 2021, Garrote et al., 2021). Catch-up courses were advertised, media talked of a “lost generation” with lower competence (Silberschmidt, 2021), and many countries granted funds for catching up on learning (Helbig, 2021). But what empirical evidence has become available? Some systematic literature reviews provided a broad overview on studies on learning losses (e.g., Donnelly & Patrinos, 2022; Hammerstein et al., 2021) and uncovered a thin database with no consistent findings for short school closures: Some studies found evidence of student learning loss among subgroups of students and in some school subjects, but others even found increased learning gains. In addition, most studies focused on cognitive skills and neglected motivational factors that are strongly intertwined with skill development, such as self-concept, even though they are especially important in challenging circumstances (Eccles & Wigfield, 2002; Marsh & Martin, 2011).

We therefore assessed math competencies and self-concepts of 1,299 students in Grade 5 at several time points during the school year when the pandemic first hit, including one measurement point in late spring 2020 when schools reopened after the closures. This allowed us to compare the level of math competencies cross-sectionally with data from a previous year's cohort (n = 11,314) and also enabled us to look at the change and reciprocal relationship of competencies and self-concept in math longitudinally over this exceptional year.

The aims of this study are twofold: First, we want to add to the quantity of studies on the effects of school closures on math competencies, while also considering differing effects on disadvantaged subgroups. With the extraordinary range of government responses worldwide, Switzerland serves as an example of education systems that closed schools for a short time only, which creates a completely different situation than longer school closures of several months to years. Second, we take an exploratory look at the change in and reciprocal effects of math self-concept and math competencies in the first school year in which the pandemic hit. A greater quantity of studies, with different geographical focuses, age groups, and background variables, will give educators, policy makers, and researchers a more nuanced picture on learning losses due to the pandemic.

How did COVID-19-related school closures affect math competencies?

The global pandemic hit schools in Europe early in the second semester of the academic year 2019/20 (Hale et al., 2021). The resulting primary school closures with regionally varying designs of distance learning at home lasted 8 weeks in Switzerland (Swiss Conference of Cantonal Ministers of Education (EDK), 2020b). All children had to cope with reorganized school and teaching structures (Fickermann & Edelstein, 2021; Voss & Wittwer, 2020) and dramatically changed family and leisure structures (Blume et al., 2021; Helm et al., 2021). Thus, learning losses are expected generally, as learning moved to the home, where school input became less important and family input increased. Various studies on the home environment during school closures found that children spent less time on tasks (e.g., Grewenig et al., 2021; Helm et al., 2021), had less external regulation by teachers and peers (Blume et al., 2021; Garrote et al., 2021), or less learning process support (Garrote et al., 2021). These studies also identified positive effects on learning during lockdown (e.g., higher motivation, greater autonomy, stronger focus on basic math principles), but due to the lack of a control group that could provide information on achievement in the absence of school closures, it is impossible to determine whether the reported effects were caused by the lockdown. For this, there is a need for studies that compare competencies after school closures with a cohort from the previous year.

Regular competencies testing provides researchers with information on where student learning should be year after year. Some research studies that focus on learning losses due to school closures are already available for various regions, school subjects, and levels. Donnelly and Patrinos (2022) conducted a thorough analysis of recorded learning loss evidence documented from March 2020 to March 2021. Their systematic review identified thousands of articles on learning losses during the pandemic; however, the majority of these pertained to hypothesizing or predicting learning loss (e.g., Azevedo et al., 2021), and only eight studies conducted student analyses and reported impacts on learning as a result of COVID-19 school disruptions. Five of the eight studies looked at primary school students with school closures of 8 to 10 weeks, similar to Switzerland. A recently published study by Schult et al. (2022) and our own literature review located only two further quantitative studies that met these criteria (Depping et al., 2021; Spitzer & Musslick, 2021). Some of the studies found small learning losses in math achievement when compared with previous cohorts (e.g., d = .05 to .09, Engzell et al., 2021; d = .03 to .09; Schult et al., 2022), small to medium learning loss (d = .19, Maldonado & De Witte, 2021) or no learning losses (Gore et al., 2021; Depping et al., 2021). A more detailed look at the data revealed in general that if learning losses occurred, they emerged for disadvantaged schools (e.g., Gore et al., 2021 for one cohort) or disadvantaged children, e.g., children with low socioeconomic status or at the bottom of the achievement spectrum (e.g., Moscoviz & Evans, 2022; Schult et al., 2022). In other groups, achievement gains were even possible (Gore et al., 2021). The shift from school-based learning to distance learning at home is seen as the main reason for the reported increase in inequality (Grewenig et al., 2021; Steinmayr et al., 2021), as the extent to which families compensate for reduced school inputs likely depends on their background (e.g., parents' time and cognitive and pedagogical skills, and the home environment). One study on learning progress in math achievement during COVID-19 distance learning is available for Switzerland. Tomasik et al. (2020) found that math learning gains in primary school students using an adaptive computer-based test system was more than twice as high 8 weeks before than 8 weeks during school closures. Additionally, data revealed an increase in interindividual differences, which might indicate increased social disparities. In contrast, a similar study by Spitzer and Musslick (2021) with students in Germany (Grades 4 to 10) who computed math problem sets in an online learning environment found that students' performance was better during the school closures than the year before. Additionally, they found a narrowing performance gap between low- and high-achieving students. Explanations for the differing results in the various studies range from different test designs and content, timing of the tests, the quality of the school systems or quality of teachers, and cultural differences, to different technical equipment in the countries. Without additional studies, no reliable conclusions can be drawn.

In summary, the few findings up to now underline the need for more empirical studies on math learning during primary school closures. Existing findings suggest that short school closures of 8–10 weeks resulted in minor learning losses, mainly in disadvantaged children. Nevertheless, parents, children, and teachers reported that children learned less during this time of distance learning than in classroom instruction (e.g., Garrote et al., 2021; Helm et al., 2021). Such self-reports during school closures on competences should not be neglected, as from the point of view of educational psychology the relevance of motivational factors for learning is undisputed (Marsh et al., 2019; Schunk & Greene, 2018). This raises the question as to how students' self-concept developed during the lockdown and how this affects math skills development.

How did COVID-19-related school closures change self-concept in mathematics and its reciprocal relationship with math competencies?

An individual's math self-concept is an important predictor of math achievement (Marsh et al., 2019; Mews & Pöge, 2019) and includes a person's general assessment and evaluation of their own math abilities based on past experiences, social comparisons, perceptions of their abilities, and feedback from tasks and from others (Marsh et al., 2019; Valentine et al., 2004). Math self-concept declines in mean level over primary school and shows high interindividual stability over one school year (e.g., Weidinger et al., 2017). According to the self-enhancement approach, the self-concept is a determinant of academic achievement, whereas the skill development approach assumes that it is academic achievement that primarily influences the academic self-concept. There is empirical evidence for both approaches (self-enhancement, e.g., Marsh & Martin, 2011; skill-development, e.g., Helmke & van Aken, 1995). A variety of longitudinal panel studies therefore point to a reciprocated relationship (i.e., Ehm et al., 2021; Marsh & Martin, 2011; Praetorius et al., 2016; Weidinger et al., 2017). Although the effects of math achievement on math self-concept can be demonstrated repeatedly, the evidence for the reverse effects is somewhat smaller. Especially in primary school, the self-enhancement effect is often rather weak (Ehm et al., 2021). However, Weidinger et al. (2017) found changes toward a reciprocal effects model from Grade 2 to Grade 4. Other studies reported enhanced self-enhancement effects when demands are high (Helmke & van Aken, 1995; Pinxten et al., 2010; Praetorius et al., 2016). This is in line with self-regulation theories, where self-concept as a central motivational orientation gains importance for successful learning processes in challenging situations when routines are no longer sufficient (Landmann et al., 2009; Pelikan et al., 2021). Presumably, school closures during COVID-19 represent such challenging situations. This emphasizes the importance of looking not only at math achievement but also at possible changes in the development of math self-concept and the effect on math competencies during the first COVID-19 wave.

A handful of studies with university or secondary school students reported that a high self-concept during distance learning was predictive of various outcomes (e.g., Pelikan et al., 2021; Chen et al., 2022). But due to the non-comparable samples and the missing control group (a problem that all studies on COVID-19 share), the results cannot provide any information on possible changes in primary school students' math self-concept during the lockdown and the resulting effect on math achievement. A conference paper by Johann et al. (2021) analyzed the math self-concept of 67 children at different primary schools (Grades 2 to 4) and reported that a higher math self-concept during lockdown but not before lockdown was associated with better arithmetic performance in June 2020. In summary, the limited evidence on self-concept and achievement during lockdown suggests that self-enhancement effects may have played a role. The important question now is what effect the school closures might have had on math self-concept. Usually there is a substantial proportion of stability in self-concept (e.g., based on math competencies, or stable environment). Variations in the self-concept across time can be attributed to changing environmental factors, such as achievement feedback or social comparison (e.g., Jansen et al., 2020), but under normal circumstances math self-concepts show very few intraindividual changes, especially in the second semester of a primary school year (Weidinger et al., 2017). On the one hand, the drastic changes in the school environment caused by the pandemic and feared learning losses caused students to doubt their own competencies (e.g., Garrote et al, 2021), which could result in declining self-concepts. This was possibly particularly in disadvantaged families, as they were less able to compensate for the lack of learning process support by the teacher (e.g., less feedback and direct contact; Steinmayr et al, 2021), and less external regulation by teachers and peers (Blume et al., 2021; Garrote et al., 2021; Steinmayr et al., 2021; Voss & Wittwer, 2020). On the other hand, the ability self-concept is also shaped by social comparisons. Due to fewer learning periods in the classroom, opportunities for social comparison decreased or shifted (Voss & Wittwer, 2020). Therefore, lower-performing students may even have experienced an increase or stabilization of their self-concept due to fewer opportunities for comparison with high-performing peers, as described in the “big-fish-little-pond-effect” (Marsh et al., 2008). In addition, the omission of grades on the report card at the end of semester might have led to a less competitive environment and a more individual reference norm, which has been found to be an important factor in maintaining high ability self-concepts (Dickhäuser et al., 2017).

In summary, it can be assumed that there is a reciprocal relationship between math self-concept and math competencies in Grade 5, with more data supporting skill-development effects. But it can also be assumed that the greatly changed learning situation might have influenced the development of self-concept as well as the reciprocal effects with math competencies. Due to the lack of empirical evidence, we can only speculate about the direction and strength. In order to support children's learning processes during possible future school closures, knowledge about the interactions between self-concepts and competencies is essential.

Research questions

The goal of this study was twofold: first, to provide further insights into possible learning losses due to COVID-19 school closures by comparing fifth-graders after the first lockdown with those from the previous pre-pandemic year. Given the evidence presented, school closures have varying effects on different groups; we therefore look at the distributions and consider possible differential effects based on children's first language. In the German-speaking part of Switzerland, children with German as a second language, the majority of whom have a migrant background and often have parents with low levels of formal education, show on average an accumulation of low social, cultural, and economic capital (Moser et al., 2017). In this way, we extend the international findings discussed above and contribute additional insights into possible inequality issues. Second, we wanted to trace reciprocal developmental trajectories in math self-concept and math competencies longitudinally over the course of the year of the first lockdown. This dual perspective allows us not only to look at differences in population means by comparing different cohorts and subgroups but also with a developmental perspective to look at inter- and intraindividual changes in math competencies, math self-concept, and reciprocal effects over the course of the challenging year. Our two main research questions were as follows:

(RQ1) How do the math competencies of fifth-graders right after the COVID-19 school closures in late spring 2020 differ from the math competencies in a previous year's cohort in late spring 2019? Lower math competencies for the 2020 (vs. 2019) cohort (1a), more pronounced negative effects for children with not German (vs. German) as their first language (1b), and a greater variability in math competencies in 2020 (1c) are expected.

(RQ2) How do math self-concepts develop and interact with math competencies over the school year 2019/20 when there were COVID-19-related school closures? Due to a lack of control groups in this challenging school year, we explore changes in math self-concept as well as the reciprocal effects between math competencies and math self-concept using an exploratory approach.

Method

Sample and study design

In the German-speaking part of Switzerland, pandemic-related school closures started on March 13, 2020 (Bundesrat, 2020). As a result, end-of-year report cards contained a COVID-19 notation (EDK, 2020a), and some regions omitted grades. The curriculum (Lehrplan 21) remained the basis of instruction in primary schools, but regions, schools, and teachers were given autonomy in methods, selection of content, and organization of distance learning. Gradual return to face-to-face instruction started on May 11, 2020 (EDK, 2020b).

Our main sample in the year of the school lockdown (2019/20) consisted of n = 1,299 students in Grade 5 in n = 128 Swiss German classes (M = 11.79 years, SD = .48), in 59 public primary schools. The schools were located in small-to medium-sized agglomerations in the German-speaking part of Switzerland and participated on a voluntary basis. In Switzerland a large majority of students (96%; Federal Statistical Office [FSO], 2022) attend public schools (including students with special educational needs). The schools in the main sample differed in size (number of staff: M = 25.8, SD = 16.0), regional context (1 = rural to 9 = urban; median = 4), and socioeconomic background of the local community (welfare ratio (in%): M = 2.26, SD = 1.7; taxable income (in CHF): M = 33,971, SD = 10,924; Min = 16,183, Max = 64,735), with no significant differences in school characteristics compared to national data (FSO, 2020). In line with ethical standards, parents and children could refuse participation in the study without giving any reason. From the 1,612 of children in Grade 5 reported by the principals at the beginning of the school year, we received informed consent to participate from 80.6% of the parents. At three measurement time points at the start (t1, from 8/23/2019 to 10/14/2019), in the middle (t2, from 01/13/2020 to 02/26/2020), and at the end of school year (t3, from 5/15/2020 to 7/1/2020), we assessed math self-concept and sociodemographic variables via paper-pencil surveys. Math tests were administered at the start (t1) and end (t3) of the school year using items from standardized math tests (Check P5; Institut für Bildungsevaluation, 2020).

We compared our sample with a reference data set of 13,341 students in Grade 5 in the northwestern German-speaking part of Switzerland from the year before the pandemic. Data was assessed in the school year 2018/19, as part of yearly regular region-wide assessments (from April 29, 2019 to May 15, 2019). The standardized math tests (Check P5; IBE, 2020) are compulsory in these regions at the end of primary Grade 5 as an independent low stake assessment, but testing was suspended in the year of the pandemic.

Propensity score matching (PSM) was applied to reduce sampling bias and help strengthen the argument that possible differences in math competencies are based on the impact of COVID-19, as due to the pandemic we had only a quasi-experimental design. As matching variables (see Table 1), we used sociodemographic background variables on children and classes known to impact math achievement, self-concept, and learning (e.g., Moscoviz & Evans, 2022). MatchIt package in R was used to apply the nearest-neighbor 1-to-1 matching algorithm (Ho et al., 2013; Randolph et al., 2014), which matches each child in the main sample with one child from the reference data set whose propensity score is closest (see electronic supplementary material [ESM] 1, Appendix A for distribution of propensity scores). One child was randomly selected if multiple children in the reference sample had similar propensity scores to the treated subject. The sample size decreased due to propensity score matching (PSM), as only children with valid covariate values could be matched (Table 1 shows means and frequencies before and after matching). Missing completely at random could not be assumed due to missing values from whole classes assessed by teachers or principals. However, missingness analyses conducted with the R package naniar raised no concern. Nonetheless, analyses were conducted with matched samples and total samples (see ESM 1, Appendix B).

Table 1 Sample size and descriptives of covariates before and after matching

Math competencies

Math competencies were assessed using items from the item bank of the Check P5 (IBE, 2020). The items belong to one of the three competency areas of the national mathematics curriculum in the German-speaking part of Switzerland: number and variables; shape and space; metrics, functions, data, and probability. They therefore show strong curricular validity (IBE, 2020). The item pool comprises around 250 calibrated items for this grade, which have been completed by up to 40,000 students to date. For the main sample, a rotating test design was applied to ensure that students did not receive identical sets of items at both time points (t1 and t3). We created four test booklet versions with overlapping item packages with the same difficulty at both time points. The item packages contained 12 to 13 individual tasks. Each package consisted of three to four randomly selected items from the three competency areas. Two item packages made one test booklet. In the reference sample all students worked through the exact same test booklet (late spring 2019). Trained individuals conducted scoring in both samples. One-dimensional item response theory (IRT) models were used to scale math competencies (van Linden & Hambleton, 1997). For the evaluation, a two-parameter logistic model (Birnbaum, 1968) was used in which in addition to item difficulty, item discrimination was estimated (De Ayala, 2009).

The TAM software package by Kiefer et al. (2017) in R (R Core Team, 2022) was used for data scaling, and weighted least square estimates (WLE) were estimated. Higher values indicate better math competencies. Expected a posteriori reliability for the total model is rEAP = 0.82, and reliability of the WLE scores is rWLE = 0.81. As the math assessments in the main sample were spread from May 15, 2020 to July 1, 2020, average individual daily learning gain was computed based on the children's individual learning gain from t1 to t3 (M = .0026, SD = .0037) and used to correct scores for the difference in time. It was verified that the instruments captured math competencies in the same way for both samples. Differential item functioning (DIF) was verified using graphical comparisons of the estimated item characteristic curves and using the empirically observed solution probabilities by person ability categories (Kiefer et al. 2017). Examination of the items revealed no differences in relative difficulty between the two samples.

Math self-concept

Math self-concept was assessed at all three time points with 3 items based on Marsh et al. (2019). It is closely related to generalized math self-efficacy (for an in-depth discussion, see Marsh et al., 2019). Children assessed statements such as “I am good at math” on a 4-point rating scale from 1 (not at all true) to 4 (exactly true). Internal consistencies of the manifest scales were satisfactory (t1: α = .797; t2: α = .809; t3: α = .815).

Sociodemographic variables

Age (in years at t3), gender (1 = girl, 2 = boy), and children's first language were assessed at the student level. Migration background, urbanity of school, highest parent education, and school type were assessed at the class level (see Table 1 for descriptives).

First language was assessed as 0 = not German as first language or 1 = German as first language. In the reference sample, the children's first language was reported by teachers. In the main sample, all children who reported speaking German with their parents always or almost always were coded as 1 (German as first language).

Migration background was assessed as 1 = classes with less/equal 50% of children with a migration background, or 2 = more than 50% of children with a migration background. In the reference sample, teachers estimated the percentage of children in their class with a migration background (both parents born abroad) as either ≤ 50% or > 50%. In the main sample, each child reported if both parents and/or the child had migrated or if the child had no migration background/one parent had migrated. Migration background at class level was then computed based on the class mean.

School type was assessed as 1 = public school, 2 = private school.

Urbanity was classified based on school district with fewer than or equal (= 1) or more than (= 2) 250,000 residents, as in the reference sample no finer-grained data was available.

Highest parent education was assessed as 1 = less/equal 50% or 2 = more than 50% of parents in class with tertiary education. Teachers or school leaders were asked about the number of children with at least one parent with a university degree (technical college, university). In the main sample, the data were recorded in percentages and in the reference sample on a 4-point Likert scale (few children, ≤ 50%, > 50%, most children).

Analyses

To analyze potential learning losses due to the pandemic cross-sectionally (RQ1), we used the data set after PSM. We compared fifth-graders' math competencies right after the COVID-19 school closure in late spring 2020 (main sample) with math competencies of the previous year's cohort in late spring 2019 (reference sample). Cohen's d was reported as standardized effect size for measuring the difference between two group means. To statistically test mean differences, we ran linear regression models in Mplus Version 8.5 (Muthén & Muthén, 1998–2016) with sample (main sample = 1, reference sample = 0) as binary predictor variable and calculated cluster-robust standard errors with the ‘type is complex' option to account for the nested data structure of children in classes. ‘First language’ was used as a grouping variable in multiple group models.

To exploratively look at changes in math self-concept and reciprocal effects with math competencies longitudinally over the school year of the first wave of the pandemic (RQ2) we analyzed data of the total main sample (before PSM). First, descriptive data analyses with manifest variables are presented. Further, all analyses in Mplus were performed with the ‘type is complex’ option, as there were meaningful differences between school classes (see ESM 1, Appendix C), with the highest values for math at t1 (ICC = .117) and the smallest differences for math self-concept at t2 (ICC = .030). To look at the estimated baseline in self-concept and its changes over the school year, latent growth curve models (LGCM) were computed. As a binary predictor variable first language (non German vs. German) was added to compare the slopes. Second, to look at reciprocal effects between math competencies and math self-concept and their respective change over the course of the year, we computed random intercept cross-lagged panel models (RI-CLPMs, Hamaker, 2018; Mulder & Hamaker, 2020). Given the three waves of data on math self-concepts, the models were identified. IRT-based math competencies were assessed at two time points and were included in the model as manifest variables. Self-concepts were included as latent variables at three time points. The RI-CLPMs decomposed each observed math self-concept score into a stable part, which measures the latent factor of a person between time points (random intercept), and a changeable part, which measures the within-person change from one time point to another as deviation above or below the person-specific mean. RI-CLPM enabled us therefore to look at intra- and interindividual differences (e.g., Ehm et al, 2019; Hamaker et al., 2015). Identically worded items were assumed to be correlated across the three measurement time points. Further, we assumed that change in self-concept is a function of time, so we imposed model restrictions such that the error correlation of closer measurement time points was set equal for each indicator. We tested and established strong factorial invariance (see ESM 1, Appendix D) over the school year (Hamaker, 2018; Mulder & Hamaker, 2020).

To test for differential effects based on first language, we performed a multiple group version of the RI-CLPM by comparing models with no constraints across the groups, with models in which the regression coefficients were constrained to be identical across the groups. Differences in model fit of the nested models with free estimation vs. restrictions were tested using the Satorra-Bentler scaled chi-square difference test (S-B χ 2-test, Satorra & Bentler, 2001). Overall attrition from t1 to t3 was small (nt1 = 1275, nt2 = 1224, nt3 = 1198; < 6.3%; see Table 2 for missings per variable), which is remarkable given the great strain put on schools after reopening in late spring. To make the assumption of missing at random more plausible, we included all sociodemographic variables as auxiliary variables in the RI-CLPM (see ESM 1, Appendix F for parameter estimates). The parameter estimates remained virtually identical. Together with the low missingness rate, this speaks for robustness of the model. For the analyses, we therefore deemed it adequate to use full information maximum likelihood estimation with cluster-robust standard errors (MLR) which allowed us to handle cases with incomplete data.

Table 2 Means, SD, range, sample size, and correlations for the main sample 2019/20

To evaluate the goodness of fit of all the tested models, we computed the chi-square test statistic, root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), and comparative fit index (CFI). Taking into account that with large samples, it is difficult to find any reasonably parsimonious models that yield p-values greater than .05, measures of fit that are insensitive to sample size were mainly considered: a CFI larger than .95, an RMSEA smaller than .05, and a SRMR smaller than .08 were taken to indicate a good model fit (Hu & Bentler, 1999). All presented results reflect standardized coefficients.

Results

Math competencies of fifth-graders after COVID-19-related school closures

To assess whether the pandemic-related challenges led to lower math competencies (RQ1), we analyzed if there was a significant mean difference between fifth-graders' competence levels in late spring 2020 after the school closures and those of the fifth-graders in the late spring of the previous year (1a). As Figure 1 shows, fifth-graders in the main sample after the lockdown (M = –0.626, SD = 1.190) showed-contrary to expectations-slightly higher math competencies than the previous year's cohort (M = –0.736, SD = 1.273, d = 0.089). Taking into account cluster-robust standard errors, this difference, tested by the effect of the cohort on math competencies, proved to be non-significant (b = .109, p = .101, n = 1718). We had to reject our hypothesis that pandemic-related challenges led to learning losses in math competencies. When we compared subgroups based on their first language (1b), for children with German as their first language again no significant differences emerged. Fifth-graders in the main sample (M = –0.548, SD = 1.183) did not differ significantly (b = –0.045, p = .314, n = 1165) from the previous year's cohort (M = –0.503, SD = 1.182, d = 0.038). Contrary to our expectations, children with not German as their first language showed higher math competencies after school closures (M = –0.794, SD = 1.191) than children in the year before the pandemic (M = –1.220, SD = 1.319, d = 0.359, b = 0.425, p = .001, n = 553). To see if school closures resulted in a broader distribution (1c), we compared the variances, but the results showed less variance in the main sample (1.415) compared to the children in the year before the pandemic (1.618).

Figure 1 Differences between fifth-graders' math competencies in late spring 2020 after the school closures and fifth-graders in spring 2019. Dashed lines represent sample means. Density estimate was slightly smoothed (value = 1.3).

Change in and reciprocal effects of math competencies and math self-concept over COVID-19-related school closures

Table 2 shows descriptive statistics and correlations for the manifest variables for the main sample (n = 1,299). Math self-concept and math competencies were significantly correlated, with a higher correlation after the lockdown (r = .446, p < .001) than at the beginning of the school year (r = .338, p < .001).

Change in math competencies. Math competencies were assessed in fall 2019 and in late spring 2020 after the lockdown. They increased from M = –1.318 (SD = 1.326) to M = –0.586 (SD = 1.290), more than half a standard deviation (d = 0.560, b = 0.707, p < .001, n = 1296). As could be expected, children whose first language was not German showed lower math competencies at t1 (M = –1.477, SD = 1.359, n = 407) than children with German as their first language (M = –1.206, SD = 1.288, n = 854, b = 0.271, p = .001) but similar learning gains, which were not significantly different from the learning gains of children with German as a first language (△ M = 0.684 vs 0.716, b = 0.032, p = .639).

Change in math self-concept. Math self-concepts were assessed in fall, winter, and late spring. One-year stabilities of the manifest self-concepts (t1 – t3: r = .638, Table 2) were in the expected range (Marsh et al., 2019). The average rate of change (slope) in the self-concept was –0.041, which indicated a linear decline in self-concept over the year (β = –.225, p < .001, model fit: χ 2(1) = 4.016, p = .045). Based on the results of the multi-group LGCM (χ 2(2) = 4.328, p = .115) with first language as a grouping variable, it can be concluded that initial levels of math self-concept did not differ significantly between children with German vs non German as their first language (mean intercept = 3.090 vs. 3.054, p = .320). Comparing the estimated slopes told us that children with non German as their first language had a significantly steeper slope (–0.069) than children with German as their first language (–0.029, p =.020). Figure 2 depicts the slopes of the estimated means and indicates that the differences in slope can be explained mainly by the steeper downward trajectory in math self-concept before the school closures.

Figure 2 Change in math self-concept over the school year 2019/20 according to first language German or first language not German.

Reciprocal effects of math self-concept and math competencies. We applied a RI-CLPM (Figure 3, syntax in ESM 1, Appendix E) with math self-concept as a latent variable at t1, t2, and t3 and math competencies at t1 and t3 as a manifest variables. Each observed math self-concept score was decomposed into a stable part (random intercept) and a changeable part, which represented deviation in math concept above or below the child-specific mean from one timepoint to another. In contrast, math competencies, which were given only for two points in time (t1 and t3), were connected autoregressively. Math t3, as controlled for math competencies at t1, could be considered a representation of gain in math competencies over the school year in the form of residualized change, i.e., relative to the general linear relationship between t1 and t3. The model fitted the data well, χ 2(38) = 56.355, p = .028, RMSEA = .019, CFI = .997, SRMR = .022 (for each path, see Table 3 for estimates, standard errors, 95% confidence intervals, and p -values). Figure 3 shows that the stable part of math self-concept was positively related to math competencies at t1 (β = .510, p < .001) and a higher learning gain (residualized change) in math at t3 (β = .224, p < .001). When we looked at the lagged effects, we saw that children's intraindividual change in their math self-concept was smaller from t2 to t3 than from t1 to t2 (when equalizing the two stability coefficients, a significantly worse model fit resulted: χ 2 (39) = 64.086, p = .007, RMSEA = .022, CFI = .995, SRMR = .030, S-B χ 2-test = 11.118, p < .001). Additionally, the positive cross-lagged path from math competencies at t1 to self-concept at t2 indicated that a positive (or negative) intraindividual change in self-concept before the pandemic depended on math competencies at t1 (β = .157, p = .031, one-tailed). The cross-lagged effect from self-concept at t2 to math at t3 was also small but significant (β = .083, p = .020, one-tailed), which meant that children's change in self-concept predicted gain (or loss) in math competencies over a school year. As we did not assess math competencies at t2, possible other explanations of this finding are discussed below. The positive correlation between math competencies and self-concept at t3 (β = .125, p = .006) indicated that children's intraindividual change in their self-concept was related to gain (or loss) in math competencies in the same direction after the pandemic. We further included time-invariant covariates such as first language, age, and gender as potential confounders of the parameters. As Mund et al. (2021) pointed out, the effects of these covariates can be constant across measurement occasions or vary over different occasions (i.e., different effects on self-concept before and after school closures). Careful model checking calls for the documentation of model results with and without covariates included as well as, whether the covariates have constant or varying effects. Across all models, parameter estimates remained very similar (see ESM 1, Appendix I for model comparison). To test for differential effects, we compared multiple group RI-CLPMs with and without constraints across children with German vs not German as their first language. The S-B χ 2-test implied that the reciprocal effects between math competencies and self-concept did not differ based on children's first language (ESM 1, Appendix G).

Figure 3 Three-waves random intercept cross-lagged panel model for math competencies and math self-concept (SC). Squares denote observed variables; circles represent latent variables.
Table 3 Standardized lagged, cross-lagged paths, and cross–sectional correlations for RI-CLPM for math competencies and math self-concept

Discussion

With the COVID-19 pandemic-related school closures in many countries in the spring of 2020, children were faced with a variety of challenges at home and at school. It is of great interest to educational policy makers, educators, and researchers to know whether students-and which students-learned less during this period and how self-concepts interacted with math competencies gains.

Math competencies: no losses, but gains?

At first glance, our results diverge from the results of many studies reporting negative effects on students' academic competencies acquisition and growing social disparities due to the COVID-19 pandemic-induced changes in school and life (e.g., Engzell et al., 2021; Maldonado & De Witte, 2021; Schult et al., 2022; Tomasik et al., 2020). The comparison with the previous year's cohort revealed no learning loss for fifth-graders in late spring 2020 when they returned to face-to-face instruction in the classroom. Additionally, learning gains in our main sample (Cohen's d = 0.56) were comparable to yearly learning gains reported in other studies that analyzed changes in math competencies over one school year without experiencing a pandemic (i.e., d = 0.61–0.71, Ditton & Krüsken, 2009; d = 0.45–0.78, Moser et al., 2017). Surprisingly, learning gains of children whose first language was not German were similar to those of children with German as their first language. Furthermore, they showed even better results on the math test than the previous year's cohort.

At second glance, comparable studies regarding duration of lockdown, measurement method, and study design also reported very small learning losses for math competencies or no learning loss at all (Depping et al., 2021; Schult et al, 2022; Gore et al., 2021). In line with those studies, possible explanations for our findings might be that distance learning during school closures in Switzerland lasted for only 8 weeks (including 2 weeks of vacation), and the impact on competencies acquisition was therefore small. Additionally, we assessed math competencies through standardized math achievement tests at the beginning and end of the school year and therefore not directly before and after the critical phase of the lockdown, as was the case in Engzell et al.’s (2021) study, or during the lockdown, as in a study by Tomasik et al. (2020). This gave teachers and children time to compensate-especially in the core subjects of math and language (Garrote et al., 2021; Helm et al., 2021), which are often considered more important and thus emphasized more by teachers than other school subjects. We know from various surveys in Switzerland and adjacent countries that teachers generally placed a stronger emphasis on reviewing and less on introducing new learning contents (Garrote et al., 2021; Helm et al., 2021; Maldonado & De Witte, 2021), which would be especially beneficial for children at risk. We can only speculate that the lack of learning losses or even gains for disadvantaged children might be explained by successful selection, adaptation, and compensation strategies by schools and teachers but also by the children's individual resources, such as self-regulation strategies and motivational factors such as self-concept (Blume et al., 2021; Garrote et al., 2021; Tomasik et al., 2020; Depping et al., 2021). Thus, in sum, our study relativizes the feared negative effects on math competencies caused by short school closures of only 6 weeks in duration.

Development and reciprocal effects of math self-concept and math competencies over the year with pandemic-related school closures

Means of self-concept decreased slightly over the school year with pandemic induced school closures and showed more intraindividual stability towards the end of the school year, which is very similar to findings from longitudinal research on primary school children in years without a pandemic (e.g., Weidinger et al., 2017). Based on our data, therefore, school closures did not have a chaotic effect on self-concept development and even slowed the downward trajectory for children whose first language is not German, who are considered as children at risk in the Swiss education system (Moser, 2017). As we do not have a control group, we can only assume that the lack of competition (e.g., due to the weaker emphasis placed on grades that semester) and less social comparison (e.g., impaired upward comparisons with peers) during distance learning at home might have helped children with sociocultural risk factors to halt the decline in self-concept. This might be also one possible factor that prevented the feared loss in math competencies in our sample, especially in children with not German as their first language. It is also plausible that self-concepts did change during lockdown but leveled to their ‘usual’ standard after the children returned to school. Further descriptive results indicate that effects of math self-concepts play a significant role for math competencies development over the year with pandemic-related school closures.

Due to the impossibility of a control group design during COVID-19, causal conclusions must be drawn with the utmost caution. Fifth-graders with higher math-self-concepts – computed as the stable interindividual part of children's self-concept over the year – showed higher math competencies in fall 2019 before the pandemic began and, more importantly, higher math learning gains during the school year. Although studies have indicated that a high self-concept is especially beneficial for academic underachievers (Butler, 2011), we found no differences in regard to first language. As different longitudinal studies often find no or only small self-enhancement effects in primary school (Ehm et al., 2021; Elbaum & Vaughn, 2001; Guay et al., 2003; Praetorius et al., 2016), our results may strengthen the argument of researchers in the field of self-regulation, who point out that a high self-concept may act as a resilience factor against learning losses and is especially important in times of crisis, when routines no longer work (Landmann et al., 2009; Pelikan et al., 2021). To pursue this argument further, we need to look at the changes in self-concept regardless of the individual mean in self-concept. The results show that children who managed to improve their math self-concept over this challenging year showed significantly higher math learning gains in late spring 2020 after the lockdown (vice versa for a shift towards lower self-concept). There was also a skill development effect of math competencies in fall 2019 on self-concept in January just before the pandemic. Together, this fits in with the proposed reciprocal models (Marsh & Martin, 2011). Because, contrary to our results, for primary school a self-enhancement effect is often not found (Ehm et al., 2021; Elbaum & Vaughn, 2001; Guay et al., 2003; Praetorius et al., 2016), measures aimed at improving self-concept in an educational context are often questioned. Based on our results, however, this would be premature. It may be that especially in challenging situations such as the COVID-19 pandemic, when children had to be more self-regulated in the open, less structured learning environment at home, the self-concept plays a more important role in the acquisition of competencies and should not be neglected. Whether our results can be transferred to other situations or motivational orientations will have to be the topic of future research. As self-regulated autonomous learning increases in regular classrooms through open forms of instruction in heterogeneous groups of students and may be more common after the pandemic-related boost in blended-learning arrangements, it seems important to raise awareness in teachers to be sensitive to students' self-concept.

Limitations and implications for further research

Some limitations should be noted. First, for understandable reasons, a control group design was not possible in the year 2019/20, and we had to deal with the impossibility to disentangle any COVID-19 effect from cohort effects-although we took the utmost care to make the two samples comparable and representative of the Swiss population of fifth-graders. Second, the reported reciprocal effects of math competencies and self-concept are only descriptive and might occur in fifth-graders also in years without school closures. As no comparable data with the same time intervals and competencies assessment is available, we cannot rule out this possibility. Nonetheless, it is important for future similar crises to be sensitive to possible stronger self-enhancement effects. Third, as math competencies was only assessed at t1 and t3, this resulted in an incomplete RI-CLPM. The significant cross-lagged effect from self-concept at t2 to math achievement at t3 might therefore also reflect a (causally inverse) positive change in math competencies at the not-assessed time point t2, which then led to higher math competencies at t3. Additionally, although RI-CLPM is state of the art, standard errors in the RI-CLPM are larger than those in the traditional CLPM, as the RI-CLPM is a model with more parameters. Further, the trait factor represents the latent average over all three measurements and therefore includes the reaction to the pandemic. Thus, panel models with more measurement time points would have been more advantageous regarding detecting changes in self-concept.

We began this article with a question: No loss, no gain? Based on our results, we cannot determine a loss in math achievement due to the 6 weeks of distance learning; possibly the change in environment even had a positive effect on the self-concept of children at risk, which seems to be a positive message. However, to talk of a gain would be inappropriate in view of the major negative impact that the COVID pandemic had on health and the economy worldwide as well as the great additional efforts that were needed on the part of the schools, teachers, and parents. Nonetheless, our study has a promising message, as distance learning caused by short-term pandemic-related school closures did not result in a negative effect on math learning in our sample. It can therefore be assumed that students, teachers, and schools regulated themselves very well in these times of crisis. Based on our results, one gain that can be drawn from the crisis is the awareness that all actors together were able to compensate for the short-term school closures in Switzerland, at least in the area of math competencies. Our study therefore puts the feared loss of learning due to pandemic challenges into perspective.

References