Abstract
Abstract. Given a consistent interest in comparing achievement across sub-populations in international assessments such as TIMSS, PIRLS, and PISA, it is critical that sub-population achievement is estimated reliably and with sufficient precision. As such, we systematically examine the limitations to current estimation methods used by these programs. Using a simulation study along with empirical results from the 2007 cycle of TIMSS, we show that a combination of missing and misclassified data in the conditioning model induces biases in sub-population achievement estimates, the magnitude and degree to which can be readily explained by data quality. Importantly, estimated biases in sub-population achievement are limited to the conditioning variable with poor-quality data while other sub-population achievement estimates are unaffected. Findings are generally in line with theory on missing and error-prone covariates. The current research adds to a small body of literature that has noted some of the limitations to sub-population estimation.
References
(1992). Overview of the scaling methodology used in the national assessment. Journal of Educational Measurement, 29, 163–175. doi: 10.1111/j.1745-3984.1992.tb00372.x
(2010). Measurement error: Models, methods, and applications. Boca Raton, FL: CRC Press.
(2006). Measurement error in nonlinear models: A modern perspective. Boca Raton, FL: CRC Press.
. (2009). Version 3.23. Princeton, NJ: ETS.
(2009). The harsher side of globalisation: Violent conflict and academic achievement. Globalisation, Societies and Education, 7, 433–456. doi: 10.1080/14767720903412242
(2009). GenItmDat Macro for SAS (Version 1) [SAS]. Princeton, NJ: ETS.
(2014). Analyses of model fit and robustness. A new look at the PISA Scaling Model underlying ranking of countries according to reading literacy. Psychometrika, 79, 210–231. doi: 10.1007/s11336-013-9347-z
(2002). Statistical analysis with missing data. Hoboken, NJ: Wiley.
(2006). Charter, private, public schools and academic achievement: New evidence from NAEP mathematics data. New York, NY: National Center for the Study of Privatization in Education, Teachers College, Columbia University.
(1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56, 177–196.
(1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133–161.
(1992). Chapter 3: Scaling procedures in NAEP. Journal of Educational and Behavioral Statistics, 17, 131–154. doi: 10.3102/10769986017002131
(2008). TIMSS 2007 Technical Report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
. (2003). Version 4.1. Skokie, IL: Scientific Software International, Inc.
(1976). Inference and missing data. Biometrika, 63, 581–592. doi: 10.1093/biomet/63.3.581
(1987). Multiple imputation for nonresponse in surveys. Hoboken, NJ: Wiley.
(2011). The impact of missing background data on subpopulation estimation. Journal of Educational Measurement, 48, 293–312. doi: 10.1111/j.1745-3984.2011.00144.x
(2014). Sensitivity of achievement estimation to conditioning model misclassification. Applied Measurement in Education, 27, 115–132.
(2010). Getting it “better”: The importance of improving background questionnaires in International Large‐Scale Assessment. Journal of Curriculum Studies, 42, 411–430. doi: 10.1080/00220272.2010.487546
(2015). Correcting measurement error in latent regression covariates via the MC-SIMEX method. Journal of Educational Measurement, Manuscript Submitted for Publication.
(2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177. doi: 10.1037/1082-989X.7.2.147
(1973). Principles and procedures of multiple matrix sampling (Vol. xviii), Oxford, UK: Ballinger.
(2009). What are plausible values and why are they useful? IERI Mongraph Series, 2, 9–36.
(2006).
The statistical procedures used in National Assessment of Educational Progress: Recent developments and future directions . In C. R. RaoS. SinharayEds., Handbook of Statistics (Vol. 26, pp. 1039–1055). Amsterdam, The Netherlands: Elsevier. Retrieved from http://www.sciencedirect.com/science/article/pii/S0169716106260322
