Developing a New Tool for International Youth Programs
The YouthPower Action Youth Soft Skills Assessment (YAYSSA)
Abstract
Abstract: We developed and evaluated the YouthPower Action Youth Soft Skills Assessment (YAYSSA), a self-report soft skills measure. The YAYSSA targets 15- to 19-year-old youth in lower resource environments. In Study 1, we identified 16 key constructs based on a review of those associated with positive youth outcomes in sexual and reproductive health, violence prevention, and workforce success. We adapted promising items measuring those constructs from existing and openly available tools. We conducted cognitive interviews with 50 youth from six schools in Uganda, for wording and response formats, leading to a first draft tool. In Study 2 we administered that tool to N = 1,098 youth in 59 schools in Uganda. Confirmatory factor analyses did not support the hypothesized 16-factor structure, but exploratory factor analyses suggested a four-factor solution (Positive self-concept, Higher-order thinking skills, Social and Communication skills, and Negative self-concept). In Study 3, a revised tool was administered to Uganda youth (N = 1,010, 59 sites). After cognitive testing with 45 youth in Guatemala, the tool was administered to youth (N = 794; 59 sites) in Guatemala once, then 5 months later, with a mixture of retested and new participants (N = 784; 67 sites). Factor analytic results supported the four-factor structure with 48 retained items and indicated that the instrument was reliable by internal consistency and test-retest correlations. The instrument correlated with demographic variables and outcomes in expected directions. We found evidence for measurement invariance across country, country and gender, country and socioeconomic status, and time. We discuss implications for scale validation and use in future research.
References
(2011).
Personality psychology and economics . In E. HanushekS. MachinL. WoessmannEds., Handbook of the economics of education (Vol. 4, pp. 1–181). Elsevier.. (2014). Standards for educational and psychological testing. American Educational Research Association.
(2002). Measuring risk and protective factors for substance use, delinquency, and other adolescent problem behaviors: The Communities That Care Youth Survey. Evaluation Review, 26, 575–601. https://doi.org/10.1177/0193841X0202600601
(2012). The retrospective pre-post: A practical method to evaluate learning from an educational program. Academic Emergency Medicine, 19(2), 189–194. https://doi.org/10.1111/j.1553-2712.2011.01270.x
(1989). Structural equations with latent variables. Wiley.
(2020). How skills shape the trajectory of life after school in Uganda: Lessons from a youth skills evaluation. Brookings. https://www.brookings.edu/blog/education-plus-development/2020/03/02/how-skills-shape-the-trajectory-of-life-after-school-in-uganda/
(2007). The impact of early cognitive and non-cognitive skills on later outcomes. Center for the Economics of Education Discussion. https://eprints.lse.ac.uk/19375/1/The_Impact_of_Early_Cognitive_and_Non-Cognitive_Skills_on_Later_Outcomes.pdf
(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
(2000). Responses to stress in adolescence: Measurement of coping and involuntary stress responses. Journal of Consulting and Clinical Psychology, 68(6), 976–992. https://doi.org/10.1037/0022-006X.68.6.976
(2010). Wellness dimensions relate to happiness in children and adolescents. Advances in School Mental Health Promotion, 3(4), 25–37. https://doi.org/10.1080/1754730X.2010.9715689
. (n.d.). School Quality Improvement Index, Student social-emotional learning. https://coredistricts.org/our-improvement-data/improvement-measures/
(1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104. https://doi.org/10.1037/0021-9010.78.1.98
(2021). Cross-country comparability of a social-emotional skills assessment designed for youth in low-resource environments. International Journal of Testing, 21(3–4), 182–219. https://doi.org/10.1080/15305058.2021.1995867
(2017). The value of soft skills in the labor market. NBER Reporter, 4, 7–11. https://www.nber.org/reporter/2017number4/value-soft-skills-labor-market
(2008). Controlling response shift bias: The use of the retrospective pre‐test design in the evaluation of a master’s programme. Assessment & Evaluation in Higher Education, 33(6), 699–709. https://doi.org/10.1080/02602930701773026
. (2014). ETS Personal Potential Index: Evaluator user’s guide. https://www.ets.org/ppi
(2014). Preliminary development and validation of the Social and Emotional Health Survey for secondary school students. Social Indicators Research, 117(3), 1011–1032. https://doi.org/10.1007/s11205-013-0373-0
(2017). Measuring soft skills in international youth development programs: A review and inventory of tools. USAID’s YouthPower: Implementation, YouthPower Action. https://www.youthpower.org/resources/measuring-soft-life-skills-international-youth-development-programs-review-and-inventory-tools
(2011). The Jamaica Youth Survey: Assessing core competencies and risk for aggression among Jamaican youth. Caribbean Quarterly, 57(1), 35–53. https://doi.org/10.1080/00086495.2011.11672393
(2016). Key soft skills for cross-sectoral youth outcomes. USAID’s YouthPower: Implementation, YouthPower Action.
(1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38(5), 581–586. https://doi.org/10.1111/j.1469-7610.1997.tb01545.x
(1998). The Strengths and Difficulties Questionnaire: A pilot study on the validity of the self-report version. European Child and Adolescent Psychiatry, 7, 125–130. https://doi.org/10.1007/s007870050057
(2017). Effects of anchoring vignettes on comparability and predictive validity of student self-reports in 64 cultures. Journal of Cross-Cultural Psychology, 48, 319–334. https://doi.org/10.1177/0022022116687395
(2006). The effects of cognitive and noncognitive abilities on labor market outcomes and social behavior. Journal of Labor Economics, 24(3), 411–482. https://doi.org/10.1086/504455
(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. https://doi.org/10.1007/BF02289447
(1995). Structural equation modeling: Concepts, issues, and applications. Sage Publications.
(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
. (n.d.). Passport to Success: Trainer observation tool. https://iyfglobal.org/sites/default/files/PTS_Curriculum_Overview_6.pdf
(2011). The errors of our ways. Journal of Educational Measurement, 48(1), 12–30. https://doi.org/10.1111/j.1745-3984.2010.00128.x
(2007). Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis, 15(1), 46–66. https://doi.org/10.1093/pan/mpl011
(2023).
Assessment of interpersonal and intrapersonal skills . In R. J. TierneyF. RizviK. ErcikanEds., International encyclopedia of education (4th ed., pp. 477–493). Elsevier. https://doi.org/10.1016/B978-0-12-818630-5.10061-2(2014).
Innovative questionnaire assessment methods to increase cross-country comparability . In L. RutkowskiM. von DavierD. RutkowskiEds., Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 277–286). CRC Press.(2014). Flourishing children: Defining and testing indicators of positive development. Springer.
(2015). Key “soft skills” that foster youth workforce success: Toward a consensus across fields. USAID, FHI 360, Child Trends. Published through the Workforce Connections project managed by FHI 360 and funded by USAID.
(2006). FACTOR: A computer program to fit the exploratory factor analysis model. Behavior Research Methods, 38(1), 88–91. https://doi.org/10.3758/bf03192753
(2009). Empirical identification of the major facets of conscientiousness. Learning and Individual Differences, 19(4), 451–458. https://doi.org/10.1016/j.lindif.2009.03.007
(2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39(3), 479–515. https://doi.org/10.1207/S15327906MBR3903_4
(2002, December 9). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Mplus Web Notes: No. 4. (version 5). http://www.statmodel.com/examples/webnote.shtml
(2015). Estimator choices with categorical outcomes. Mplus paper. https://www.statmodel.com/download/EstimatorChoices.pdf
. (2014). PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems(Vol. V). OECD Publishing.
(2022, October 31). YAYSSA supplementary materials. https://osf.io/2gj7m/?view_only=42427728142843e6b10e5b6d67931760
(2016). Development of an inventory assessing social and emotional skills in Brazilian youth. European Journal of Psychological Assessment, 32(1), 5–16. https://doi.org/10.1027/1015-5759/a000343
(2016). Can they make adolescent self-reports of social-emotional skills more reliable, discriminant, and criterion-valid? European Journal of Psychological Assessment, 32, 39–51. https://doi.org/10.1027/1015-5759/a000336
. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
(2017). A systematic review of personality trait change through intervention. Psychological Bulletin, 143(2), 117–141. https://doi.org/0.1037/bul0000088
(2017). Measurement invariance in international surveys: Categorical indicators and fit measure performance. Applied Measurement in Education, 30(1), 39–51. https://doi.org/10.1080/08957347.2016.1243540
(1995).
Generalized Self-Efficacy Scale . In J. WeinmanS. WrightM. JohnstonEds., Measures in health psychology: A user’s portfolio. Causal and control beliefs (pp. 35–37). NFER-NELSON.(2007). The Chinese Positive Youth Development Scale: A validation study. Research on Social Work Practice, 17(3), 380–391. https://doi.org/10.1515/ijamh.2006.18.3.547
(2000). Equipped for the future content standards: What adults need to know and be able to do in the 21st century. National Institute for Literacy.
(2015). The Strengths and Difficulties Questionnaire: Psychometric properties of the parent and teacher version in children aged 4–7. BMC Psychology, 3(1), Article
4 . https://doi.org/10.1186/s40359-015-0061-8(2019). Multiple-group invariance with categorical outcomes using updated guidelines: An illustration using Mplus and the lavaan/semTools packages. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 111–130. https://doi.org/10.1080/10705511.2019.1602776
(2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
(1987, August 28–September 1). Re-examining basic dimensions of natural language trait descriptors. Paper presented at the 95th Annual Meeting of the American Psychological Association, New York. APA
. (n.d.). Attachment D: Optional work readiness tool for the WIA youth program. https://wdr.doleta.gov/directives/attach/TEGL/TEGL07-10a4.pdf
(2011). Anchors: Software for anchoring vignette data. Journal of Statistical Software, 42(1), 1–25. https://doi.org/10.18637/jss.v042.i03
. (2017). The Washington group short set on functioning: Question specifications. http://www.washingtongroup-disability.com/wp-content/uploads/2016/12/WG-Document-4-The-Washington-Group-Short-Set-on-Functioning-Question-Specifications.pdf
(1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063–1070. https://doi.org/10.1037/0022-3514.54.6.1063
(1985). Toward a consensual structure of mood. Psychological Bulletin, 98(2), 219–235. https://doi.org/10.1037/0033-2909.98.2.219
(2016). Identification of confirmatory factor analysis models of different levels of invariance for ordered categorical outcomes. Psychometrika, 81(4), 1014–1045. https://doi.org/10.1007/s11336-016-9506-0
. (2022). Youth in development policy, 2022 update. https://www.usaid.gov/sites/default/files/documents/USAID-Youth-in-Development-Policy-2022-Update-508_2.pdf
. (2023). Puentes project. https://www.usaid.gov/guatemala/programs/puentes
(2018). The effects of vignette scoring on reliability and validity of self-reports. Applied Psychological Measurement, 42(4), 291–306. https://doi.org/10.1177/0146621617730389
. (2022). YouthPower2: Learning and evaluation. https://www.youthpower.org/youthpower2-learning-and-evaluation-yp2le
