Defining and Evaluating Fidelity at the Program Level in Psychosocial Treatments
A Preliminary Investigation
Abstract
One challenge to research on the implementation of effective psychosocial treatments is how to define and measure fidelity at the program level. The purpose of this study was to evaluate an approach to defining, measuring, and observing over time fidelity at the program level for Multisystemic Therapy (MST). For this study, program fidelity was conceptualized as a program’s performance on several key areas identified in the literature as contributing to sustainability. A composite index, the MST Program Performance Index (PPI), was developed using data from the MST quality assurance system. The PPI included indicators of treatment adherence, treatment completion, program operations, program capacity, clinical supervisor leadership, and stakeholder relationships. A PPI score was calculated for 496 MST teams every 6 months for a two year period, during which time, the teams served over 25,000 young people. The predictive validity of the PPI score was supported by both client- and team-level outcomes. Specifically, youth treated by teams with lower PPI scores were more likely to be rearrested during their course of treatment. Likewise, teams with lower PPI scores were more likely to close during the time period covered by the study. Analysis of scores over the two year period showed substantial within-team variability in the PPI; however, the scores did not follow a linear pattern of change. In summary, preliminary evidence suggests the PPI may be a useful tool to index program-level fidelity for comparative purposes and as an additional tool for a decision support system.
References
(2011). Advancing a conceptual model of evidence based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23. doi: 10.1007/s10488-010-0327-7
(2011). Measurement of fidelity of implementation of evidence-based practices: Case example of the IPS Fidelity Scale. Clinical Psychology: Science and Practice, 18, 126–141.
(1995). Assertive outreach for frequent users of psychiatric hospitals: A meta-analysis. Journal of Mental Health Administration, 22, 4–16.
(2003). Treating chronic violent offenders: Advances made through the Oregon multidimensional treatment foster care model. Washington, DC: American Psychological Association.
(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.
(1995). Dissemination of assertive community treatment programs. Psychiatric Services, 46, 676–687.
(2004). How to spread good ideas: A systematic review of the literature on diffusion, dissemination, and sustainability of innovations in health service delivery and organization. Report for the National Coordinating Centre for NHS Service Delivery and Organisation Research & Development (NCCSDO). London: NCCSDO
(2003). The program of assertive community treatment: Implementation and dissemination of an evidence-based model of community-based care for persons with severe and persistent mental illness [Special series]. Cognitive and Behavioral Practice, 10, 290–303.
(2011). Efficacy studies to large-scale transport: The development and validation of Multisystemic Therapy programs. Annual Review of Clinical Psychology, 7, 351–381. doi: 10.1146/annurev-clinpsy-032210-104615
(2009). Multisystemic therapy for antisocial behavior in children and adolescents (2nd ed.). New York, NY: Guilford.
(2004). The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth, 4, 83–105.
. (2009). 2008 MST Data Report. Retrieved from www.mstinstitute.org
. (2011). Multisystemic Therapy Organizational Manual. Charleston, SC: .
(2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75, 829–841. doi: 10.1037/0022-006X.75.6.829
(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65–76.
(2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
(2009). HLM 6.08 [Computer software and manual]. Skokie, IL: Scientific Software International.
(2011). It’s a bird, it’s a plane, it’s … fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18, 142–147.
(2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York, NY: Oxford University Press.
(2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science, 7, 17 doi: 10.1186/1748-5908-7-17.
(1998). Program fidelity in Assertive Community Treatment: Development and use of a measure. Orthopsychiatry, 68, 216–232.
