Speaker
Description
Authors: Annie O'Brien (The University of Manchester), Margarita Panayiotou (The University of Manchester), Neil Humphrey (The University of Manchester), Joao Santos (The University of Manchester)
Background: The relationship between implementation (e.g., fidelity, quality of delivery, responsiveness) and intervention outcomes in educational settings is difficult to establish due to a plethora of assessment and reporting approaches that vary in quality. This results in difficulty interpreting and comparing study findings. In response to a recent call for more, and high quality, research examining this relationship, this study investigates whether implementation variability moderates intervention effects using CACE (complier average causal effect) estimation, a robust analytic method that models compliance yet it is criticised for reliance upon arbitrary decision making. As theory and empirical evidence indicates that each implementation dimension has the potential to influence outcomes, we adopt a multiverse approach to investigate the influence of all dimensions on student outcomes while limiting researcher degrees of freedom.
Methods: This study uses secondary data from a cluster randomised control trial of Passport, a universal, school-based social-emotional learning intervention delivered to Year 5 students in mainstream primary schools across Greater Manchester between 2022-25. Schools (k=62, N=2,425 children) were randomly allocated to intervention (k=33; N=1,264) or control (k=29; N=1,161) conditions. Teachers implemented Passport, a curriculum-based programme with 18 sequential sessions, over one academic year. Student relational outcomes (loneliness, bullying, peer support) were measured pre- and post-intervention.
Results: Multilevel intent-to-treat analysis revealed null intervention effects. Analysis is currently underway [and will be available prior to EUSPR] to see whether the ITT effect changes when modelling intervention compliance.
Discussion: This research has the potential to make theoretical, methodological and empirical advancements to the fields of implementation science and social-emotional learning by rigorously investigating which implementation dimensions moderate intervention outcomes. A multiverse approach can reveal the instability or robustness of results that hinged on data processing decisions, ensuring confidence in the validity of conclusions drawn. Implications for design and delivery of school-based universal interventions are discussed.
| Conflict of interest | The authors report no conflict of interest. |
|---|
