A programme theory and process evaluation of a youth development programme

Swingler, Susan

A programme theory and process evaluation of a youth development programme

Master Thesis

2020

Abstract

Background The CoolPlay youth development programme (CYDP) is an after-school sport-for development (SFD) programme with an integrated social-emotional learning (SEL) component delivered to schools in low-income communities in Cape Town, South Africa. The CYDP is implemented by CoolPlay, an SFD organisation funded by the Laureus Sport for Good Foundation. The aim is to provide youth with pro-social after-school activities that equip them with the social-emotional competence fundamental to becoming responsible and socially engaged citizens. The main programme stakeholders include the CoolPlay board of trustees, an operations manager, four area managers and 26 sport and SEL coaches (Champions). In 2018, I was appointed as a monitoring and evaluation (M&E) intern within the organisation. In my capacity as an M&E intern, I conducted an evaluability assessment (EA), which determined the scope and approach of the evaluation for this dissertation. Evaluation Focus The objective of the evaluation was to determine: (a) whether or not the CYDP can realistically produce the intended outcomes, and (b) the extent to which the programme is implemented in line with quality parameters identified in the literature. As such, a theory evaluation and a process evaluation were conducted sequentially to address the following evaluation questions: 1. What is the theory and logic underlying the CYDP design? 2. Is the programme theory and logic plausible? 3. What elements of the CYDP theory could be modified to maximise intended outcomes? 4. Do the Champions implement the CoolPlay sport sessions with sufficient quality? 5. What contextual factors may be influencing the implementation of the CoolPlay sport sessions? 6. Are the Champions adequately trained to deliver the CoolPlay sport sessions? 7. Is there adequate organisational support in place to facilitate implementation of CoolPlay sport sessions? Methodology The evaluator used Donaldson's (2007) five-step procedure for constructing and assessing the programme theory. An initial programme theory was constructed using programme documentation and focus group data derived from a purposive sample of seven programme stakeholders (evaluation question 1). The plausibility of the initial programme theory was assessed by means of an extensive literature review (evaluation question 2). The programme theory was critically reconstructed using Brouselle and Champagne's (2011) logic analysis procedure (evaluation question 3). The process evaluation used a combination of quantitative and qualitative data collection, analysis, and integration methods (concurrent mixed methods design). An observational rating scale was administered to a convenience sample of 16 Champions during a routine observation session conducted by area managers to address evaluation question 4. A purposive sample of three area managers and a stratified randomised sample of four Champions participated in interviews to elicit the contextual factors influencing implementation (evaluation question 5), the adequacy of Champion training (evaluation question 6) and the adequacy of programme support functions (evaluation question 7). The operations manager also provided qualitative input on the adequacy of programme support functions through an interview. The qualitative data was analysed using Braun and Clarke (2006)'s procedure for thematic analysis. The interviews were supplemented by quantitative data derived from a mentor survey and an organisational survey completed by four Champions and five members of the programme management team, respectively. Key Findings and Conclusions The programme theory evaluation found that the initial programme theory and logic (evaluation question 1) is plausible, however effect sizes are likely to be low and programme effectiveness is highly dependent on developmentally appropriate content and structure, psychologically and physically safe programme environment, staff characteristics, parent/caregiver involvement, and youth participation and engagement (evaluation question 2 and 3). Preliminary insights derived from the EA suggest that the CYDP does not meet these quality parameters/requirements fully, thus highlighting the need for programme design and implementation improvement, as captured in the critically reconstructed programme theory (evaluation question 3). The process evaluation found deficiencies in implementation (evaluation question 4) linked to schools' commitment and capacity, parent/caregiver involvement, and youths' capacity to engage (evaluation question 5). Certain aspects of the training structure and content were found to be inadequate for developing all relevant competencies/skills, and poor Champion engagement was flagged as a critical issue (evaluation question 6). Implementation of the programme was further constrained by limited organisational capacity (evaluation question 7). These process evaluation findings confirm that the CYDP does not fully meet the programme quality parameters/requirements identified in the literature. Recommendations Key recommendations unpacked in this dissertation include the following: • Alignment of programme content with appropriate developmental milestones and unique interests of programme beneficiaries. It is recommended that the content is structured in a curriculum that follows the SAFE approach to SEL facilitation. • Provision of transport, nutrition, and first aid supplies and training, as well as implementation of strategies to address deviant behaviour to ensure that programme beneficiaries are physically and psychologically safe. • Efforts to hire, train and retain qualified Champions. These include adherence to documented selection criteria, a more comprehensive induction process, ongoing training and support including collaborative planning, debriefing and provision of mental health services. • Implementation of parental and teacher involvement strategies, such as invitation to CoolPlay meetings and workshops, distribution of newsletters, and provision of transport to and from CoolPlay events. • Implementation of M&E systems for participant enrolment, attendance, drop-out and engagement. • Allocation of resources toward securing qualified, full-time staff with experience in positive youth development. • Strengthening of programme quality before the CYDP is rolled out to additional sites, given that the programme has the potential to support both positive and negative developmental outcomes in youth.

Keywords

Programme Evaluation

Reference:

Collections

Masters

Full item page