Highlights

Introduction

Implementation analysis focuses on the relationship between the intervention and its context. It analyzes the causal relationships between the context, activities, and effects. All intervention implementations are shaped by local dynamics and characteristics. For example, interventions will encounter some barriers or, on the contrary, some facilitators specific to the implementation context. Sometimes the intervention will be modified during its implementation. Not all components will be implemented or completely implemented due to the local conditions (Brousselle & Champagne, 2004). Implementation analysis tries to understand the context’s influence on the intervention and its effects. It provides an explanation of the implementation and the results produced. It offers complementary information to an effect analysis. It can explain why a program worked or why it didn’t work, which is equally important (King et al., 1987; Patton, 1997; Saunders et al., 2005). We often hear that it opens the “black box” to show what happened in the intervention (Saunders et al., 2005). “Unless the programmatic black box is opened and its activities made explicit, the evaluation may be unable to identify strengths or suggest appropriate changes” (King et al., 1987, p. 9). Implementation is an important evaluation subject with questions meant to understand the conditions under which an intervention can be generalized to other contexts.

In the Literature

There is no unique definition of implementation analysis in the evaluation literature (Brousselle, 2004). Some authors call it ‘implementation analysis’ and others call it ‘process analysis’. For some, it largely overlaps with ‘implementation monitoring’ but for others there is no overlap (Brousselle, 2004; Champagne et al., 2011b; Love, 2004; Rossi et al., 2019; Scheirer, 1994; Weiss, 1998).

Initially, implementation evaluation was similar to discrepancy evaluation (Provus, 1971). The idea was to determine how far the program could deviate from the ideal and at what point it could be considered an acceptable variant of the original idea, rather than a completely new program (Patton, 1997). Subsequently, authors have taken positions to define implementation evaluation. It is striking how much the objectives of this type of evaluation can vary among authors, as well as the variety of subtypes that can be distinguished. Implementation evaluation is found under two names: process evaluation and implementation evaluation (Brousselle, 2004; Chen, 2005). (Free translation, Brousselle et al., 2006, p. 64)

The field has not arrived at a consensus on what it encompasses.

Implementation Analysis Questions

Implementation analysis can cover areas such as (King et al., 1987; Love, 2004):

• Implementation context. Does the implementation context favor the implementation of the program as planned? How? What are the characteristics of the intervention implementation site, of the people involved in the program, of the clientele? What are the main differences between the different implementation contexts and how do they interact with the intervention and the results production? How does the implementation vary across different implementation sites? These last two questions will be important when the same program is implemented in different contexts.

• Intervention history and purpose. What’s the program’s implementation history? Consider its inception, key milestones, and implementation steps. What is the rationale for its implementation and its objectives? Is the intervention implemented as planned (e.g. intensity of activities, unexpected events, etc.)? What aspects of the program design needs to (or should) be changed to obtain the expected results in this implementation context? What are the facilitators and obstacles to its implementation and success? What problems have been encountered, and how do they differ across implementation sites? How are activities meeting expectations? Were beneficiaries satisfied?
• Stakeholders and implementors. Who are the main groups involved? Have they changed? What’s their level of engagement/support? Any issues with staff recruitment and retention? Are professionals well trained and qualified? Are there important external actors supporting or impeding the implementation of the intervention?
• Participants. What are the characteristics of the participants? How are they selected? Is there any issue with the selection process? With retention? Does the program reach the targeted clientele? Are potential clients rejecting or leaving the program?
• Budget and Funding. Is the program well-funded? Are the costs as planned? Is the intervention sustainable in the longer term and financially healthy? What are the risks and threats to its sustainability?
• Results. What are the successes, weaknesses, strengths? Is the intervention leading to expected results? Are elements missing for achieving results? Which components seems the most important for reaching the expected results? Do the results vary from one site to the other? Has any factor prevented the production of effects? Which components seem the most important for reaching the expected results? How do results differ according to variations of the context, such as participants, professionals, contextual dynamics?

Implementation analysis can explain why interventions may not work in some contexts yet work well in other contexts. Effect analysis is often conducted in conditions where researchers try to control the context to isolate the causal effect, usually before intervention implementation. However, many contextual elements can impede or facilitate the production of effects. For example, culture may become a factor with a major influence when it comes to implementation. Such elements can be documented when conducting an implementation analysis and show how the specific contextual characteristics may influence the production of effects, even when the intervention is fully implemented.

Evaluation Designs for Implementation Analysis

Implementation analysis provides an explanation of what works under specific conditions and identifies what does not. It’s important to gather as much information about the intervention as possible prior to doing fieldwork. Having a sense of the information already available will be essential for orienting the evaluation questions and for planning for the data collection. In particular, it is helpful to have a good description of the intervention, information about the broader context, detailed information on what was implemented and what was not, and achievement of results (King et al., 1987). Background documents or insiders’ views on implementation difficulties and dynamics will also be helpful. Implementation analysis often relies on complementary information provided by the monitoring to document discrepancies between what was planned and what has been implemented.

Assessment of program implementation requires taking into account the surrounding social systems, including characteristics of the organization in which the program is being implemented, characteristics of persons delivering the program (Viadro et al., 1997; Zapka et al., 2004), existing structures of the organizations and groups, organizational social system characteristics (e.g., interorganizational linkages, community agency partnerships; (Scheirer et al., 1995; Zapka et al., 2004), and factors in the external environment (e.g., competing events, controversy about the program, external political factors, and history and events that happen concurrently with the program (Scheirer et al., 1995; Viadro et al., 1997; Zapka et al., 2004). (as cited in Saunders et al., 2005, pp. 136-137)

Different designs can be used, according to the evaluation project objective and to the intervention implementation context.

• Process evaluation. Process evaluation aims at documenting implementation steps and dynamics. This type of evaluation “focusses on the internal dynamics and actual operations of a program in an attempt to understand its strengths and weaknesses” (Patton, 1997, p. 206). “Process evaluations search for explanations of the successes, failures, and changes in a program” including “unanticipated consequences in the full context of program implementation and development” (Patton, 1997, p. 206). The work of Langley and colleagues on process analysis and Brousselle & Champagne’s evaluation of the UNAIDS initiative in Chile can both provide insights and examples of how to conduct such analysis (Brousselle & Champagne, 2004; Langley, 1999; Langley et al., 2013).
• Comparative study. A comparative study across different implementation sites can provide information on the contextual determinants of the intervention’s implementation both in terms of its integrality and level of implementation (Champagne et al., 2011b, p. 246). Such a comparative study may benefit from existing monitoring data on process such as on fidelity and reach.
• Realist evaluation. Realist evaluation can be interpreted as a theory-based implementation analysis (Brousselle & Buregeya, 2018). Realist evaluation assesses complex programs by analyzing what works, for whom, and under what circumstances (Pawson & Tilley, 1997, 2004).

It is a logic of inquiry that illuminates the program theory underlying the inherent characteristics of program implementation (Hewitt et al., 2012; Pawson & Tilley, 1997, 2004) to investigate the generative mechanisms associated with the program (M), the contexts under which the pathways operate (C), and the ways in which outcomes occur (O) (Salter & Kothari, 2014). Context–mechanism–outcome (CMO) configurations, as outlined by Pawson and Tilley (1997, 2004), foster the examination of recurrent patterns in the midst of complex social reality through in-depth explanations of causal pathways. This helps the evaluator articulate the program theory to be investigated and test hypotheses to produce transferable advice based on that theory and to inform decisions as well as evidence-based policy-making processes (Hewitt et al., 2012). (as cited in Brousselle & Buregeya, 2018, p. 157)

Realist evaluation involves four core steps: 1) articulating the program theories to be tested; 2) collecting data to test the hypotheses; 3) testing the hypotheses; and 4) interpreting and refining them (Mehdipanah et al., 2015; Pawson & Tilley, 1997, 2004; Ranmuthugala et al., 2011; Salter & Kothari, 2014). It uncovers underlying implicit or explanatory theory leading to the program and its multiple components, and it identifies contextual factors that spearhead pathways of change to produce expected outcomes (Jagosh et al., 2015; Pawson, 2002; Pawson & Tilley, 1997, 2004; Ridde et al., 2012; Salter & Kothari, 2014). (as cited in Brousselle & Buregeya, 2018, p. 157)

Impacts on Planetary Health

Analyzing the intervention in relation to its implementation context, including environmental conditions, will provide an opportunity to discuss implementation analysis questions related to planetary health dimensions and intervention’s impacts. These questions could explore how environmental disruptions influence the intervention’s effects, as well as explore the intervention’s impacts on both environmental and human systems.

Climate events affect many communities and vulnerable groups and disrupt interventions’ activities. For example, heatwaves affect people’s day-to-day lives, droughts reduce crop yields and water supply, smoke episodes affect outdoor activities and physical health, and forest fires displace whole population groups. The list could be extended. Engaging in discussions about disruptions associated with potential climate events may raise awareness and motivate preparedness which will ultimately increase the capacity to adapt the intervention to this new state of the world.

Additionally, the impacts of the intervention itself on human and natural systems can also be discussed. As we have seen, impacts on planetary health are tightly linked to the context’s characteristics. When conducting an implementation analysis, it makes sense to discuss and explore the intervention’s larger impacts, their range and variety, to document these in the logic model, and to further scrutinize the intervention’s role in mitigating negative impacts and fostering positive impacts. Adopting planetary health lenses when designing interventions will help create positive, regenerative impacts on natural and human systems. Furthermore, it will avoid the shortcomings of siloed approaches that ignore dimensions that are not directly related to the intervention, but which place natural and human systems at greater risk.

Implementation analysis is a type of analysis which often involves data collection from a wide range of actors: from participants to professionals to program commissioners. It constitutes a good opportunity to include planetary health lenses into your evaluation and to engage in group discussions.

Conclusion

Understanding how the context influences the intervention implementation will potentially provide an explanation about why an intervention was either not implemented or not implemented as expected, what worked and what worked less, and the difficulties experienced. It will also provide information for those interested in implementing the same intervention in other contexts. Adopting planetary health lenses and embedding questions in the implementation analysis design related to the environmental and social contexts might bring a prospective thinking that could help implementers prepare for future dynamics which may be very necessary in today’s fast changing context.