Highlights

Introduction

Logic analysis is a theory-based evaluation approach that uses scientific and expert knowledge to evaluate the plausibility of the intervention’s theory and/or to identify promising alternatives to achieve the desired effects (Brousselle & Champagne, 2011; Rey et al., 2012).

The aim of logic analysis is to identify the best ways to get where we want to go, that is, to achieve the desired effects. Logic analysis will identify (a) the important characteristics the interventions must have to achieve the effects and (b) the critical conditions required to facilitate the implementation and produce the effects. (Rey et al., 2012, p. 63)

Numerous applications of logic analysis exist (Breton et al., 2021; Brousselle & Champagne, 2011; Brousselle et al., 2009; Contandriopoulos et al., 2015; Hudon et al., 2020; Hurtubise et al., 2020; Rey et al., 2012; Tremblay et al., 2013).

Despite the growth and potential use of logic analysis, the dimensions contributing to planetary health have not yet been integrated into this approach, mainly because awareness of intervention impacts on natural and human systems on which living species depend has been lacking. Logic analysis is founded on the integration of scientific and expert knowledge to offer guidance on the most promising approaches for reaching pre-determined objectives. Intervention impacts on natural and human systems are rarely documented in scientific articles except when interventions are presented as direct alternatives to harmful technologies. Therefore, evaluators need to add a specific step to the standard logic analysis process to analyze what the plausible impacts on planetary health will be and then integrate these aspects into their feedback and, ultimately, influence decision-making processes.

Foundations of Logic Analysis

A fundamental principle of logic analysis is to validate or identify the theory of intervention to make sure it is based on evidence and not on beliefs.

As evaluators, we should also question the validity of the intervention’s chain of action (validity of the means), and we should test the scientific plausibility of the program’s theory. In fact, it could be argued that program theory does not really reflect how the intervention produces the intended outcomes, but rather, stakeholders’ perceptions and beliefs, right or wrong, about the mechanisms that operate between the delivery of the intervention and the intended outcomes. The whole evaluation is then built on the consensus reached on participants’ beliefs and perceptions. But what do we do as evaluators if these are incomplete or, worse, if they are wrong? Can we really build valid evaluations based on the prior analysis of a program’s theory that reflects what people think, but not what the intervention does (Chen, 1990a, 1990b)? (as cited in Brousselle & Champagne, 2011, p. 69)

Program theory is:

a specification of what must be done to achieve the desired goals, what other important impacts may also be anticipated, and how these goals and impacts would be generated. (Chen, 1990, p. 43)

central to evaluation practice and has evolved as a response to black box evaluations and the related difficulty of results interpretation (Funnell and Rogers, 2011). (as cited in Brousselle et al., 2022, p. 337)

Logic analysis goes beyond drafting the program theory, it provides a process for analyzing its plausibility for achieving the expected results.

Logic analysis is useful for better understanding the intervention’s strengths and weaknesses and for analyzing whether the intervention is designed in a way that can logically produce the desired results (Champagne et al., 2009). Furthermore, it allows us to assess the strength of the causal link between the intervention and the intended effects. A strong causal chain is a precondition for conducting an effect analysis; even so, in the absence of such a chain, logic analysis can provide insights into the intervention’s potential (although that potential in no way guarantees the effects will be achieved). (as cited in Brousselle & Champagne, 2011, p. 70)

Types of Logic Analysis

Two types of logic analysis exist. Choosing which type to use depends on the objectives of your evaluation project. Direct logic analysis is a theory-based evaluation that aims at identifying the contextual conditions and the mechanisms that allow the production of effects. Reverse logic analysis is used to identify the best possible options for achieving some specific results. It is about finding the most promising theory of change for reaching specific objectives. This type of logical analysis is useful for pinpointing alternatives that could achieve the desired outcomes, thereby expanding the range of possible interventions. It is fundamentally summative in nature because, in exploring different courses of action, it can either support or challenge the intervention being examined (Brousselle & Champagne, 2011). Steps for conducting a direct logic analysis and for a reverse logic analysis are slightly different.

Steps for Conducting a Direct Logic Analysis

The direct logic analysis approach has a lot of similarities with the realist synthesis developed by Pawson et al. (2005). Direct logic analysis involves four steps.

Step 1: Building the Logic Model

The first step consists in drafting the logic model using diverse sources of data (documents, interviews with professionals, etc.). This logic model will be the basis for understanding how resources and activities are supposed to lead to the desired effects (Brousselle & Champagne, 2011). The logic model should include relevant impacts on natural and human systems (see Figure 7.3).

Step 2: Developing the Conceptual Framework

Next, the evaluator gathers expert and scientific information on effective intervention mechanisms and the contexts in which the effects are produced. This step involves identifying the knowledge domains that should be surveyed. There’s often a leap between how the logic model is drafted and how the concepts are named in the scientific literature. Related topics may span different disciplines. For example, knowledge use can be linked to knowledge transfer, lobbying, evaluation use, and more. Once those concepts and domains have been identified, the evaluator searches library databases and synthesizes the information to build a theory of change. Different domains of knowledge may be surveyed, and the synthesis exercise may require reflecting on what works in which context. Different domains may provide complementary or even contradictory information. The role of the evaluator is not to eliminate some domains of knowledge, but rather interpret the value of this knowledge in the particular contexts of study. Sometimes, the juxtaposition of domains of knowledge is necessary.

Step 3: Integrating Impacts on Planetary Health in the Theory of Change

The next step consists of integrating consideration of the intervention’s impacts on natural and human systems. As mentioned earlier, the evaluator will seldom find this information in the scientific literature consulted to build the theory of change. A dialogic step will be required to integrate these dimensions into the evaluation (Brousselle et al., 2022). The evaluator may want to create a first draft of the potential impacts based on either their own knowledge or that of relevant experts consulted. Then, the evaluator will need to consult with the people involved in the implementation of the program to identify broader potential impacts on health, equity, prosperity, pollution, biodiversity, land, and water. These impacts will vary according to implementation context, hence the need to discuss with people who are knowledgeable about how the intervention is implemented. Definitions and illustrations of these impacts might need to be provided to the participants to make sure they have a shared understanding of what these concepts encompass. The Planetary Health Rapid Impact Assessment Tool (see chapter 3 and Figure 3.1) can be used to support engagement on this aspect.

Step 4: Comparing the Logic Model to the Theory of Change

The final step involves analyzing the logic model and comparing it to the theory of change synthesized in Steps 2 and 3. Ideally this work is completed with the professionals involved in the intervention evaluated. When done collaboratively, this step should deepen the understanding of how, within a particular context, the mechanisms and elements identified in the theory of change are mobilized or not. The discussion could be organized around the potential discrepancies between the logic model and the theory of change, and on the ways the implemented intervention could be improved to increase its effectiveness.

The discussion with participants should include an engagement activity about the impacts of the intervention on human and natural systems. This dialogue should include the identification of negative impacts and a discussion on either alternatives or ways to shift the impacts from negative to positive (see Figure 7.1). By explicitly discussing these aspects and using planetary health lenses, deliberate amendments to the intervention can be made to make it a more desirable intervention.

Figure 7.1 Designing program theories for planetary health

Source: Brousselle, A., McDavid, J., Curren, M., Logtenberg, R., Dunbar, B., & Ney, T. (2022). A theory-based approach to designing interventions for Planetary Health. Evaluation, 28(3): 346. https://doi.org/10.1177/13563890221107044

Steps for Conducting a Reverse Logic Analysis

Reverse logic analysis involves similar but slightly different steps. The objective is to identify different alternatives for achieving desired results. Reverse logic analysis is particularly useful when no intervention preexists, but it could be used when an intervention already exists to broaden the range of possible actions to achieve specific results. It provides valuable information on how to design effective interventions to achieve specific, predetermined effects. Before starting, the different parties involved in the evaluation should agree on the expected results they are trying to achieve with the intervention. Reverse logic analysis uses 2 steps.

Step 1: Identifying Different Alternatives

By consulting experts and conducting a search of the scientific literature, the evaluator will identify one or more interventions—or alternative approaches—that could lead to the expected effects. The intervention(s) will be the result of synthesizing various writings, while considering the context and its influence on the production of effects. If the existing body of knowledge suggests multiple alternatives that cannot be combined, distinct interventions could be identified, followed by a deliberative process to select the intervention most appropriate for the implementation context. The goal is to identify, based on scientific and expert knowledge, one or a small number of interventions that will lead to the desired results. All effects documented in the literature review should be incorporated into the interventions' theory of change, as some interventions may also lead to undesirable effects.

As the objective of this type of evaluation is to provide evidence-informed knowledge to help make the best decision possible on what kind of intervention to implement, all relevant information should be included. This also involves documenting the potential impacts on natural and human systems. As mentioned above, these impacts will depend on the context of implementation. Even if discussions are needed with the implementers at a later stage, it is the role of the evaluator to identify the areas of environmental and social impacts at this stage; this step can be completed based on the evaluator’s own knowledge or by consulting relevant experts.

In summary, the role of the evaluator will be, first, to synthesize the information available from the scientific literature and related to impacts on planetary health. This step may lead to the formulation of one or more theories of change identifying the mechanisms and the contextual conditions by which the intervention will generate effects, as well as the full spectrum of anticipated outcomes, including those affecting environmental and human systems.

Step 2: Selecting the Most Promising Intervention

The second step involves organizing a deliberative activity with the actors in the field to identify the most relevant intervention according to the implementation context (Tremblay et al., 2013). This step involves identifying the range of possibilities and mechanisms of action based on expert and scientific knowledge, and then integrating these insights with the specific characteristics of the implementation context. The choice of intervention will depend on the contextual characteristics (Rey et al., 2012). The deliberative approach should include discussions about alternative interventions and their potential impacts on planetary health, followed by strategies to explore ways to revise the theory of change in order to shift from negative impacts to positive outcomes.

Conclusion

Logic analysis is an effective evaluation approach to contribute to the implementation of evidence-informed interventions and shift our impact towards planetary health protection, restoration, regeneration, and positive health and social impacts. This type of analysis requires important synthesis and facilitating skills. However, it is an approach that does not require a lot of resources and is quite accessible to anyone, whatever their knowledge and expertise. Engaging people in discussions about the potential intervention impacts on planetary health can have a positive, lasting contribution to the environment and human health and well-being. In the longer term, this approach may contribute to shifting the way we think about interventions.