Highlights

Introduction

Evaluators analyze and assess all kinds of objects such as products, projects, programs, and policies. For brevity, all these objects are called interventions. What these interventions have in common is that they are all purposeful, that is, they are designed and implemented with the intention of accomplishing some goals or objectives. Interventions are always situated in a context. For our purposes, contexts include both human and natural systems.

As a first step in defining an intervention, the evaluator needs to have a good understanding of the object of analysis; this understanding needs to be shared with a diversity of actors who have an interest in the intervention and in the evaluation. Building models, such as causal models and logic models, helps clarify what the intervention is about and how it is supposed to have an impact.

Logic models are essential for defining an intervention, but they are equally valuable for engaging different actors in sharing their perspectives on the intervention and helping to define the focus of the evaluation. The creation of a logic model is typically an exercise that involves various parties, either in its conceptualization or validation. The evaluator should not view this phase as a one-person responsibility, but rather as an engaging process that fosters a shared understanding of the intervention, builds trust, and collectively orients the next steps of the evaluation project. For these reasons, some evaluators have been exploring non-linear and more creative visual representations and metaphors that are better aligned with local cultures (Lambert et al., 2024).

When reading different evaluation articles and manuals, you will encounter many different terms. These include logic model, operational model, theory of change, program theory, and change model, among others. Each term represents the intervention, but with a slightly different focus. This section reviews how to build a causal model and a logic model then discusses how they are linked. It also briefly presents system mapping.

The Causal Model

Each intervention is meant to address one problem or more. A first helpful step is to understand the role the intervention plays in solving the problem. For this step, building a causal model can be useful. A causal model represents the problem and its root causes that the intervention is meant to address (Renger & Titcomb, 2002).

But first, what is a cause? The study of causes and causality has structured discussions for centuries (Shadish et al., 2002).

A cause is that which makes any other thing, either simple idea, substance or mode, begin to be; and an effect is that, which has its beginning from some other thing. (Locke, 1975, p. 325, cited in Shadish et al., 2002, p.4)

However, causes are not always of the same nature. Some elements, called causes, require specific conditions to produce effects. Shadish et al. provide the example of a match, which can be a cause of a forest fire, but for this to happen, specific conditions are needed, such as something to light the match, dry weather, etc. “Many factors are usually required for an effect to occur, but we rarely know all of them and how they relate to each other” (Shadish et al., 2002, p. 5). These causes are referred to as "INUS conditions," which stands for "an insufficient, but non-redundant part of an unnecessary but sufficient condition" (Mackie, 1974, cited in Shadish et al., 2002, p.5).

The causal model should represent the factors involved in producing effects. Following Rengers and Titcomb’s advice on building a causal model, the main question you should ask at this early stage about the intervention is “why?” (Renger & Titcomb, 2002). Here are the steps to follow:

• The first step is to identify the problem(s) and state it clearly.
• Then, you need to ask why this problem exists and document its causes. Scientific or expert knowledge are the best sources of information to inform this step.
• After identifying a cause (or causes), continue to ask “why?” it’s happening to identify the determinants of the cause, as if peeling back the layers of an onion.
• Continue to ask “why?” until it feels like recording root causes is no longer fruitful or when the topic is exhausted.
• Repeat for each cause and their determinants.

Figure 6.1 presents the start to building a causal model for a Community Based Primary Health Care research initiative that combined 12 different research programs (the intervention).

Figure 6.1 How to Build the Causal Model

Source: Brousselle, A. (2015). Research program logic models: Some reflections. Presentation at the Canadian Association for Health Services and Policy Research, Montreal.

Identification of the root causes of the targeted problem will inevitably offer a simplification of the problem and its underlying causes. For example, the obesity map (see Figure 6.2) developed by the UK Government’s Foresight office (Butland et al., 2007) is a good illustration of the complexity of illness determinants at play. This mapped representation reveals the complexity of the causes of the problem more accurately than a linear representation’s offering. However, when trying to understand the potential importance of the evaluated intervention and its role in addressing the root causes of a problem the mapped representation is probably less easy to work with. The evaluator needs to determine the right balance between finding a model that is sufficiently accurate and deep, and that meets the evaluation’s practicability requirements, according to the use and role of such models in the evaluation.

Figure 6.2 The Obesity Map

Source: Butland, B., Jebb, S., Kopelman, P., McPherson, K., Thomas, S., Mardell, J., & Parry, V. (2007). Tackling obesities: future choices - project report (2nd edition), 90: https://assets.publishing.service.gov.uk/media/5a759da7e5274a4368298a4f/07-1184x-tackling-obesities-future-choices-report.pdf

Once the causal model is developed, the evaluator can identify and highlight the causes and their determinants that are targeted by the intervention, as not all causes will be addressed by the intervention (see Figure 6.1’s example, in blue).

Causal models are of great importance in evaluations that focus on the relevance of the intervention and in effect analysis.

The Logic Model

What is a logic model? Basically, it is a tool that describes the theory of change underlying an intervention, product, or policy. It characterizes a project through a system of elements that include components and connections, with context being an important qualification. (Fretchtling, 2007, p. 1)

The logic model is a visual representation of the resources, activities, and the results as well as their intended causal relationships. Results include outputs, outcomes, and impacts (Mayne, 2015). Outputs are the first results of the activities. They indicate the “quantity of service delivered” and “the first change for which the program is accountable” (Funnel & Rogers, 2011, p. 390). Outcomes are results in relation to goals and objectives (Fretchtling, 2007). Outcomes can also be unexpected (positive or negative). Impacts are generally considered more distal results or results in specific domains such as health, prosperity, or the broader environment. In a logic model, arrows matter and carry meaning: they represent causal relationships. In some models the intervention’s objectives will be indicated (usually before the resources), as well as the reach (i.e. the population targeted by the intervention).

A way to build the logic model is to start by identifying the intended outcomes of the intervention (see Figure 6.1’s blue box for example), then keep asking “how?” and build the logic chain backwards, from outcomes to resources. Existing documentation on the intervention and the knowledge of people involved in its conception, funding, or delivery will be used for this process. Once a rough sketch is drafted, the evaluator needs to work on the chain of results. Interventions have more results than the main intended effects. Interventions can contribute to different kinds of effects beyond the expected ones, including unintended or undesirable effects and impacts. All of these should be included in the logic model if they can be predicted in advance of collecting data. Furthermore, longer term impacts on natural and human systems are often ignored in logic models as they are not considered as core to the intervention, given funding and the timeframe for implementing a program or policy. However, given that all actions have environmental and human impacts, they should be systematically represented in logic models. Dimensions to consider are impacts on pollution (including greenhouse gas emissions), land and water, biodiversity, prosperity, equity, and health.

Other contextual factors should also be represented (MacDonald, 2018) as they often influence access to resources, the implementation of the activities, and the intervention’s results. Relevant factors related to governance and power relations can be documented while elaborating the plan of the intervention, when mapping the context, and when identifying groups to involve (see Figure 6.3).

Figure 6.3 Logic Model for Planetary Health

Source: Brousselle, A., McDavid, J., Curren, M., Logtenberg, R., Dunbar, B., & Ney, T. (2022). A theory-based approach to designing interventions for Planetary Health. Evaluation, 28(3), 341. https://doi.org/10.1177/13563890221107044

Logic models are built using existing documentation on the intervention and in consultation with individuals familiar with it, such as professionals involved in its design and implementation including funders and regulators.

The visual representation of logic models varies according to the characteristics of the intervention. Logic models can include stages and/or feedback loops. Components of the model and arrows linking these components are equally important. Some projects are more complex than others. Good logic models represent the intervention with accuracy, but they are also simple enough to provide a representation that is easily understood by a diversity of actors. Among the steps in an evaluation, building and communicating the logic model is often the most effective for encouraging engagement.

Use of Logic Models

Logic models and related intervention representations are central in many facets of evaluation projects; they can be used for different purposes. Mayne (2015) lists a wide range of possible uses:

Different Visual Representations of the Intervention

One aspect that Mayne did not explicitly mention is the use of logic models to engage with diverse groups or local communities. A linear logic model can be very useful for identifying evaluation questions and structuring the evaluation project. However, it may also appear dry and technical to people outside the field of evaluation. Moreover, in certain contexts, its linear nature may lead to frustration by failing to represent the real complexity of a phenomenon. Depending on the context of the evaluation, the evaluator may want to explore building more complex, non-linear, or culturally responsive models that better align with the audience's expectations and ways of thinking. When the logic model is used to communicate and engage with specific groups, the visual representation may need to be adapted. Below, we will present two contrasting approaches: system mapping, which is used to represent the complexity of an intervention and its many interconnections, and culturally significant visuals and metaphors designed to align with different cultural frameworks.

System Mapping

One recurring criticism of causal and logic models is their linearity. This characteristic presents the advantage of creating a simple representation of the intervention and clarifying causal relationships. However, some argue that it oversimplifies the intervention and its interaction with the context. In a world where accounting for the intervention’s impacts is increasingly important, finding a way to represent the interaction of the intervention in its system is attractive. Penn and Barbrook-Johnson (n.d.) make suggestions on how to build system-based causal models and theories of change. They suggest building models based on the suggestions resulting from a brainstorm activity with approximately a dozen participants. The group should start identifying a focal point, brainstorm the elements contributing to it, consolidating and connecting the different factors, and checking the connections again. The process is less linear than the one suggested above for building the causal and logic models. It offers interesting engagement possibilities and representation avenues. “The maps are causal models of a system, represented by a network of factors and their causal relations. They are almost always annotated and layered with information about the factors and their connections” (Barbrook-Johnson & Penn, 2022, p. 62). Wilkinson and colleagues (2021) point to software resources available online to support different graphic representations. The use of system mapping is an interesting avenue for exploring the complexity of the intervention and the many influences at play. However, even with system mapping, a simplification of the map may still be needed.

Incorporating Culturally Significant Visuals and Metaphors

When building trust and developing a shared perspective on the intervention is the priority, adapting the visual representation of the logic model to the audience is important. Lambert et al. (2024) provide different illustrations of what logic models using metaphors and symbols could look like. Several models incorporate animals with symbolic significance or use circular rather than linear representations (see Figure 6.4). Values are also sometimes included in these figures.

Figure 6.4 The Kārearea Diagram Stemming from the Analysis of the Collaborative Group Process

Source: Te Tira Whakamātaki, cited in Lambert, S., Heimlick, M., Marzano, M., Mark-Shadbolt, M., & Smith, V. (2024). Theory-of-Change Visuals: Using Diagrams, Metaphors, and Symbols to Communicate Complex Ideas and Get Buy-In. Canadian Journal of Program Evaluation / La Revue canadienne d’évaluation de programme, 39(2), 300. https://doi.org/10.3138/cjpe-2024-0028

Deciding to engage in this direction involves several steps and commitments. This work cannot be led by someone external to the community without proper engagement. It requires respecting protocols, meeting with the community, listening to the people, and collaborating to identify a representation that aligns with their views of the intervention while remaining respectful of their culture. Many animals, for example, hold significant cultural meaning in Indigenous cultures, and symbols cannot be used without the community's collective consent. The evaluator should also inquire about the responsibilities associated with using such symbols and ensure that shared agreements are made regarding how the symbol will be used and shared beyond the community. This co-creation process will likely be iterative, involving several rounds of discussion.

Conclusion

Logic models and causal models define the object of study, which is a first step before conducting any kind of evaluation. The process used for building the logic model is as important as the end product. Evaluators should consider this step as the first engagement activity to get to know the intervention, but also to connect with the different parties involved in the evaluation and in the intervention. This step may help surface diverging representations of the objectives and activities defining the intervention, and potential challenges, offering evaluators insight into the broader dynamics at play. Building logic models as a collective exercise will help in creating a shared understanding of the intervention, communicating and engaging different parties, and building trust among the different participants, including the evaluand and the evaluator (Funnel & Rogers, 2011).