Highlights

Introduction

An evaluation needs to be carefully prepared.

Designing evaluations requires making clear choices about many things including, for example: the purpose of an evaluation; the resources needed; required skills; ethical guidelines; data collection and analysis procedures; and how to encourage evaluation use. Getting these choices right at the beginning is essential to ensure any evaluation will be of good quality. (Stern, 2015, p. 10)

An evaluation project is also a dynamic process. One can expect that some of the process won’t go as planned. If there is clear understanding of the political context, agreement on the objectives, and if the evaluation project was developed with care to ensure strong coherence, one will be able to adapt the project according to changing conditions. A detailed plan that is agreed upon in advance of starting the project can be used as a compass when challenges arise. Sometimes, during the field work, one can be caught in day-to-day dynamics. The evaluation plan will help provide direction on how to maintain the project’s coherence even if some changes need to be made. “Effective evaluators learn to adapt to changed conditions” (Patton, 1997, p. 118). Below we will review elements that need to be taken into consideration when preparing for an evaluation.

Evaluation Use and the Importance of Mapping the Context

A useful evaluation is one that is influential. The field has extensively debated the importance of use and its determinants. Various types of evaluation use have been identified: (1) symbolic use is when evaluation is caught in a political game; (2) instrumental use is when results are used to contribute to policy or decision-making; and (3) process use when the process itself can be used to achieve some objectives such a empowering communities (Contandriopoulos & Brousselle, 2012). Even if the evaluation project is planned with specific expected uses in mind, some users may focus on elements of the project that we, as evaluators, did not anticipate. Of course, use can also change over time, depending on components of evaluation projects (Patton, 1997). Users vary too; they are not a homogenous group and the evaluator may have little control over the aspects of the project that are used and/or who uses them. And, in some cases, no use will occur and the evaluation project is forgotten as soon as it is finished.

Use is mainly determined by the characteristics of the context (see Figure 5.1) (Contandriopoulos & Brousselle, 2012). This explains why some evaluators experience difficulties using the results to influence the situation studied. It is not the process that seems to be the main determinant of use, but rather the characteristics of the context. Long-standing debates in the field of evaluation have centered around the issue of use, one of the most notable being the debate between Patton and Weiss, both highly respected evaluators (Patton, 1988; Weiss, 1988). Patton argued that the process was a main determinant of use, while Weiss suggested that the process could have very little impact. This debate can be better understood by analyzing the characteristics of the contexts in which Weiss and Patton worked.

Some contexts are more favourable to use, others less. “As soon as users are willing to bear most of the costs, the potential for use increases dramatically” (Contandriopoulos & Brousselle, 2012, p. 70). Collaborative and participatory approaches work well in those favorable contexts. However, these approaches are unlikely to work as expected in other types of contexts.

Low-polarized contexts in which the evaluator bears most of the costs are associated with very little potential for use. Results produced in this quadrant [as shown in Figure 5.1 and described as ‘the knowledge-driven swamp’] are likely to join the ever-growing pile of ignored advice and shelved reports. (Contandriopoulos & Brousselle, 2012, p. 70)

In practice, evaluators often specialize in a particular evaluation approach and tend to choose to work in contexts which are a fit for these approaches. Ideally an evaluator should be able to work in a diversity of contexts and choose the evaluation approach that fits the context.

Figure 5.1 Evaluation Use, Evaluation Models and Contextual Characteristics

Source: Contandriopoulos, D., & Brousselle, A. (2012). Evaluation Models and Evaluation Use. Evaluation, 18(1), 70. https://doi.org/10.1177/1356389011430371

The current post-truth context is creating extraordinary conditions and creating new challenges for evaluators.

Drafting an Evaluation Plan

“An evaluation plan is a written document that describes how you will monitor and evaluate your program, as well as how you intend to use evaluation results for program improvement and decision making” (Centers for Disease Control and Prevention, 2011, p. 1). It consists of a written document of a few pages that results from extensive consultations to find the right arrangement to fit the evaluation purpose and context.

Elaborating a plan will help you prepare but also negotiate agreement among different uses and participants on the expectations. This step can support building trust among the different actors. It can also be a safeguard for potential subsequent influence from different actors. This entire exercise begins with the expressed interest of a commissioner, who has typically identified an evaluation question. Before proposing an evaluation plan, the evaluator must conduct their own research to gain a rapid yet thorough understanding of the topic and the evaluation context.

Coherence between the different components of an evaluation plan is key. Many evaluation questions, approaches, and methods exist. Therefore, finding an adequate combination of plan elements to meet the characteristics of the context is essential (see Figure 5.2).

Figure 5.2 The Design Triangle

Source: Adapted from Stern, E. (2015). Impact Evaluation. A Guide for Commissioners and Managers. https://www.bond.org.uk/wp-content/uploads/2022/08/impact_evaluation_guide_0515.pdf

Digging into the Topic

What is this project about? What is the existing state of knowledge on the problem the intervention or similar interventions target? What is the scientific consensus and what are potential tensions or disagreements? What impacts on planetary health can be anticipated?

Conducting a short review of the scientific literature and summarizing the information is the first step to familiarize yourself with the project topic. Some topics are very broad, and you will need to narrow the scope of your search. If you have no expertise on the topic, consider consulting an expert to get guidance on the most recent scientific knowledge and on how to orient your search. Be mindful that online resources can be biased. When you conduct a web search try to prioritize sites that make scientific articles accessible publicly, for example, PubMed in the health sector (https://pubmed.ncbi.nlm.nih.gov). Widely available and peer-reviewed information ensures both credibility and barrier-free access to information for all users.

Mapping the Context

Mapping the context includes:

Identify Who Should be Involved

Identifying who should be around the evaluation project table is an important step. The evaluation should ask if some voices are currently missing and who should be invited.

Guba and Lincoln (1989) identified three main groups of stakeholders in the evaluand/intervention: the agents, those ‘involved in producing, using and implementing the evaluand’ (Guba & Lincoln, 1989, p. 40), the beneficiaries, those who ‘profit in some way from the use of the evaluand’ (Guba & Lincoln, 1989, p. 40), and the victims, those who are negatively affected by the use of the evaluand’ (Guba & Lincoln, 1989, p. 41) and we should add, those who are excluded from benefitting from the evaluand. The scope of the current environmental challenge is such that it affects all groups in the population and work needs to be done on how we identify who should be involved. Different perspectives need to be systematically introduced, such as Indigenous perspectives, but also voices from groups usually excluded from our democratic (electoral) systems such as youth (Montrosse-Moorhead et al., 2019). (as cited in Brousselle & McDavid, 2021, pp. 175-176)

Determining the Evaluation’s Focus

Once the previous information is gathered, the evaluator can, in collaboration with the commissioner and potentially other actors, discuss the purpose and intended use of the evaluation. This process supports the confirmation or formulation of evaluation question(s) and facilitates the selection of an approach that best aligns with the expectations and context.

At this stage we suggest evaluators include a discussion about integrating planetary health considerations, specifically the intervention’s impacts on human and natural systems. By leading these conversations evaluators can raise awareness of the importance of adopting a holistic evaluation approach and the potential for designing and implementing interventions with this aim. Raising this option early will facilitate getting an agreement on the type of assessment done.

International commitments exist to protect and promote biodiversity and to reduce greenhouse gas emissions (United Nations, 2015, 2022). Solutions also exist in all sectors and at all levels from the individual to community to regional, even for corporations, and governments that are establishing policies and regulations. When an organization takes steps to first assess the whole impact of their intervention(s) and then takes action to enhance its positive impacts on the environment and population, they can be positioned favorably with funders and other organizations. More organizations are becoming accountable for their program impacts (see for example Environmental, Social and Governance [ESG] investment criteria). The planetary health assessment doesn’t need to be extensive. Planetary health dimensions can be included at all evaluation stages starting with the logic model.

The approaches and tools presented in this book are intended to support evaluators in offering a planetary health perspective to their clients. The consideration of planetary health dimensions shouldn’t represent a cost for clients, as the approach and tools offered in this book do not involve specific expertise or extensive data collection. Once practiced, the evaluator will have enough experience to apply planetary health lenses to any kind of project with minimal additional effort. The evaluator could even present this layer as a bonus to the evaluation project. Organizations willing to engage in assessing their planetary health impacts will create a path for others and develop a reputation as innovators and responsible investors.

Choosing the Evaluation Design

One component of the evaluation project’s validity or credibility is the coherence between the context and choices of approach, questions, design, and methods. Many options exist. While some combinations work well, others are just not possible. For example, you won’t use the same research designs for an effect analysis and an implementation analysis. The choice of methods will be determined by your research design. Similarly, some evaluation approaches will determine the choice of questions and research design. For example, in community-based projects or Utilization-Focused Evaluations (Patton, 1997) participants or users determine the direction of the evaluation project early in the process. When working in Indigenous contexts, the entire process must be shaped according to existing protocols and with respect for the communities' cultures. Working with Indigenous peoples and communities will determine the evaluation process, particularly the choice of approach, frameworks, ownership of data, and ethical procedures (Chilisa & Mertens, 2021; Chouinard & Cram, 2020; Cram, 2018a, 2018b; Evans et al., 2020; Government of Canada, 2014). How you engage with different parties and define your own role as an evaluator will be determined by the specific context of evaluation and the type of approach that will be relevant for this context.

Once the right combination is determined and agreed on with the evaluation partners, the evaluator can continue designing the project by identifying the analysis strategy, getting agreement on the knowledge-sharing activities, the timeline, the budget and meeting the ethical approval requirements. Again, creating the evaluation plan presents an opportunity to engage with the different parties and build trust while facilitating agreement of the evaluation’s shape and its different components. The written document summarizes what should be a negotiated understanding of what the evaluation project aspires to look like.

Summary

Table 5.1 presents a summary of the different questions that need to be addressed in your evaluation plan document.

One important consideration when building an evaluation plan is that it should be as concise as possible. First, it makes no sense to spend more on the evaluation than on the program itself. Second, the evaluator should propose a plan that is simple, focusing on answering the evaluation questions efficiently.

Table 5-1 Elements to Consider for Elaborating an Evaluation Plan

Conclusion

One can’t expect an evaluation to be straightforward. Only a minority of projects will proceed without any difficulties or surprises. Evaluative contexts are political by nature, with some contexts exhibiting greater political complexity than others. From the start, you should enhance your projects’ credibility by negotiating and communicating with the different actors and parties. In more polarized contexts, you will face opposition and resistance, or, on the contrary, some actors will want to help you as much as possible. Often you will be placed in positions where you will need to balance the political context and your own position as an evaluator, particularly regarding engagement and protection of your data/confidentiality. These evaluation considerations shouldn’t be minimized. In closing, whatever the context, as evaluators you are accountable for the process and for the results you produce; your choices should be explained, made transparent, and throughout the project you should reflect on the best approach to ensure your results will be credible and valid.