Highlights

Introduction

Monitoring is a “cousin” to evaluation research questions we covered in previous chapters. As related concepts, they complement and sometimes rely on each other, yet they are different. Monitoring is an exercise that compares different aspects of the intervention to some norms. It is used to record the intervention evolution as it tries to achieve some pre-determined objectives. Monitoring is intended to gather information about the performance of an intervention or an organization. Unlike evaluation research—which examines causal relationships between elements related to the intervention—monitoring does not seek to establish causality (McDavid et al., 2019). Instead, it focuses on tracking progress through selected indicators to provide an ongoing sense of how the intervention is unfolding.

The arrows in Figure 4.3 illustrate the possible monitoring focal points: resources, activities, and intervention results. Monitoring information can be used in some evaluative research projects, but its main purpose and how it is conducted are different. Monitoring is used mainly for management purposes, including learning, control, and accountability (Van Dooren et al., 2015).

Monitoring systematically tracks progress against a range of predetermined areas, often against performance indicators and targets. (Markiewicz & Patrick, 2022/2016, p. 122)

Monitoring reports can provide crucial information when conducting evaluations.

Evaluation uses the results of monitoring, complemented by other forms of data gathering, to undertake further and deeper investigations, and through a logical pattern of reasoning, arrives at evaluative conclusions. (Markiewicz & Patrick, 2022/2016, p. 123)

Focus

Monitoring refers to the planned, ongoing, and systematic process of collecting and analyzing program-related information in order to inform management and decision-makers about implementation progress and the program’s performance in relation to its stated objectives and expectations (Markiewicz & Patrick, 2022/2016). Different focuses can be determined. The first step consists of identifying what needs to be monitored. Criteria include the fidelity of the intervention in comparison to the original plan (including costs); the reach; the intervention quality; whether or not the goals were achieved; and impacts on planetary health.

These five criteria apply to either the structure, the process, or the results (see Table 12.1). Quality includes two dimensions: a technical dimension that involves the application of science or technology to manage the problem, and an interpersonal dimensions which includes factors such as client satisfaction or dimensions related to “art of care” (Donabedian, 1980).

Table 12-1 Performance Measurement Criteria

Source: Adapted from Champagne, F., Hartz, Z., Brousselle, A., & Contandriopoulos, A. P. (2011d). L’appréciation normative. In A. Brousselle, F. Champagne, A. P. Contandriopoulos, & Z. Hartz (Eds.), L’évaluation: concepts et méthodes. Deuxième édition mise à jour. Les Presses de l’Université de Montréal: 98.

Establishing a Monitoring System

Once the focus is determined, the next step consists of identifying indicators, targets, and sources of data. Table 12.2 provides an illustration of the different focal points and the corresponding criteria, indicators, targets, data sources, and collection processes.

Choosing targets to monitor is important. According to Champagne et al. (2011d), targets can be identified from different sources such as best practice guidelines, institutional norms, actual practices in similar organizations, and experts’ views. The source will be identified according to the monitoring system’s purpose. The evaluator may choose to select representative targets, such as the mean, or opt for elitist ones (Champagne et al., 2011d). For example, an education program aimed at excellence may choose not to compare its programs to the domain’s average achievement but instead compare itself to the best ones or specific targets of excellence in sub-areas. Targets can be ambitious but should be achievable (Markiewicz & Patrick, 2022/2016).

Monitoring can adopt a longitudinal perspective and track progress over time, using the initial data collected as a baseline for monitoring evolution over time. Monitoring can also be used to compare performance across programs or similar organizations or units (McDavid et al., 2019).

When designing a monitoring system, the evaluator should be specific about the areas that will be monitored and ensure measurable indicators are selected, data is accessible, and the organization can collect and analyze it (Markiewicz & Patrick, 2022/2016).