Table 12-2 Developing the Monitoring Plan
Focus | Criteria | Sample Questions | Indicator | Target | Data Source | Whose responsibility and timing |
Structure | Fidelity | Is the intervention implemented as planned? | Investments Recruitment of professional Qualifications | As set in the plan | ||
Process | Fidelity | Are all activities taking place? Are the costs respected? | Amount and number of activities delivered Operation costs | As set in the plan | ||
Reach | Is the target population participating as expected? | Target population versus participants | As set in the plan | |||
Quality | Are the activities offered according to best practices? | Wait times Survival rates | According to best practices | |||
Are clients satisfied (interpersonal dimensions of quality)? | Satisfaction rates Dissatisfaction | Agreed on levels Nb of complaints | ||||
Results | Goal achievements | Determined according to the logic model, for example, are students succeeding? | Graduate rates Post-graduate employment Studies duration | Best results in similar programs | ||
Environmental impacts: pollution, biodiversity, land and waters | Is the intervention meeting its environmental commitments? | Greenhouse gas emissions Use of plastic Greening of parking lots | Established targets | |||
Impacts on human systems | Is the intervention contributing to better health, greater equity and more prosperity? | Mental health indicators Rates of burn out Diversity of beneficiaries Proportion of spending in local business | According to best practices or agreed pre-set targets |
Source: Adapted from Markiewicz, A., & Patrick, I. (2022). Developing Monitoring and Evaluation Frameworks. SAGE Publications, and from Champagne, F., Hartz, Z., Brousselle, A., & Contandriopoulos, A. P. (2011d). L’appréciation normative. In A. Brousselle, F. Champagne, A. P. Contandriopoulos, & Z. Hartz (Eds.), L’évaluation: concepts et méthodes. Deuxième édition mise à jour. Les Presses de l’Université de Montréal.
Reporting and Other Considerations
Once the data collected, how will it be used? Planning for the use of the results should be considered from the start. Evaluators can consider questions such as: How will the data be compiled? How regularly will it be shared? And with whom— managers, organizational professionals, intervention beneficiaries, or the general public? The compiled information can be used for learning, steering, and formative purposes with the intention of identifying areas and strategies of improvement carried out by the organization (Van Dooren et al., 2015). It can also be used to make decisions about whether to pursue an initiative (i.e. control and accountability functions).
Implementing a monitoring system requires careful planning. Success depends on a series of steps, including engaging with the organization, consulting to build support and anticipate resistance, and designing the system to be both useful and minimally burdensome. It is also essential to ensure that data collection, compilation, and reporting are straightforward. McDavid et al. outline the key steps for designing and implementing performance measurement systems (see Table 12.3).
Table 12-3 Key Steps to Designing and Implementing a Performance Measurement system
1. | Leadership: Identify the organizational champions of this change. |
2. | Understand what a performance measurement system can and cannot do and why it is needed. |
3. | Communication: Establish multichannel ways of communicating that facilitate top-down, bottom-up and horizontal sharing of information, problem identification, and problem-solving. |
4. | Clarify the expectations for the uses of the performance information that will be created. |
5. | Identify the resources and plan for the design, implementation, and maintenance of the performance measurement system. |
6. | Take the time to understand the organizational history around similar initiatives. |
7. | Develop logic models for the programs or lines of business for which performance measures are being developed. |
8. | Identify constructs that are intended to represent performance for aggregations of programs or the whole organization. |
9. | Involve prospective users in reviewing the logic model and constructs in the proposed performance measurement system. |
10. | Translate constructs into observable measures. |
11. | Highlight the comparisons that can be part of the performance measurement system. |
12. | Reporting results and then regularly review feedback from users and, if needed, make changes to the performance measurement system. |
Source: McDavid, J., Huse, I., & Hawthorne, L. R. (2019). Program Evaluation and Performance Measurement. An Introduction to Practice. Third Edition. Sage Publications: 375.
Gaming and Other Behavioural Effects
By establishing a monitoring system, a new set of incentives are implemented. One should be very cautious, not only on the choice of focus, questions, indicators, and norms, but most importantly, attention should be paid to the distortion a monitoring system can create. When people know that their actions are being monitored, they may change their behaviours; for example, in focusing on reaching the set targets they may decide to reduce their efforts in other areas of their work (Van Dooren et al., 2015).
Once incentives are put in place, one cannot change the system easily. In fact, studies have shown that there is no back-to-normal; getting rid of a system results in a new reinterpretation of the rules. People will adapt their behaviours to the system with positive impacts and potentially negative impacts too. Furthermore, in the longer term, the performance system in place may lose its relevance due to the “performance target paradox”:
Behavioural effects, both functional and dysfunctional, lead to a clustering of performance around targets. Over time, indicators lose their ability to differentiate between high and low performers as organisations adjust their performance to align with the indicators. Meyer and Gupta (1994) refer to this phenomenon as the performance paradox (also highlighted by Van Thiel and Leeuw (2002)). We prefer the term performance target paradox because the effect described by Meyer and Gupta should be linked to how performance is measured rather than merely to the pursuit of performance in itself. (as cited in Van Dooren et al., 2025)
Finally, once a monitoring system is put in place and effectively used as a control or an accountability mechanism, some gamification can occur. It happened in the Province of Quebec when the Ministry of Health decided to monitor the patient’s length of stay and the number of stretchers in emergency rooms. Some hospitals felt unable to improve their numbers and decided to create side rooms where emergency patients were placed. As these locations were not considered part of the emergency room, this patient data was not compiled in the Ministry monitoring system. Scores improved, but not patients’ situations.
Conclusion
Implementing monitoring systems is often justified by the desire to become highly productive, effective, or efficient. However, such systems can be onerous to implement and inevitably distort employees’ behaviours, which leads to positive and negative consequences. Monitoring systems’ learning function is the most interesting because of its potential to really improve the intervention and steer organizations in innovative strategic directions. However, to achieve this objective, all facets of the system, including its reporting and potential use, should be carefully designed to ensure monitoring systems’ learning function is activated, rather than control and accountability functions.