Chapter 4: Clinical Adoption Meta-Model | Handbook of eHealth Evaluation: An Evidence-based Approach

Chapter 4

Clinical Adoption Meta-Model

Morgan Price

4.1 Introduction

The Clinical Adoption Meta-Model (CAMM) was developed to support those implementing, studying and evaluating health information systems (HIS) when they are planning evaluations of HIS deployments, and how they are used and incorporated into practice over time (Price & Lau, 2014). This model can inform expectations of stakeholders and evaluation plans so that the correct types of evaluation metrics are considered at appropriate times after an HIS implementation. The CAMM was designed to be accessible to evaluators and stakeholders.

This chapter will begin with conceptual foundations; it briefly describes several common adoption models (some of which are found elsewhere in this handbook). It will outline the four dimensions of the CAMM and then illustrate several CAMM archetypes or representative adoption trajectories. The archetypes are followed by a real-world illustration of how the CAMM can guide a benefits evaluation plan.

4.2 Conceptual Foundations

There are several general adoption models that have been developed to inform adoption such as the Technology Acceptance Model or TAM (Lee, Kozar, & Larsen, 2003) and TAM 2 (Holden & Karsh, 2010), the Unified Theory of Acceptance and Use of Technology or UTAUT (Venkatesh, Morris, Davis, & Davis, 2003), the IS success model (Delone & McLean, 2003), and the diffusion of innovation (Rogers & Shoemaker, 1971) to name a few. Many of these have been applied to describe or explain adoption of HIS and other health technologies, such as the TAM (Holden & Karsh, 2010) and diffusion of innovation (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004).

Several adoption models have been developed for specific types of HIS. Healthcare Information and Management Systems Society (HIMSS) Analytics has three Electronic Medical Record (EMR) adoption models for U.S. hospitals, Canadian hospitals, and U.S. ambulatory EMRs (Palacio, Harrison, & Garets, 2010; Pettit, 2013). The picture archiving and communication system (PACS) maturity model (van de Wetering & Batenburg, 2009; van de Wetering, Batenburg, & Lederman, 2010) describes functionality and integration of PACS systems into hospital workflows. The EMR adoption model (Price, Lau, & Lai, 2011) assesses the use of office-based EMRs over ten functional categories to describe current adoption of the EMR in practice, similar to HIMSS.

In HIS adoption evaluation, we are interested in understanding how health information systems are adopted into healthcare in meaningful ways that improve patient outcomes, quality and sustainability of the healthcare system (Wu, Chaudhry, Wang, & Maglione, 2006). Without understanding the adoption process, we may make inaccurate assumptions about the HIS and attribute the HIS to benefits or lack of benefits seen in evaluation.

4.3 The Four Dimensions of the CAMM

The CAMM was developed to help consider and describe adoption post-deployment of an HIS across four dimensions over time. Figure 4.1 illustrates the CAMM with its four dimensions: availability, use, behaviour, and outcomes. The four dimensions are dependent on each other (e.g., use requires availability) and should be considered collectively when planning an evaluation. The CAMM intentionally focuses on the four dimensions to help shape a focused understanding of adoption over time.

The CAMM was designed to apply to a range of health information systems. Thus, the specific measures and metrics in each of the dimensions would depend on the specific HIS or HIS component being deployed and evaluated. Also, the timelines will vary with the specific HIS being evaluated, how it is being deployed, and the context into which it is being deployed. Smaller components and apps may be quickly adopted and show early outcome changes in shorter periods of time than larger, more comprehensive systems or wider deployments that may take years to adopt.

Changes in the four dimensions are dependent on many factors beyond just time, such as: the HIS itself, its deployment plan, training of users, user expectations, IT support, related information systems, culture, funding, and organizational and jurisdictional regulations. When considering metrics and seeking to understand successes or failures within and across dimensions, it is important to consider, broadly, the factors that can influence adoption. The same HIS may (and likely will) have markedly different adoption trajectories depending on where and how it is deployed.

4.3.1 CAMM Dimension: Availability

Availability is the first dimension. Availability is defined as the end user’s ability to interact with the HIS and its content, when and where needed. Availability can have multiple aspects and here we consider three: user access, system availability, and content availability. User access is the ability for end users to access the system. This can be measured with, for example, the number of user accounts, the numbers of users trained, or the number of accounts with remote access. System availability describes how available the HIS is to its intended end users. This can be measured with, for example, metrics of HIS uptime, the number of terminals deployed, or platforms supported. Content availability considers the information that is accessible in or through the HIS. Content could include patient health data (e.g., lab results) or knowledge base information (e.g., drug monograms, rules for decision support). Content availability can be considered in terms of breadth (types of content), depth (amount of each type), and currency (how quickly the content is updated and available). As availability increases, one would expect the potential to use would increase. An HIS that has only a few trained users or that is only turned on for a few hours a day or that lacks content may not be used extensively.

4.3.2 CAMM Dimension: Use

Use is the second dimension of the CAMM and describes the actual interactions of the intended end users with the HIS. Use is dependent on availability and has two aspects: use of the system and user experience. Use can be measured through a number of metrics, such as: number of log-ins, duration of time the system is used, locations from which the HIS is used, areas of the HIS that are used. User experience describes the subjective experience of end users when using the system. User experience should consider the user’s internal state and the context of the interaction (Hassenzahl & Tractinsky, 2006). Intention to use is excluded from the CAMM as this model specifically describes actual adoption and, thus, use and the user experience of that use are considered, not intention to use; intention to use is included in some other models (see chapter 2, for example). Intention could be considered in pre-deployment evaluations or could be considered when understanding why a system was not used.

4.3.3 CAMM Dimension: Clinical (Health) Behaviour

Behaviour is the third dimension of the CAMM. It describes meaningful adaptation of clinical or health workflows to leverage the HIS features. Behaviour can be considered in terms of two aspects: general capacity and specific behaviours. General capacity is a global change in the healthcare organization. General capacity measures could include the increase or decrease in the number of patients seen per day, or the average length of stay and average cost of stay. Specific behaviours can be assessed that are linked to HIS features (e.g., decision support and a change in the completion of screening tests, or more A1c tests ordered for diabetics), as well as specific workflows impacted by HIS implementation. Intended changes and unintended consequences should be considered when developing an evaluation plan that measures clinical or health behaviour changes, as there may be surprise impacts when workflows are changed.

4.3.4 CAMM Dimension: Clinical (Health) Outcomes

Clinical (Health) Outcomes, the fourth dimension of the CAMM, is defined as impacts that are attributable to the adoption of the HIS. Five aspects of outcomes can be considered when developing measures and metrics for HIS adoption: patient outcomes, provider outcomes, organizational outcomes, population outcomes and cost outcomes. Outcomes could be considered early or late, depending on evaluation timing. Patient outcomes include aspects directly related to individual patient health, such as patient changes in complications due to diabetes. Provider outcomes include provider-centric measures, such as better physician retention. Organizational outcomes include factors measured at an organizational level (e.g., nosocomial infection rates) whereas population outcomes are measured across organizations (e.g., obesity rates, lifespan, myocardial infarction rates). Finally, cost outcomes can be considered that describe relative or absolute costs to the healthcare system. The specific outcomes will depend on the HIS, how it is deployed, and the goals of the project. Not all aspects need to be measured.

There can be some confusion or overlap between behavioural changes and early outcomes and there are grey areas between the two. Consider the behaviours as those that are directly related actions under the control of the HIS user. If an electronic medical record recommends that a physician check blood pressures and the rate of blood pressure checking in the office goes up, that is a behaviour change. An early outcome may be a decrease in the values of the blood pressure readings as people then are better managing their blood pressure.

4.4 CAMM Archetypes

CAMM archetypes were developed to help with understanding and applying the CAMM. Archetypes are representational adoption trajectories for health information systems. These would not chart the precise path that an adoption must or would take. Indeed, most real-world adoptions will fall somewhere between two or more of these archetypes. Still, these are helpful illustrations for discussing the ranges of successes, challenges, and failures that can be seen with HIS adoption. The CAMM archetypes are:

No Deployment.
Low Adoption.
Adoption without Benefit (behaviour and outcome).
Behaviour Change without Outcome Benefit.
Adoption with Outcome Benefits.
Benefit without Use.
Adoption with Harm.

4.4.1 No Deployment

This archetype describes an HIS initiative that does not reach the end users in a clinical or health setting. No deployment of an HIS can occur for several reasons, including: an incomplete product, lack of funding, strategic change within an organization, significant delays in the product, or unsuccessful testing of a component. Whatever the reason(s), the deployment to end users was stopped prior to a planned go-live event. End users may be involved in the design or testing but there is not a deployment into a real-world setting. This is often the clearest, most obvious archetype.

4.4.2 Low Adoption

In this archetype, the HIS is deployed and available, but availability is followed by minimal or rapidly declining use (Figure 4.2). Users may explore the HIS, but use is not sustained. Without use, it is not reasonable to expect a benefit from the tool. This can been seen with systems that do not support and fit the clinical environment and where use of the HIS is voluntary. This archetype (along with Benefit without Use) highlights the importance of measuring the multiple CAMM dimensions. If only outcomes are measured, one may make an assumption that an intervention is not beneficial even when it is not used.

An example of the Low Adoption archetype would be assessing the impact of decision support alerts in a system that allows users to turn on or off the alerts. An evaluation may show the implementation of specific decision support alerts is not impacting outcomes. If all dimensions were evaluated, it may be found that use was low because most of the users simply turned the alerts off. Without sufficient use, one cannot expect the outcomes to change.

4.4.3 Adoption without Benefit (behaviour and outcome)

Here we see an HIS that is both available and used by end users; however, it is not achieving the intended behaviour changes or the expected outcomes (Figure 4.3). This archetype can be seen when the HIS functions and features do not directly align with the metrics being measured or the HIS features are not sufficiently evidence-based to facilitate the desired behaviour changes and outcomes. It may be seen when the measured clinical behaviours and outcomes are already positive, that there is a ceiling effect, or, conversely, when the clinical environment has limited capacity for change. It can also happen when the timing of the evaluation is premature and adaptations or changes to health outcomes have not yet occurred.

4.4.4 Behaviour Change without Outcome Benefit

This archetype occurs when an adopted HIS produces the expected changes in behaviour, however the behaviours are not leading to the expected outcomes (Figure 4.4). This can be seen when the intervention isn’t sufficiently evidence-based, or the causal chains in the evidence are not sufficient to lead to the outcomes. Again, it may be possible that the outcomes are already good (the ceiling effect) or that the duration or timing of evaluation is too short to see the outcomes. It is important to note that some clinical outcomes are not immediately evident. Successful preventive care programs may not be expected to show benefits in mortality for many years, as the natural history of several diseases are described in years or decades. Thus, early surrogate markers that are connected to evidence are often chosen to support stakeholders in their decision-making.

4.4.5 Adoption with Benefits

This is the archetype that HIS adoption programs expect and hope to see: a clear progression of HIS availability that leads to ongoing use of the HIS (Figure 4.1). HIS use then leads to observable changes in clinical and health behaviours that, in turn, result in improvements in measured outcomes. Note that while the CAMM suggests a causal link between each of the four dimensions, the reality is that healthcare is a complex environment, often with multiple programs seeking the same types of improvements. It is important to note that causation cannot be assumed between the HIS and the outcomes just because they are measured. Some evaluation methods can only describe the correlation of events and the CAMM does not specify evaluation methods. Methods should be sufficiently rigorous to support both the scope of the initiative and ongoing decision-making, in addition to adjustments to the HIS as required over time.

4.4.6 Benefit without Use

Here we see the expected behaviour changes and/or outcomes but without the use of the HIS. This occurs (as described above) where there can be multiple overlapping initiatives, each striving to improve the same or similar outcomes (Figure 4.5). Here, another program confounds and impacts the measurements of the HIS behaviour or outcome metrics.

As an example, consider a scenario where a new eHealth tool may be developed to support chronic disease management. Many target users do not use the eHealth tool as they feel it is too cumbersome and their current practices are more efficient. However, at the same time a new funding program for chronic disease management is initiated. This motivates users and many of the chronic disease management activities envisioned to be enabled by the new component are taking place, but through other means. Chronic disease management improves. Clearly, there is a correlation between deployment of the eHealth tool but there is not a correlation with use. This archetype highlights the importance of measuring each of the dimensions as part of an evaluation. Without measuring use, the evaluator and stakeholders could erroneously assume that the HIS is enabling and responsible for the improvement in chronic disease management.

4.4.7 Adoption with Harm

Although we like to focus on benefits, HIS adoption may lead to unintended consequences and harm (Figure 4.6). This archetype highlights the risk of negative effects caused by the use of an HIS. HIS deployments can result in harm from unexpected changes brought about by the implementation of the HIS. Harm can occur from improper design, improper use, or from changes in other workflows (often informal workflows) resulting from the HIS implementation. Potential harm should be considered when planning and should be measured in the evaluation to avoid or limit unintended effects.

4.5 Using the CAMM

The four CAMM dimensions and their aspects help describe trajectories of HIS adoption over time. The CAMM suggests a logical causal chain from availability to use to behaviour changes to resulting changes in outcomes. The CAMM can be helpful in planning evaluations and in explaining findings.

For those who are planning HIS evaluations, the CAMM provides a framework to consider metrics and measures that will change at differing points over time. The CAMM also highlights the need to consider multiple dimensions within an HIS evaluation and when each evaluation dimension will be expected to be most helpful in an adoption’s life cycle. Stakeholders will have evaluation needs that have their own timing. The CAMM can help inform and focus the kinds of evaluations that would best support stakeholder needs. It would not be helpful to measure changes in outcomes three months after a diabetes prevention app is published for mobile phones, for example. Those outcomes would not be expected to be measurable for years. Instead, the CAMM would suggest considering metrics for availability (presence on the app stores, presence on smart phones as indicated by number of downloads) and use (number of times the app is opened by how many users, content reviewed). These will show stakeholders meaningful early metrics, which can evolve to the later metrics over time.

The images of the CAMM suggest individual trajectories for each dimension, but the reality is there can often be multiple metrics for each dimension that follow different trajectories. For example, positive and negative outcomes can occur at the same time, depending on the specific metrics an evaluation considers. A targeted intervention may have unintended consequences due to a shifting of resources away from good practice. A particular HIS may have strong areas and weaker areas and thus only measuring its impact in one functional department may fail to present a full picture. Further, adoption of an HIS may be variable across an organization. The evaluation of an HIS is complex and the CAMM provides an accessible framework to begin planning an evaluation over time.

As an explanatory framework, CAMM can be also applied retrospectively. It can be used to consider the results of an adoption. CAMM can be used with stakeholders to reflect and point to areas of an implementation that should be better explored to understand some results. Availability issues and partial or unexpected use could be discovered in projects where benefits are not being realized. Quantitative and qualitative metrics can be sought retrospectively if needed to help understand an HIS implementation.

4.5.1 Case Study: Using the CAMM to Inform a Personal Health Portal Evaluation

The CAMM was initially developed to help engage stakeholders in the discussion and planning of benefits evaluations for the deployment of a multiphased personal health portal program. This case study will focus on developing an evaluation plan for the initial deployment of the personal health record (PHR) component of a larger Personal Health Portal project. As part of the multi-stakeholder engagement, the goals of the initial PHR deployment were prospectively elicited. The focus of the initial deployment was within a single clinical site and an evaluation plan was developed for this initial deployment, with an eye to stakeholder needs, that included planning for future, broader deployments.

The CAMM dimensions are presented here in “reverse” order as it can be helpful to “start with the end in mind” when developing the evaluation metrics.

4.5.2 Setting: Current State

The site of the initial PHR deployment was a cardiac rehabilitation program for patients who had recently suffered a heart attack. It was a 12-week outpatient program started after patients were stable and discharged from hospital. Patients currently engage with a team of cardiologists, cardiac nurses, dieticians, and exercise therapists to educate and create a personalized program of rehabilitation (which included diet, exercise, medications, monitoring, and self-management) to improve and maintain function. The PHR was being deployed to patients at the start of their program with functions tailored to management of cardiac care and related conditions (e.g., hypertension, diabetes) and a mechanism allowing trusted providers to access the record by virtual check-in.

4.5.3 Predicted Outcomes

From the stakeholder engagement, the key outcome for this deployment was to reduce recurrent heart attacks in patients who had already suffered a heart attack (and thus improve mortality). It was expected that would be achieved through better proximal outcomes like improved blood pressure control, better management of congestive heart failure (CHF), and overall improved patient knowledge of cardiac care and their own care plans.

4.5.4 Expected Behaviours

The stakeholders linked use of the PHR with several health behaviours that would lead to the predicted outcomes. First, the patients would be more engaged, which would result in better blood pressure tracking, improved diet, more exercise, and better adherence to medications. The PHR allowed linking to providers, so stakeholders expected closer and longer follow-up of patients in the cardiac rehabilitation program. This would be seen both in an increase in the number of contacts with the patient and an extension of the rehab programs to more than 12 weeks.

4.5.5 Expected Use

The stakeholders expected patients to use the system regularly to track weight, blood pressure and medication use. They also expected patients to use cardiac rehab self-management plans (e.g., care plans) that were in the PHR. They expected Registered Nurses to log in at least weekly to check on patients in the program. The expectation was that the user experience was easy and intuitive for the patients, thus facilitating self-management.

4.5.6 Expected Availability

The stakeholders assumed availability would be 100% for all participants. Further discussion elicited several specifics: All users (patients and providers) would have access, which included passwords and training; the system would be available through the Internet at points where users expect it to be available (clinic, home); and the PHR had tools available that would support the self-management of patients’ cardiac care issues.

4.5.7 Evaluation Metrics and Results

Timing of the deployment and evaluation was relatively short (12 weeks). This was necessary as a key decision was to be made by the steering committee on future deployments within four months of this pilot. As a consequence, outcomes could not be selected, as outcome evaluation would have likely resulted in a null result. Thus, the evaluation focused on early dimensions: Availability and Use. Data was collected through interviews and focus groups at multiple points in time over the 12-week pilot. Description of adoption would be described for patients and then providers.

Patients: Availability: All patients had accounts and training. The PHR was running without issue for the 12 weeks; however, some patients expected the PHR could be accessed through smartphones or tablets and it was (at that time) designed for desktop browsers. Not all patients had computers as they had transitioned to tablets. Content included provincial medication dispensing records and whatever information the patient entered. Use: Most patients used the PHR regularly as part of the study. The user experience could have been improved through streamlining the navigation and providing more valuable tools in the PHR that would help patients meet their care plan goals (e.g., reminders). Patients did not note any behaviour changes, as the PHR for them was primarily a documentation tool.

Providers: Availability: There were delays in availability. Specifically, accounts were created for providers but the process for connecting providers to patients was challenging due to timeouts. An asynchronous process, the account linking required multiple steps and with patients not logging into the PHR daily and providers perhaps only working part-time, the window to link provider accounts to patient accounts in the PHR proved difficult. Use: Provider use was limited by availability. Virtual connections and monitoring had not begun during the pilot.

The use of the CAMM in this case study intentionally highlights the importance of measuring early dimensions of availability and use in implementations. These can facilitate important improvements in the deployment plans to better achieve adoption and expected benefits. In this case study, the findings were used to inform the next planned deployment and the CAMM was used to frame subsequent deployments in this large, phased program.

4.6 Summary

The CAMM is an adoption model that highlights how evaluation of HIS deployments should change over time. The adoption of health information systems can follow a trajectory of linked activities that are described by the four dimensions: availability, use, behaviour, and outcomes. Each of these dimensions can be used to consider when specific metrics should be measured over time during an ongoing evaluation of an HIS deployment.

The CAMM highlights that evaluations early in the adoption process, such as the case study, should focus on early dimensions of availability and use. Later evaluations should not only focus on the later behaviour and outcomes dimensions, but also should include some assessment of availability and use to ensure that the outcomes are not being seen without the expected adoption of the tools.

4.6.1 Acknowledgements

We would like to thank the Alberta Ministry of Health for supporting the evaluation program for the Patient Health Portal that led to the development of the CAMM.

References

DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success: A ten-year update. Journal of Management Information Systems, 19(4), 9–30.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581–629.

Hassenzahl, M., & Tractinsky, N. (2006). User experience: A research agenda. Behavior and Information Technology, 25, 91–97.

Holden, R. J., & Karsh, B. T. (2010). The technology acceptance model: Its past and its future in health care. Journal of Biomedical Infomatics, 43, 159–⁠172.

Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003). The technology acceptance model: Past, present, and future. Communications of the Association for Information Systems, 12, 752–780.

Palacio, C., Harrison, J. P., & Garets, D. (2010). Benchmarking electronic medical records initiatives in the U.S.: A conceptual model. Journal of Medical Systems, 34(3), 273–279.

Pettit, L. (2013). Understanding EMRAM and how it can be used by policy-makers, hospital CIOs and their IT teams. World Hospitals and Health Services, 49, 7–9.

Price, M., & Lau, F. (2014). The clinical adoption meta-model: a temporal meta-model describing the clinical adoption of health information systems. BMC Medical Informatics and Decision Making, 14, 43. Retrieved from http://www.biomedical.com/1472-6947/14/43

Price, M., Lau, F., & Lai, J. (2011). Measuring EMR adoption: a framework and case study. Electronic Healthcare, 10(1), e25–e30.

Rogers, E. M., & Shoemaker, F. F. (1971). Communication of innovations: A cross-cultural approach. New York: The Free Press.

van de Wetering, R., & Batenburg, R. (2009). A PACS maturity model: A systematic meta-analytic review on maturation and evolvability of PACS in the hospital enterprise. International Journal of Medical Informatics, 78(2), 127–140.

van de Wetering, R., Batenburg, R., & Lederman, R. (2010). Evolutionistic or revolutionary paths? A PACS maturity model for strategic situational planning. International Journal of Computer Assisted Radiology and Surgery, 5(4), 401–409.

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. Management Information Systems Quarterly, 27(3), 425–478.

Wu, S., Chaudhry, B., Wang, J., & Maglione, M. (2006). Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 144(10), E12–E22.

Figure 4.1. The clinical adoption meta-model.

Note.From “The clinical adoption meta-model: a temporal meta-model describing the clinical adoption of health information systems,” by M. Price and F. Lau, 2014, BMC Medical Informatics and Decision Making, 14(43), p. 2. Copyright 2014 by Price and Lau. CC BY Licence.

Figure 4.2. Low adoption archetype.

Figure 4.3. Adoption without benefits archetype.

Note. From “The clinical adoption meta-model: a temporal meta-model describing the clinical adoption of health information systems,” by M. Price and F. Lau, 2014, BMC Medical Informatics and Decision Making, 14(43), p. 5. Copyright 2014 by Price and Lau. CC BY Licence.

Figure 4.4. Behaviour change without outcome benefits archetype.

Note. From “The clinical adoption meta-model: a temporal meta-model describing the clinical adoption of health information systems,” by M. Price and F. Lau, 2014, BMC Medical Informatics and Decision Making, 14(43), p. 6. Copyright 2014 by Price and Lau. CC BY Licence.

Figure 4.5. Benefit without use archetype.

Note. From “The clinical adoption meta-model: a temporal meta-model describing the clinical adoption of health information systems,” by M. Price and F. Lau, 2014, BMC Medical Informatics and Decision Making, 14(43), p. 6. Copyright 2014 by Price and Lau. CC BY Licence.

Figure 4.6. Adoption with harm archetype.

Chapter 5: eHealth Economic Evaluation Framework

EPUB

Show the following:

Adjust appearance:

Notes

Annotate