“Chapter 20: Evaluation of eHealth Adoption in Healthcare Organizations” in “Handbook of eHealth Evaluation: An Evidence-based Approach”

Chapter 20

Evaluation of eHealth Adoption in Healthcare Organizations

Jim Warren, Yulong Gu

---

20.1 Introduction

Healthcare innovations, including eHealth technologies, aim to support faster, more reliable and more transparent healthcare services. These technologies may facilitate the design and delivery of high-quality healthcare, improved patient outcomes and patient safety, and further generation of innovation in healthcare processes (Chaudhry et al., 2006; Finkelstein et al., 2012; Lau, Kuziemsky, Price, & Gardner, 2010). However, adoption of eHealth technologies in healthcare organizations involves complex sociotechnical issues and often fails (Kaplan & Harris-Salamone, 2009). Adopters of eHealth technologies face challenges such as the complexity inherent to the healthcare services context and a multitude of risk factors in the eHealth development/procurement and implementation processes. These challenges need to be understood and promptly addressed to support successful implementation and sustained use of health innovations. As such, one fundamental starting goal for any process of eHealth evaluation is to evaluate adoption, particularly in terms of the uptake of the technology by the intended end users.

In chapter 6 we categorized uptake (or simply “use”) under the “product” dimension of our Health Information Technology (IT) Evaluation Framework as a component of usability. While this is valid, uptake is an essential step on the pathway to any and all progress on the “impact” dimension of the framework (i.e., for improvement in work and communication patterns, organizational culture, safety and quality of healthcare, or overall effectiveness). This is well illustrated by a study of clinical decision support effectiveness for chronic condition management that was undertaken in the context of general practice in the United Kingdom (U.K.). The system showed no impact whatsoever on process of care or patient outcomes, while noting that usage was low (Eccles et al., 2002). Further in-depth investigation with users identified barriers to use that included concerns about the timing of the guideline trigger, ease of use and helpfulness of content, as well as problems in the delivery of training (Rousseau, McColl, Newton, Grimshaw, & Eccles, 2003). The lack of overall impact becomes unsurprising once the story with respect to barriers to adoption is understood.

Health IT adoption has received considerable attention in recent years. For instance, the Healthcare Information and Management Systems Society (HIMSS) Analytics Electronic Medical Record Adoption Model (EMRAM) ranks healthcare organization progress into one of seven stages based on the types of systems that are in place (HIMSS, 2015). In the government context, health IT adoption in the United States is being driven by financial incentives that are tied to achievement of a spectrum of specific “meaningful use” criteria (Marcotte et al., 2012). Similarly, the U.K. Quality and Outcomes Framework (QOF) provides substantial financial incentives to general practitioners (GPs; i.e., community-based family physicians) based on monitoring and management levels as automatically assessed through their practice electronic medical record systems (Lester & Campbell, 2010). It could be said that the aforementioned models are tied to ticking the boxes to achieve financial incentives (or “bragging rights”). At a more conceptual level, sophistication in health IT use can be broken down into technological support, information content, functional support, and IT management practices (Raymond & Paré, 1992), as well as extent of systems integration (Paré & Sicotte, 2001). A further dimension of IT sophistication, in terms of application domain, concerns administrative activities, patient care and clinical support; and in any of the above domains one can assess the range of computerized activities and system availability, as well as extent of use (Kitsiou, Manthou, Vlachopoulou, & Markos, 2010; Paré & Sicotte, 2001).

In this chapter, we illustrate the approach to evaluation of system adoption with two case studies that are based on electronic referral (eReferral) technologies. The United States National Library of Medicine has defined “referral and consultation” as “the practice of sending a patient to another program or practitioner for services or advice which the referring source is not prepared to provide” (National Library of Medicine, 2014), which implies a transfer of care. In the New Zealand (N.Z.) healthcare context, referral is most often from a GP to a specialist medical service. Moreover, the general practices in N.Z. tend to be private for-profit or charitable trust organizations (although supported by government subsidies). Individual general practice sites are small (for instance, they may be part of a strip mall) and are characterized as being situated in “the community” alongside other services including a community pharmacy and home-care nursing. Conversely, specialist services are provided in large part at public hospitals operated directly by District Health Boards; eReferral aims to use IT to bridge the communications gap between these two types of providers and their contrasting sites. While eReferral may simply replace a postal or fax process with e-mail, more advanced IT offers opportunities for rich and rapid feedback that transforms the process. At the more extreme end, eReferral can merge into a portal-based “shared care” model that challenges the original concept of referral (Gu, Warren, & Orr, 2014).

Although the two case studies in this chapter are both situated in the New Zealand context of bridging community-hospital divides, we believe they can be generalized to any situation where IT is mediating healthcare communication across provider roles and sites. Moreover, these cases serve to illustrate contexts where providers can potentially work around uptake of the technology (e.g., side-stepping with phone and fax) and thus adoption is a valuable measure of success.

20.2 Selected Case Study Examples

In eHealth evaluation literature, both qualitative and quantitative methods have been used to measure a range of indicators on usability and outcome. The value of these evaluations is not limited to collecting robust evidence on the impact of eHealth innovations, which of course is important for measuring project success or supporting the decision-making process with regard to technology purchase and further rollout (or abandonment) of the technology adoption. The evaluation research can also provide substantial support to the technology development and implementation process. That is, if you evaluate early and often, learnings from evaluation can be used to improve the acceptability and effectiveness of the technology in its current implementation sites, as well as being fed into subsequent phases of implementation.

The following two examples of eHealth evaluation are introduced to demonstrate impact analysis-focused evaluation and Action Research-oriented evaluation, respectively.

20.2.1 Case Study One — Impact of an Electronic Referral System

This case describes a retrospective evaluation study of the impact of introducing an eReferral system that manages referrals from a community into public secondary healthcare services (Warren, White, Day, Gu, & Pollock, 2011). The eReferral system evaluated was introduced in 2007 to 30 referring general medical practices and 28 hospital-based secondary services at an N.Z. regional healthcare jurisdiction, Hutt Valley District Health Board (HVDHB). HVDHB serves a population of 150,000 and has one principal facility for provision of secondary services, the 260-bed Hutt Hospital.

By October 2007, eReferral to 28 services at Hutt Hospital — to all services but the Emergency Department — had been deployed across 25 general practices. A GP, or in some cases a practice nurse, creates a referral from within their electronic medical record system (in New Zealand, it is often called practice management system, or PMS) using PMS-based templates. The form is pre-populated with PMS data including the patient’s demographics and medical history, which can be edited by the GP prior to submission. The GP referral is messaged as Extensible Markup Language (XML) via the regional service where it is mapped to Health Level 7 (HL7) message format and sent on to the Integration Engine, which underlies the HVDHB’s Clinical Workstation. The Integration Engine generates an acknowledgement back to the network confirming receipt of the referral, which is relayed back to the GP PMS.

Hospital staff can view the eReferral and process the eReferrals in a Clinical Workstation displaying in automated role-based work lists. It allows clinicians to communicate within the service administration to manage clinics, required tests, or follow-up with patients prior to the appointment. The referral management activity creates an automated sequence of process events through clinician triage (assignment of priority) and, if not declined at triage, regarding creation of a booking for a first specialist appointment (FSA). The relevant referral workflow is shown in Figure 20.1.

Figure 20.1. Referral workflow.

As of November 2010, there was no central referral management at Hutt Hospital; however, the general Outpatient Department with eight administration staff manages 15 services; the remainder receive and manage their own referrals. In the context of an N.Z. public hospital, services undertake clinical triage of referrals that assigns priority levels to them, including declining to service some requests (noting that private services are also available). Referral management within the hospital involves two concurrent systems: a Clinical Workstation and a Patient Information Management System (PIMS, providing general inpatient tracking, by a different vendor from the Clinical Workstation). All referrals (electronic or paper) are logged to the PIMS.

Based on a literature review, Hutt eReferral project business case and documentation review, and stakeholder feedback, the eReferral evaluation hypothesis was developed as: eReferral, if uptake is substantial and sustained, should result in more efficient (and thus timely), as well as more transparent, processing of referrals. To test this hypothesis, 33,958 transactional records from October 2007 to the end of October 2010 were collected from the eReferral database, as stored with the Clinical Workstation: 108,652 records of all GP referrals (electronic and paper) from the hospital PIMS were extracted from January 2004 to end October 2010. These data allowed examination of eReferral’s impact, in terms of uptake (eReferral volume over time and proportion of referrals that are electronic) and changes in latency from letter date to triage at secondary services. The extracts, de-identified and using encrypted health identifiers (matchable across data sets, but not reidentifiable by the evaluators), were made available to the evaluators by HVDHB. Qualitative feedback from interviews and focus groups further provided insight on benefits and/or liabilities of the solution, including influence on workflow and usability.

The eReferral use rose steadily to 1,000 transactions per month in 2008, thereafter showing moderate growth to 1,200 per month in 2010. Rate of eReferral from the community in 2010 is estimated at 56% of total referrals to the hospital from general practice, and as 71% of referrals from those having done at least one referral electronically. Figure 20.2 graphs the PIMS volumes for referral records indicating source as General Practice along with the transaction volume for all eReferrals (from the Clinical Workstation database) by year. A boost in total general practice referrals after relative stability in earlier years, tracking with increased eReferrals particularly between 2008 and 2009, indicates interaction of eReferral uptake and increase in total referrals.

Figure 20.2. General practice referral volumes by year (* 2010 data inflated by 6/5ths to estimate full year volume).

Note. From “Introduction of electronic referral from community associated with more timely review by secondary services,” by J. Warren, S. White, K. Day, Y. Gu, and M. Pollock, 2012, Applied Clinical Informatics, 2(4), p. 556. Copyright 2011 by Schattauer Publishing House. Reprinted with permission.

Referral latency from letter date to hospital triage improves significantly from 2007 to 2009 (Kolmogorov-Smirnov test, p < 0.001), from a paper referral median of eight days (inter-quartile range, IQR: 4–14) in 2007 to an eReferral median of five days (IQR: 2–9) and paper referral median of six days (IQR: 2–12) in 2009; see also Figure 20.3.

Qualitative feedback confirmed that the significant speed-up in referral processing shown in Figure 20.3 was achieved without changes in staffing levels. The evaluation concluded that substantial, rapid, and voluntary uptake of eReferrals was observed, associated with faster, more reliable, and more transparent referral processing. Clinical users appreciated improvement of referral visibility (status and content access); however, both GPs (referral senders) and specialists (receivers) point out system usability issues such as difficulties surrounding attachments in terms of both attaching at the sender’s end and opening at the receiver’s end.

 

Figure 20.3. Median, first and third quartile (‘Med’, ‘1stQ’ and ‘3rdQ’ respectively) of letter-to-triage latency for e-referrals and paper referrals by year.

Note. From “Introduction of electronic referral from community associated with more timely review by secondary services,” by J. Warren, S. White, K. Day, Y. Gu, and M. Pollock, 2012, Applied Clinical Informatics, 2(4), p. 557. Copyright 2011 by Schattauer Publishing House. Reprinted with permission.

20.2.2 Case Study Two — Promoting Sustained Use of a Shared Care Planning Program

The evaluation of New Zealand’s National Shared Care Planning Program (NSCPP) was a case of Action Research-oriented evaluation that was planned during the eHealth program’s business case stage in 2010. The evaluation was concurrently undertaken with the pilot development and implementation (2011 to 2012) with the aim to assess success as well as to support the pilot processes (Gu, Humphrey, Warren, & Streeter, 2014; Gu, Humphrey, Warren, Tibby, & Bycroft, 2012; Warren, Gu, & Humphrey, 2012). This example applied the principle that eHealth evaluation should begin before the new technology is introduced into the health workflow and be planned for along with the planning of the implementation itself. It demonstrated how evaluators could work in collaboration with the broader eHealth project team to understand and improve the user experience.

NSCPP took an IT-enabled approach to support shared care, shared decision-making, and care planning for long-term condition management. A Web-based technology solution was developed to provide a shared care record and coordination capability, including care plans, messages, and task assignment, for multidisciplinary care teams including patients themselves. The goal was to enable a patient-centred approach to care irrespective of the current care provider in general, specialist or allied healthcare settings, by facilitating both care coordination and supported self-management. The technology was integrated with GPs’ PMS and has browser access to patient records for other community-based providers, hospital providers and patients.

NSCPP evaluation took an iterative action research approach applying both qualitative and quantitative methods. The pilot software was refined in response to ongoing feedback from the evaluation, which emphasizes attention to user feedback through interviews, focus groups and questionnaires to both participating healthcare professionals and patients, thematic analysis of communication records via the pilot system such as tasks and messages, as well as quan­ti­tative anal­ysis of pilot system transaction records and health service usage data. Findings were used to identify any pressing issues and, according to evidence and expert experience, the corresponding recommendations for addressing the problems. This multifaceted data collection framework supported rapid synthesis of information and routine feedback loops to the program team to inform ongoing approaches in the rollout of the program. With an action research orientation, the methods and tools for the evaluation study were constantly examined and developed to accommodate the NSCPP development needs.

The program uptake, in terms of technology usage pattern and user experience, was closely monitored via qualitative feedback as well as analysis of user activities interacting with the pilot technology. And these were examined in the context of the users’ professional roles, for example GP, general practice nurse, specialist physician, secondary nurse, allied health professional (including pharmacist and physiotherapist), and patient. Figure 20.4 captures user activities of creating and modifying tasks, notes, care plan elements and messages in the first nine months of the program, including the “Exploration” Phase, from March to June 2011 with one participating general practice and one secondary service and the “Limited Deployment” Phase (since July), extending to eight general practices, five secondary services and four community pharmacies. The modification activity includes marking a task as completed (which is the only such action available to patients in the patient portal at the time). Figure 20.5 shows user activities in terms of viewing the records, including tasks, notes, plans, messages, diagnosis, measurement results, medication and record summary, by month and roles.

 

Figure 20.4. Sum of entries created or modified (over notes, care plan elements, messages and tasks) by role

The above figures show the emergence of patient and allied users from August to October, with steady growth in allied health professional’s activity and patient viewing. The role of specialist physicians as direct users, particularly with respect to element creation/modification, is quite small. The role of nurses is dominant for viewing and the creation or modification in all time periods except for a few cases in the early “Exploratory Phase” in February. In “Limited Deployment Phase” (since July), the role of general practice in element creation is highly dominant (at least two-thirds of entries), but is more balanced by other users with respect to viewing (roughly 50%). The observed pattern of nurses being the most active users extended to task assignment, an indicator of who is “driving”. However, user interviews indicated that this might underestimate the guidance provided by physicians (indeed, at times, even literally looking over the shoulder of the nurse operating the software). And, of course, the technology does not capture verbal communications that occur between nurses and physicians onsite. It is recognized that there is exciting potential for workforce transformation with NSCPP, but with the related challenge of defining the new responsibilities (and determining if these are met by new people or reorientation of existing roles). The question of who funds the time to create care plans had been raised repeatedly. A designated — and appropriately compensated — lead care coordinator (perhaps a nurse), would facilitate the solution of a further problem regarding the need to ensure timely responsiveness to issues emerging in the care of any given patient.

Figure 20.5. Elements viewed by user role based on number of NSCPP system audit log entries.

In fact, NSCPP has highlighted a range of fundamental challenges, including: (a) sociotechnical issues (e.g., interoperability in the non-standardized system environment, shortage of workforce skills to deliver care planning, lack of time/personnel to implement shared care, IT interface challenges, and mechanisms to involve patients and families); (b) governance of information, clinical workflows, privacy and funding models; and (c) patient safety concerns in relation to information access (and potentially input) by patients (e.g., detailed clinical communications can present difficulty for patient interpretation and are readily misunderstood). Moreover, there was no agreed definition consensus — in theoretic ground and among participating organizations or individuals — for “shared care planning” or the essential elements, roles or responsibilities needed for its delivery. On the other hand, most pilot participants acknowledged the notion that shared care is equally about sharing the care and responsibility with patients and their families as it is about sharing care within the interdisciplinary team. The NSCPP evaluation concluded that while many issues remain unresolved, the NSCPP experience is making the issues far more concrete and is building a wide community of clinical, and patient, users that now have first-hand experience to inform continued technology and policy development.

20.3 Discussion

Adoption is the essential first step in benefits realization for health IT. All evaluation studies must include investigation of adoption or risk misleading results. Substantial and sustained uptake in use of a system indicates success across a range of issues in project management, leadership, deployment, training, usability and overall “value proposition” of the system for the users. Conversely, failure in adoption indicates a breakdown. Continuing to pursue other aspects of evaluation in the face of poor adoption can lead to mistaking the IT system for the cause in a situation where other factors in fact account for observed variations in performance. Moreover, it is important to recognize that adoption is not an all-or-nothing proposition. Users may adopt some features of a system but not others, or uptake may be greater with one class of users than another (or at one site versus another). Such variation in uptake warrants more in-depth investigation and can lead to the discovery of opportunities for improvement wherever needed — in usability, training, system features or broader workflow and work role expectations.

Obtaining quantitative measures of adoption is usually relatively easy in the context of health IT because the systems, by their nature, lay down transactional “footprints” of their activity: a computerized physician order entry (CPOE) system creates records of orders, an electronic referral system creates records of referrals. Moreover, most systems will create usage logs for other purposes (e.g., security audit), although advanced planning to ensure the logging of the right information can greatly facilitate subsequent analysis.

The greatest benefits of adoption evaluation come, however, with the qualitative analysis in follow-up to areas where uptake is quantitatively weak. When interviewed, users can generally state the barriers to adoption that they are facing. Further, particularly if the interviews are structured to support such feedback, users may already have suggestions for improvements, or shed light on fundamental problems that underlie failure to adopt the system. This information can then be fed back to the broader project team to reduce the barriers as effectively as possible. As with all aspects of evaluation, the opportunities are greatest with a deployment process that is iterative and staged so that time and resources are available to learn from initial evaluation activities and to apply those lessons in later deployments.

20.4 Summary

In this chapter we have emphasized the importance of studying adoption in terms of substantial and sustained uptake of the system by its intended users as a cornerstone of eHealth evaluation. Through two case studies we have illustrated quantitative and qualitative approaches, with the quantitative dimensions underpinned by analysis of the quantity, source and timing of system transactions and the qualitative dimension underpinned by interviews. We have shown that uptake can be heterogeneous — with differences in usage levels for different classes of users, for example. We have seen that in follow-up with users about barriers to uptake we can achieve insights into both the current and potential role of the health IT-based innovation.

References

Chaudhry, B., Wang, J., Wu, S., Maglione, M., Mojica, W., Roth, E., . . . Shekelle, P. G. (2006). Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 144(10), 742–752.

Eccles, M., McColl, E., Steen, N., Rousseau, N., Grimshaw, J., Parkin, D., & Purves, I. (2002). Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. British Medical Journal, 325(7370), 941.

Finkelstein, J., Knight, A., Marinopoulos, S., Gibbons, M. C., Berger, Z., Aboumatar, H., . . . Bass, E. B. (2012, June). Enabling patient-centered care through health information technology. Evidence Reports / Technology Assessments No. 206. Rockville, MD: Agency for Healthcare Research & Quality.

Gu, Y., Humphrey, G., Warren, J., & Streeter, J. (2014). Measuring service utilisation impact of the Shared Care Planning Programme. Health Care and Informatics Review Online, 18(1), 16–24.

Gu, Y., Humphrey, G., Warren, J., Tibby, S., & Bycroft, J. (2012). An innovative approach to shared care — New Zealand pilot study of a technology-enabled national Shared Care Planning Programme. Paper presented at the Health Informatics New Zealand Conference, November 7 to 9, Rotorua.

Gu, Y., Warren, J., & Orr, M. (2014). The potentials and challenges of electronic referrals in transforming healthcare. New Zealand Medical Journal, 127(1398), 111–118.

Healthcare Information and Management Systems Society (HIMSS). (2015). Electronic Medical Record Adoption Model (EMRAM). Retrieved from http://www.himssanalytics.com/provider-solutions#block-himss-general-himss-prov-sol-emram

Kaplan, B., & Harris-Salamone, K. D. (2009). Health IT success and failure: recommendations from literature and an AMIA workshop. Journal of the American Medical Informatics Association, 16(3), 291–299. doi: ⁠10.1197/jamia.M2997

Kitsiou, S., Manthou, V., Vlachopoulou, M., & Markos, A. (2010). Adoption and sophistication of clinical information systems in Greek public hospitals: Results from a national web-based survey. In P. Bamidis & N. Pallikarakis (Eds.), XII Mediterranean Conference on Medical and Biological Engineering and Computing 2010 (Vol. 29, pp. 1011–1016). Berlin, Heidelberg: Springer.

Lau, F., Kuziemsky, C., Price, M., & Gardner, J. (2010). A review on systematic reviews of health information system studies. Journal of the American Medical Informatics Association, 17(6), 637–645. doi: ⁠⁠10.1136/jamia.2010.004838

Lester, H., & Campbell, S. (2010). Developing Quality and Outcomes Framework (QOF) indicators and the concept of ‘QOFability’. Quality in Primary Care, 18(2), 103–109.

Marcotte, L., Seidman, J., Trudel, K., Berwick, D. M., Blumenthal, D., Mostashari, F., & Jain, S. H. (2012). Achieving meaningful use of health information technology: A guide for physicians to the EHR incentive programs. Archives of Internal Medicine, 172(9), 731–736. doi: ⁠10.1001/archinternmed.2012.872

National Library of Medicine. (2014). 2014 MeSH descriptor data: Referral and consultation. Retrieved from http://www.nlm.nih.gov/cgi/mesh/2014/MB_cgi?mode=&index=11500

Paré, G., & Sicotte, C. (2001). Information technology sophistication in health care: An instrument validation study among Canadian hospitals. International Journal of Medical Informatics, 63(3), 205–223. doi: 10.1016/S1386-5056(01)00178-2

Raymond, L., & Paré, G. (1992). Measurement of information technology sophistication in small manufacturing businesses. Information Resources Management Journal, 5(2), 4–16. doi: ⁠10.4018/irmj.1992040101

Rousseau, N., McColl, E., Newton, J., Grimshaw, J., & Eccles, M. (2003). Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. British Medical Journal, 326(7384), 314.

Warren, J., Gu, Y., & Humphrey, G. (2012). Usage analysis of a shared care planning system. Paper presented at the The AMIA 2012 Annual Symposium (AMIA 2012), November 3 to 7, Chicago.

Warren, J., White, S., Day, K., Gu, Y., & Pollock, M. (2011). Introduction of electronic referral from community associated with more timely review by secondary services. Applied Clinical Informatics, 2(4), 546–564.