Skip to main content

Handbook of eHealth Evaluation: An Evidence-based Approach: Introduction

Handbook of eHealth Evaluation: An Evidence-based Approach
Introduction

  • Show the following:

    Annotations
    Resources

  • Adjust appearance:

    Font
    Font style
    Color Scheme
    Light
    Dark
    Annotation contrast
    Low
    High
    Margins
  • Search within:
    • Notifications
    • Privacy
  • Project HomeHandbook of eHealth Evaluation: An Evidence Based Approach
  • Projects
  • Learn more about Manifold

Notes

table of contents
  1. Cover
  2. Half Title Page
  3. Title and Copyright
  4. Contents
  5. List of Tables and Figures
  6. Preface
  7. Acknowledgements
  8. Introduction
    1. What is eHealth? by Francis Lau, Craig Kuziemsky
    2. What is eHealth Evaluation? by Francis Lau, Craig Kuziemsky
    3. What is in this Handbook? by Francis Lau, Craig Kuziemsky
  9. Part I: Conceptual Foundations
    1. 1. Need for Evidence, Frameworks and Guidance by Francis Lau
    2. 2. Benefits Evaluation Framework by Francis Lau, Simon Hagens, Jennifer Zelmer
    3. 3. Clinical Adoption Framework by Francis Lau, Morgan Price
    4. 4. Clinical Adoption Meta-Model by Morgan Price
    5. 5. eHealth Economic Evaluation Framework by Francis Lau
    6. 6. Pragmatic Health Information Technology Evaluation Framework by Jim Warren, Yulong Gu
    7. 7. Holistic eHealth Value Framework by Francis Lau, Morgan Price
  10. Part II: Methodological Details
    1. 8. Methodological Landscape for eHealth Evaluation by Craig Kuziemsky, Francis Lau
    2. 9. Methods for Literature Reviews by Guy Paré, Spyros Kitsiou
    3. 10. Methods for Comparative Studies by Francis Lau, Anne Holbrook
    4. 11. Methods for Descriptive Studies by Yulong Gu, Jim Warren
    5. 12. Methods for Correlational Studies by Francis Lau
    6. 13. Methods for Survey Studies by Francis Lau
    7. 14. Methods for eHealth Economic Evaluation Studies by Francis Lau
    8. 15. Methods for Modelling and Simulation Studies by James G. Anderson, Rong Fu
    9. 16. Methods for Data Quality Studies by Francis Lau
    10. 17. Engaging in eHealth Evaluation Studies by Craig Kuziemsky, Francis Lau
  11. Part III: Selected eHealth Evaluation Studies
    1. 18. Value for Money in eHealth: Meta-synthesis of Current Evidence by Francis Lau
    2. 19. Evaluation of eHealth System Usability and Safety by Morgan Price, Jens Weber, Paule Bellwood, Simon Diemert, Ryan Habibi
    3. 20. Evaluation of eHealth Adoption in Healthcare Organizations by Jim Warren, Yulong Gu
    4. 21. Evaluation of Picture Archiving and Communications Systems by Don MacDonald, Reza Alaghehbandan, Doreen Neville
    5. 22. Evaluation of Provincial Pharmacy Network by Don MacDonald, Khokan C. Sikdar, Jeffrey Dowden, Reza Alaghehbandan, Pizhong Peter Wang, Veeresh Gadag
    6. 23. Evaluation of Electronic Medical Records in Primary Care: A Case Study of Improving Primary Care through Information Technology by Lynne S. Nemeth, Francis Lau
    7. 24. Evaluation of Personal Health Services and Personal Health Records by Morgan Price, Paule Bellwood, Ryan Habibi, Simon Diemert, Jens Weber
    8. 25. Evaluating Telehealth Interventions by Anthony J. Maeder, Laurence S. Wilson
  12. Part IV: Future Directions
    1. 26. Building Capacity in eHealth Evaluation: The Pathway Ahead by Simon Hagens, Jennifer Zelmer, Francis Lau
    2. 27. Future of eHealth Evaluation: A Strategic View by Francis Lau
  13. Glossary
  14. About the Contributors

Introduction
Francis Lau and Craig Kuziemsky
What is eHealth?
eHealth is an overarching term that refers to the use of information and communication technology (ICT) in the healthcare sector. Despite being a widely used and popular term there is no single universally agreed-upon definition of eHealth. At the dawn of the 21st century, an editorial on eHealth published in an online journal broadly defined the term as follows:
eHealth is an emerging field in the interaction of medical informatics, public health and business, referring to health services and information delivered or enhanced through the Internet and related technologies. In a broad sense, the term characterizes not only a technical development, but also a state-of-mind, a way of thinking, an attitude, and a commit­ment for networked, global thinking, to improve health care local­ly, regionally, and worldwide by using information and communication technology. (Eysenbach, 2001, p. e20)

According to a scoping review by Pagliari et al. (2005) on the definition and meaning of eHealth, the term first appeared in year 2000 and has since become widely used. Of the 387 relevant articles these authors reviewed in 154 different journals, the most common usages were related to information technology (IT) and telemedicine, with an emphasis on the communicative aspects through networks and the Internet. The definitions they found varied widely in terms of the functions, stakeholders, contexts and theoretical issues involved.
In a systematic review on eHealth studies by Oh, Rizo, Enkin, and Jadad (2005), 51 definitions were found in 430 journals and 1,158 websites. All of the definitions mentioned health and technology. Most included varying aspects of stakeholders, their attitudes, the role of place and distance, and the expected benefits from eHealth. For health it usually referred to care processes rather than outcomes. For technology it was seen as both an enabling tool for a healthcare process or service and also as the resource itself such as a health information website. Moreover, there was an overwhelming sense of optimism in the definitions.
It is important to note that even now the term eHealth is used differently across countries. Here are examples of how the term eHealth is being used in Canada, the United States, Europe and Australia:
  • Canada: eHealth is defined by Health Canada as the application of ICT in the healthcare sector with the electronic health record (EHR) as the basic building block (Health Canada, n.d.). Canadian jurisdictions have all used eHealth to refer to a broad range of ICT-based systems, services and resources in their business and IT plans. These include the electronic medical record (EMR), the personal health record (PHR), consumer health, telehealth/telemedicine, and public health surveillance. Note that in the Canadian context EHR includes information from laboratory and drug information systems, diagnostic imaging repositories, provider and patient registries, telehealth applications, and public health surveillance made available through privacy-protected interoperable platforms (Infoway, n.d.). Other terms that have also been used are electronic health information systems (Infoway, 2004) and more recently digital health (Infoway, 2016).
  • United States: Both the terms health IT and eHealth are in common use. For instance, the Office of the National Coordinator for Health Inform­ation Technology (ONC) HealthIT.gov website section for patients and families explains that health IT “refers to technologies and tools that allow health care professionals and patients to store, share, and analyze health information” (ONC, n.d.). Examples of health IT listed include EHR and PHR that are used to store and share one’s electronic health information. The ONC website also has a section on consumer eHealth programs which are intended to support ONC efforts to empower individuals to improve their health and healthcare through the use of health IT. Examples of eHealth programs include the Meaningful Use Incentives, Blue Button, Sharecare and Innovation Challenges (ONC, 2015).
  • Europe: The European Commission (2012) defines eHealth as “the use of ICT in health products, services and processes combined with organ­isational change in healthcare systems and new skills, in order to improve health of citizens, efficiency and productivity in healthcare delivery, and the economic and social value of health” (p. 3, footnote 1). Examples are ICT-supported “interaction between patients and health-service providers, institution-to-institution transmission of data, or peer-to-peer communi­cation between patients and/or health professionals” to assist in disease prevention, diagnosis, treatment and follow-up (p. 3, footnote 1). Of particular interest is the inclusion of wearable and portable personal health systems collectively referred to as mHealth.
  • Australia: The National E-Health Transition Authority (NEHTA) defines eHealth as “electronically connecting up the points of care so that health information can be shared securely” (NEHTA, n.d.). One example is the My Health Record System, with such products as the shared health summary, discharge summary, specialist letter, eReferral, and prescrip-tion and dispense records that are accessible through the Web-based national consumer portal.

We should point out that, while some regard eHealth as being the same as health informatics, we believe the two are fundamentally different concepts. As described earlier, eHealth is broadly defined as the use of ICT-based systems, services and resources as an enabler in managing health. In contrast, we view health informatics as an academic discipline that deals with the science and practice of health information with respect to its meaning, capture, organization, retrieval, communication and use in decision-making. Since much of the health information is electronic in nature, health informatics also deals with the underlying ICT systems that support the health information in use.
What is eHealth Evaluation?
The Merriam-Webster Dictionary (n.d.) defines evaluation as an act to “judge the value or condition of (something) in a careful and thoughtful way.” By extension, we can define eHealth evaluation as an act to assess whether an eHealth system is functioning and producing the effects as expected. In this context, the eHealth system can be any ICT-based application, service or resource used by organizations, providers, patients or consumers in managing health. Here the concept of health refers to one’s physical and mental condition, and its management refers to a wide range of health services and information resources used to maintain or improve one’s state of well-being. Note that an eHealth system covers not only the technical ICT artefact but also the socio-organizational and environmental factors and processes that influence its behaviours.
The scope of eHealth evaluation can cover the entire life cycle, which spans the planning, design, implementation, use, and maintenance of the eHealth system over time. Depending on the life cycle stage being evaluated there can be different questions raised. For instance, in the planning stage of an eHealth system, one may evaluate whether the intended system is aligned with the organization’s overall strategy, or if an adequate governance process is in place for the sharing of sensitive patient information. In the design stage one may evaluate whether the specifications of the system have been met in terms of its features and behaviour. In the implementation stage one may evaluate whether the deployment of the system is on time and within budget. In the use stage one may evaluate the extent to which the system is used and its impact on provider performance, health outcomes and economic return. In the maintenance stage one may evaluate how well the system is being supported and adapted to accommodate the changing needs of the organization over time.
Different eHealth evaluation approaches have been described in the literature ranging from randomized controlled trials (RCTs), qualitative studies, to usability engineering. These approaches all have unique philosophical and methodological assumptions, leading to confusion as to when and how a particular approach should be applied and the implications involved. Some also regard eHealth evaluation as a form of research that is only relevant to those in academia. Our position is that eHealth evaluation should be scientifically rigorous, relevant to practice, and feasible to conduct in routine settings. By rigorous it means the approach should be credible and defensible. By relevant it means the problem being addressed should be important to the stakeholders. By feasible it means the design should be practical and achievable within a reasonable time frame using reasonable resources.
In their evaluation textbook, Friedman and Wyatt (2006, pp. 25–27) introduced the notion of an evaluation mindset with the following characteristics to distinguish it from research:
  • Tailor the study to the problem, ensuring questions that are relevant to stakeholders are being addressed.
  • Collect data useful for making decisions, focusing on data from processes that are relevant to decision-makers.
  • Look for intended and unintended effects, assuming the effects of an eHealth system cannot be known in advance.
  • Study the system while it is under development and after it is deployed, thus acknowledging the dynamic nature of an eHealth system where its effects can change over time.
  • Study the system in the laboratory and in the field, thereby assessing the performance and effects of an eHealth system in both simulated and natural settings.
  • Go beyond the developer’s point of view, ensuring the perspectives of different stakeholders who are affected by the eHealth system are taken into account.
  • Take the environment into account, understanding the surroundings in which the eHealth system resides.
  • Let the key issues emerge over time, understanding the need for time passage before some issues become evident.
  • Be methodologically broad and eclectic, recognizing the need for and importance of different approaches when planning, conducting and appraising an evaluation study.

In other words, eHealth evaluation should be considered in all endeavours related to an eHealth system because of the significant time and resources required to adopt and adapt these systems. Therefore it is important to find out whether and how much such effort has led to tangible improvement in one’s performance and/or outcomes. In addition, there is an opportunity cost associated with investing in eHealth systems since that investment could be spent elsewhere, for example to reduce surgical wait times by increasing the volume of surgeries performed. Within the current climate of fiscal restraint in the health systems of many jurisdictions, there has to be a strong business case to justify the deployment of eHealth investments.
Thus far, eHealth evaluation studies are often conducted and reported by academic and leading health institutions that have made significant investments in eHealth systems and expert resources to improve their provider performance and health outcomes. While in recent years we have seen increased interest from health organizations in general to engage in eHealth evaluation, what appears to be missing are the necessary eHealth infrastructures and expertise to tackle such activities. By infrastructures we mean the ability to capture and extract the types of clinical and operational data needed to perform the evaluation. By expertise we mean the know-how of the different approaches used in evaluation. Therefore, some form of guidance is needed for stakeholders to engage in eHealth evaluation in a rigorous, relevant and pragmatic fashion. We offer this handbook as one source of such guidance.
What is in this Handbook?
This handbook presents the science and practice of eHealth evaluation based on empirical evidence gathered over many years within the health informatics discipline. The handbook describes different approaches used to evaluate the planning, design, implementation, use and impact of eHealth systems in different health settings. It also provides a snapshot of the current state of knowledge on the consequences of opting for eHealth systems with respect to their effects and implications on provider performance and health outcomes.
The science part of this handbook covers the conceptual foundations of and methodological details in eHealth evaluation. Conceptual foundations refer to the theories, models and frameworks that have been used as organizing schemes and mental roadmaps by eHealth practitioners to illuminate and clarify the makeup, behaviour and effects of eHealth systems beyond that of a technical artefact. Methodological details refer to the different approaches and methodologies that have been used to evaluate eHealth systems. Collectively they provide a rich set of tried and proven methods that can be readily applied or adapted for use by eHealth practitioners responsible for the evaluation of specific eHealth systems.
The practice part covers the ground-level application of the scientific eHealth evaluation approaches described in Parts I and II of the handbook, through the presentation of a set of published case examples in Part III. These case studies provide a summary of the current state of evidence in selected eHealth systems and domains, and how the evaluation studies were designed, conducted and reported. Part IV of the handbook covers the future of eHealth evaluation. It describes the need to build intellectual capacity as a way of advancing the field by ensuring eHealth practitioners are well versed in the science and practice of eHealth evaluation. Also of importance is the need for a more strategic view of eHealth evaluation within the larger healthcare system to be successful.
This handbook has been written as an open electronic reference text or e-book that is to be freely available to students and practitioners wishing to learn about eHealth evaluation or apply the content in their workplace. This e-book is a “living book” in that the co-authors can add such content as new reviews, evaluation methods and case studies as they become available over time. An online learning community is also being considered depending on whether there is sufficient interest from the co-authors and the eHealth communities.
Note that throughout this handbook there are numerous terms mentioned in the form of acronyms and abbreviations. Rather than repeating the full spellings of these terms every time they are mentioned in the chapters, we have opted for the short form and provided a glossary of the acronyms and abbreviations at the end of the handbook (pp. 473–477).
References
European Commission. (2012, December 6). eHealth action plan 2012-2010: Innovative healthcare for the 21st century. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. Brussels: Author. Retrieved from http://ec.europa.eu/health/ehealth/docs/com_2012_736_en.pdf
Eysenbach, G. (2001). What is e-health? Journal of Medical Internet Research, 3(2), e20.
Health Canada. (n.d.). Health care system – eHealth. Ottawa: Author. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/ehealth-esante/index-eng.php
Infoway. (n.d.). Electronic health records – Electronic health record components. Toronto: Author. Retrieved from https://www.infoway-inforoute.ca/en/solutions/electronic-health-records
Infoway. (2004). Annual report, 2003-2004. Corporate plan summary, 2004-2005. Toronto: Author. Retrieved from https://www.infoway-inforoute.ca/en/component/edocman/153-annual-report-2003-2004-corporate-plan-summary-2004-2005/view-document?Itemid=101
Infoway. (2016). Summary corporate plan, 2016-2017. Toronto: Author. Retrieved from https://www.infoway-inforoute.ca/en/component/edocman/2858-summary-corporate-plan-2016-2017/view-document?Itemid=101
Merriam-Webster. (n.d.). Evaluation. Merriam-Webster Dictionary. Retrieved from http://www.merriam-webster.com/
National E-Health Transition Authority (NEHTA). (n.d.). What is eHealth? Sydney, Australia: Author. Retrieved from http://www.nehta.gov.au/get-started-with-ehealth/what-is-ehealth.
Office of the National Coordinator for Health Information Technology (ONC). (n.d.). Health IT for you. Washington, DC: Department of Health and Human Services. Retrieved from https://www.healthit.gov/sites/default/files/consumerfactsheetfinal.pdf
Office of the National Coordinator for Health Information Technology (ONC). (2015). Consumer eHealth program. Washington, DC: Department of Health and Human Services. Retrieved from https://www.healthit.gov/policy-researchers-implementers/consumer-ehealth-program
Oh, H., Rizo, C., Enkin, M., & Jadad, A. (2005). What is eHealth (3): A systematic review of published definitions. Journal of Medical Internet Research, 7(1), e1.
Pagliari, C., Sloan, D., Gregor, P., Sullivan, F., Detmer, D., Kahan, J. P., Ortwijn, W., & MacGillivray, S. (2005). What is eHealth (4): A scoping exercise to map the field. Journal of Medical Internet Research, 7(1), e9.Handbook of eHealth Evaluation Handbook of e

Annotate

Next Chapter
Part 1: Conceptual Foundations
PreviousNext
EPUB
Powered by Manifold Scholarship. Learn more at
Opens in new tab or windowmanifoldapp.org