Skip to main content
“Chapter 27: Future of eHealth Evaluation” in “Handbook of eHealth Evaluation: An Evidence-based Approach”
Chapter 27
Future of eHealth Evaluation
A Strategic View
Francis Lau
27.1 Introduction
In this handbook we have examined both the science and the practice of eHealth
evaluation in different contexts. The first part of the handbook, on conceptual
foundations, has provided examples of organizing schemes that can help make
sense of eHealth as interdependent sociotechnical systems, and how these
systems can be defined and measured. Depending on the purpose and scope of the
planned evaluation, an eHealth system may be conceptualized under different
assumptions and viewed from multiple lenses in terms of its makeup, behaviours
and consequences. For example, an eHealth system may be evaluated in a narrow
context at the micro level as an artefact for its technical performance under
the information system quality dimension of the Benefits Evaluation Framework.
Alternatively, the evaluation may take on a broader scope focusing on the
macro-level governance, standards, and funding dimensions of the Clinical
Adoption Framework.
The second part of the handbook concerns methodological details and has provided
a collection of research approaches that can be applied to address different
eHealth evaluation questions. They range from such quantitative methods as
comparative and correlational studies to such qualitative methods as
descriptive and survey studies. There are also methods that utilize both
qualitative and quantitative data sources such as economic evaluation,
modelling and data quality studies. In addition, there are published guidelines
that can enhance the reporting quality of eHealth evaluation studies. The
repertoire of such methods offers ample choice for the evaluator to plan,
conduct, publish and appraise eHealth evaluation studies to ensure they are
simultaneously rigorous, pragmatic and relevant. The third part of the
handbook, on selected eHealth evaluation studies, has provided detailed
examples of field studies to demonstrate how the scientific principles of
select eHealth evaluation frameworks and methods have been applied in practice
within different settings.
The last part of the handbook on future directions addresses, first, the need to
build capacity in eHealth evaluation and, second, the shifting landscape for
eHealth evaluation within the larger healthcare delivery system. This final
chapter of the handbook offers some observations on what this future may hold
in the years ahead. This discussion is outlined under the topics of eHealth as
a form of complex intervention, the need for guiding principles on eHealth
evaluation methods, and taking a more strategic view of eHealth evaluation as
part of the larger healthcare system. The chapter closes with some final
remarks on key take-home messages on eHealth evaluation for readers.
27.2 eHealth as a Complex Intervention
There is growing recognition that healthcare interventions can be highly complex
in nature. This can be due to the number of interacting components that exist
in a given intervention, the types of behaviours required by those delivering
and receiving the intervention, the number of targeted groups or organizations
involved, variability in expected outcomes, and the degree of tailoring
permitted in the intervention. Such complexity can lead to variable study
findings and an apparent lack of tangible impact from the intervention (Craig
et al., 2008).
According to Shcherbatykh, Holbrook, Thabane, and Dolovich (2008), eHealth
systems are considered complex interventions since they are often made up of
multiple technical and informational components influenced by different
organizational, behavioural and logistical factors. The technical components
include the eHealth system’s hardware, software, interface, customizability, implementation and integration. The informational components include
the operational logic, clinical expertise, clinical importance, evidence-based
guidelines, communication processes and promotion of action. The organizational
factors that can influence the system include its financing, management and
training, the degree of vendor support, the stance of local opinion leaders,
and feedback given and received. The behavioural factors include user
satisfaction, attitudes, motivation, expectations, interdisciplinary
interaction and self-education. The logistical factors include system design,
workflow, compatibility, local user involvement, ownership, technological
sophistication and convenience of access. Collectively these components and
factors can interact in an unpredictable fashion over time to produce the types
of emergent system functions, behaviours and consequences that are observed.
For complex eHealth interventions, Eisenstein, Lobach, Montgomery, Kawamoto, and
Anstrom (2007) have emphasized the need to understand the intervention
components and their interrelationships as prerequisites for effectiveness
evaluation. These authors suggested that the overall complexity of an
intervention can be a combination of the complexity of the problem being
addressed, the intervention itself, inputs and outputs of the healthcare
setting and the degree of user involvement. The group has developed the Oxford
Implementation Index as a methodology that can be applied to eHealth evaluation
(Montgomery, Underhill, Gardner, Operario, & Mayo-Wilson, 2013). This index has four implementation components that can
affect intervention fidelity: intervention design; intervention delivery by
providers; intervention uptake by patients; and contextual factors. These have
been organized as a checklist to assess intervention study results. The
checklist items are listed below.
- Intervention design – refers to core components of the intervention and the sequence of intended activities for the intervention group under study, as well as the usual practice activities for the control group.
- Intervention delivery by providers – refers to what is actually implemented which can be affected by staff qualifications, quality, use of system functions, adaptations and performance monitoring over time, such as the use of electronic preventive care reminders.
- Intervention uptake by participants – refers to the experience of those receiving the actual intervention that has been implemented, such as the patients who receive electronic preventive care reminders.
- Contextual factors – refers to characteristics of the setting in which the study occurs such as socio-economic characteristics, culture, geography, legal environment and service structures.
May and colleagues (2011) have proposed a Normalization Process Theory (NPT) to explain implementation processes for complex interventions in healthcare
that can be extended to eHealth systems. The NPT has four theoretical constructs aimed to illuminate the embedding of a practice
through what people actually do and how they actually work. These constructs
are briefly described below (May et al., 2011, p. 2).
- Coherence – processes to understand, promote, or inhibit the intervention as a whole to its users. They require investments of meaning made by the participants.
- Cognitive participation – processes that promote or inhibit users’ enrolment and legitimation of the intervention. They require investments of commitment by the participants.
- Collective action – processes that promote or inhibit the enactment of the intervention by its users. They require investments of effort made by the participants.
- Reflexive monitoring – processes that promote or inhibit the comprehension of the effects of the intervention. They require investments in appraisal made by the participants.
To translate NPT into practice, May et al. (2011) created an online survey as a Web-based
toolkit to be completed by non-experts. The survey was field tested with 59
participants who responded to the questions and provided feedback to improve
the content. The final version of the online survey has 16 statements where
respondents can record their extent of agreement to each statement along a
sliding bar from “completely agree” to “don’t agree at all”. See the Appendix for the 16 NPT statements and refer to the NPT website to access the toolkit (Normalization Process Theory [NPT], n.d.).
Mair and colleagues (2012) have conducted an explanatory systematic review to
examine factors that promote or inhibit the implementation of eHealth systems
using NPT as the organizing scheme. Of the 37 papers included in the review, they found
there was little attention paid to: (a) work to make sense of the eHealth
systems in terms of their purposes and values and to establish their value to
users, and planning the implementation; (b) factors that would promote or
inhibit stakeholder engagement and participation; (c) the effects on changing
roles and responsibilities; (d) risk management; and (e) ways to reconfigure
the implementation processes through user-produced knowledge. These findings
suggest further work is needed to better understand the wider social framework
and implications to be considered when introducing new technologies such as
eHealth systems. The NPT may be a new and promising way to unpack the complexities associated with
eHealth interventions that are currently not well addressed by traditional
evaluation methods.
27.3 Guiding Principles for eHealth Evaluation Methods
There is a growing demand for governments and healthcare organizations to
demonstrate the value of eHealth investments in ways that are rigorous and
relevant. As such, eHealth evaluation is no longer considered an academic
research activity but one that should be integral to the adoption of eHealth
systems by healthcare organizations. As eHealth evaluation is increasingly
being done by practitioners who may not be experienced in various evaluation
approaches, there is an urgent need to ensure these evaluation studies are
methodologically robust and reproducible. To explain and emphasize this need,
Poon, Cusack, and McGowan (2009) have identified a set of common evaluation
challenges faced by eHealth project teams funded by the Agency for Healthcare
Research and Quality in the United States to deploy eHealth systems in their
organizations. These were mostly non-academic institutions with project teams
that had a paucity of evaluation experience. The challenges found included
having: evaluation as an afterthought; unrealistic evaluation scope and
inadequate resources; a mismatch between the metrics chosen and the system
being implemented; inadequate statistical power; limited data available; an improper
comparison group; insufficient details on data collection and analysis; and an
exclusive focus on quantitative methods.
There have been calls for the establishment of guiding principles to make
eHealth evaluation more rigorous, relevant and pragmatic. For instance, Liu and
Wyatt (2011) have argued for the need for more RCTs to properly assess the impact of eHealth systems. Rather than promoting the
universal use of RCTs, however, they have pointed to the need for clarity on how to match study
methods to evaluation questions. Specifically, an RCT is considered appropriate if there are significant costs and risks involved,
since the study can answer questions on whether and how much an eHealth system
improves practitioner performance and patient outcomes. Lilford, Foster, and
Pringle (2009) have advocated the use of multiple methods to examine
observations at the patient and system level, as well as the use of formative
and summative evaluation approaches performed as needed by internal and
external evaluators during different stages of the eHealth system life cycle.
Similarly, Catwell and Sheikh (2009) have suggested the need for continuous
evaluation of eHealth systems as they are being designed, developed and
deployed in ways that should be guided by the business drivers, vision, goals,
objectives, requirements, system designs and solutions.
Greenhalgh and Russell (2010) have offered an alternative set of guiding
principles for the evaluation of eHealth systems. Their principles call for a
fundamental paradigm shift in thinking beyond the questions of science, beyond
the focus on variables, and beyond the notions of independence and objectivity.
The argument being made is that eHealth evaluation should be viewed as a form
of social practice framed and enacted by engaging participants in a social
situation rather than a form of scientific testing for the sole purpose of
generating evidence. As such, the evaluation should be focused on the
enactments, perspectives, relationships, emotions and conflicts of participants
that cannot be reduced to a set of dependent and/or independent variables to
explain the situation under study. It also recognizes that evaluation is
inherently subjective and value-laden, which is at odds with the traditional
scientific paradigm of truth seeking that is purportedly independent and
objective. In particular, these authors have compared these alternative
paradigms under seven key quality principles described below (Greenhalgh & Russell, 2010, Table 1, p. 3).
- Hermeneutic circle versus statistical inference – Understanding of the situation through iterating between its different parts and the whole that they form rather than an adequately powered, statistical and representative sample from the population being studied.
- Contextualization versus multiple interacting variables – Recognizing the importance of context, its interpretive nature and how it emerges from a particular social and historical background rather than reliance on examining the relationships of a predefined set of input, output, mediating and moderating variables.
- Interaction and immersion versus distance – Focusing on engagement and dialogue between the evaluator and stakeholders and immersing in the socio-organizational context of the system under study rather than maintaining a clear separation for independence and objectivity.
- Theoretical abstraction and generalization versus statistical abstraction and generation – Relating observations and interpretations into a coherent and plausible model to achieve generalizability rather than demonstrating validity, reliability and reproducibility among study variables and findings.
- Reflexivity versus elimination of bias – Understanding how the evaluator’s background, interests and perceptions can affect the questions posed, data collected and interpretations made rather than minimizing bias through rigorous methodological designs.
- Multiple interpretations versus single reality amenable to scientific measurement – Being open to multiple viewpoints and perspectives from different stakeholders rather than pursuing a single reality generated through robust study designs and methods.
- Critical questioning versus empiricism – There may be hidden political influences, domination and conflicts that should be questioned and challenged rather than assuming a direct relationship between the reality and the study findings based solely on the precision and accuracy of the measurements made.
From these quality principles we can expect different types of knowledge to be
generated based on the underlying paradigms that guide the evaluation effort.
For instance, under the traditional scientific paradigm we can expect the
evaluation to: (a) employ objective methods to generate quantitative estimates
of the relationships between predefined input and output variables; (b)
determine the extent to which the system has achieved its original goals and
its chain of reasoning; and (c) produce quantitative statistical generalization
of the findings with explanatory and predictive knowledge as the end point.
By contrast, an evaluation under an interpretive/critical paradigm would tend
to: (a) co-create learning through dialogue among stakeholders to understand
their expectations, values and framing of the system; (b) define the meaning of
success through the struggles and compromises among stakeholder groups; and (c)
provide a contextualized narrative with multiple perspectives on the system and
its complexities and ambiguities (Greenhalgh & Russell, 2010, Table 2, p. 3).
27.4 A Strategic View of eHealth Evaluation
Since 2001 the Canadian federal government has invested $2.1 billion in eHealth
through incremental and targeted funding allotments. Its provincial and
territorial counterparts have also invested in cost-shared eHealth projects
that included client and provider registries, interoperable EHRs, primary care EMRs, drug and lab information systems, diagnostic imaging systems, telehealth and
consumer health. Despite such major investments, the evidence on eHealth
benefits has been mixed to date (Lau, Price, & Bassi, 2014). Similarly, mixed findings are found in other countries as well.
In the United Kingdom, progress toward an EHR for every patient has fallen far short of expectations, and the scope of the
national programme for IT has been reduced significantly without any reduction in cost (National Audit
Office [NAO], 2011). In the United States, estimated projected savings from health IT were $81 billion annually (Hillestead et al., 2005). Yet the overall results in
the U.S. have been mixed. This may have been due to the sluggish adoption of eHealth
systems that are neither interoperable nor easy to use, and the failure of
healthcare organizations and providers to re-engineer their care processes,
including provider payment schemes, in order to reap the full benefits of
eHealth systems (Kellermann & Jones, 2013).
To guide eHealth policies, there is a need to expand the scope of eHealth
evaluation beyond individual systems toward a more strategic view of where, how
and in what ways eHealth fits into the broader healthcare system to demonstrate
the overall return on value of the investments made. Kaplan and Shaw (2004)
have suggested the evaluation of eHealth system success should extend beyond
its technical functionality to include a mix of social, behavioural and
organizational dimensions at a more strategic level that involve specific
clinical contexts, cognitive factors, methods of development and dissemination,
and how success is defined by different stakeholders. In order to evaluate
these dimensions Kaplan and Shaw (2004, p. 215) have recommended 10 action
items, which have been adapted as follows for this handbook:
- Address the concerns of individuals/groups involved in or affected.
- Conduct single and multisite studies with different scopes, types of settings and user groups.
- Incorporate evaluation into all phases of an eHealth project.
- Study failures, partial successes and changes in project definition or outcome.
- Employ evaluation approaches that take into account the shifting nature of healthcare and project environment, including formative evaluations.
- Incorporate people, social, organizational, cultural and ethical issues into the evaluation approaches.
- Diversify evaluation approaches and continue to develop new approaches.
- Conduct investigations at different levels of analysis.
- Integrate findings from different eHealth systems, contextual settings, healthcare domains, studies in other disciplines, and work that is not published in traditional research outlets.
- Develop and test theory to inform both further evaluation research and informatics practice.
In Canada, Zimlichman et al. (2012) have conducted semi-structured interviews
with 29 key Canadian eHealth policy and opinion leaders on their domestic
eHealth experiences and lessons learned for other countries to consider. The
key findings are for eHealth leaders to emphasize the following: direct
provider engagement; a clear business case for stakeholders; guidance on
standards; access to resources for mid-course corrections of standards as
needed; leveraging the implementation of digital imaging systems; and
sponsoring large-scale evaluations to examine eHealth system impact in
different contexts.
Similarly, at the 2011 American College of Medical Informatics (ACMI) Winder Symposium, a group of health informatics researchers and practitioners
examined the contributions of eHealth to date by leading institutions, as well
as possible paths for the nation to follow in using eHealth systems and
demonstrating its value in healthcare reform (Payne et al., 2011). In terms of
the role of eHealth in reducing costs and improving the quality of healthcare,
the ACMI group suggested that eHealth systems can provide detailed information about
healthcare, reduce costs in the care of individual patients, and support
strategic changes in healthcare delivery.
To address the question of whether eHealth is worth the investment, the ACMI group have suggested the need to refocus the effort on more fundamental but
strategic issues of what evidence is needed, what is meant by eHealth, what is
meant by investment and how it is measured, and how we determine worth. These
questions are briefly discussed below.
- What evidence is needed? Currently we do not routinely collect the data needed to help us determine the actual costs of eHealth systems and their economic and health impacts, including any unintended consequences. To do so on a continual basis would require structural changes to our healthcare operations and data models.
- What is meant by eHealth? We need to develop ways to articulate eHealth systems in terms of their functionality and co-factors that affect their design, deployment and use. Examples of co-factors include such areas as policies, process re-engineering, training, organization and resource restructuring, and change management. Also important is the recognition of the therapeutic dosage effect where there can be a differential impact with varying levels of eHealth system investment and adoption.
- What is meant by investment and how it is measured? We need to clarify who is making the investment, the form of that investment and the scope of the intended impacts. These can vary from the micro level that is focused on the burden and benefits for individual providers, to the macro level with a national scope in terms of societal acceptance of eHealth and its effects. For measurement, currently there are no clear metrics for characterizing the appropriate costs and benefits that should be measured, nor are there standardized methods for measuring them.
- How do we determine worth? While value is typically expressed in terms of dollars expended, productivity and effectiveness, we do not know what constitutes a realistic return on eHealth investments. This may depend on the initial states with respect to the level of investment made and the extent of eHealth system adopted. For example, with limited eHealth investment a healthcare organization may achieve only limited impact, whereas with a higher level of investment and broader stakeholder support one may achieve significant impact. For meaningful comparison these initial states may need to be normalized across studies and, given the small amount of evidence available to date, the focus should be on how to collect appropriate evidence in the future rather than pursuing a definitive answer on the worth of eHealth systems at this time.
27.5 Concluding Remarks
This chapter examined the future direction of eHealth evaluation in terms of its
shifting landscape within the larger healthcare system, including the growing
recognition of eHealth as a form of complex intervention, the need for
alternate guiding principles on eHealth evaluation methods, and taking a more
strategic view of eHealth evaluation as part of the larger system. This future
should be built upon the cumulative knowledge acquired over many years in
generating a better understanding of the role, makeup, behaviour and impact of
eHealth systems through the application of rigorous methods in pragmatic
evaluation studies that are relevant to multiple stakeholder groups. While
there is still mixed evidence to date on the performance and impact of eHealth
systems, the exemplary case studies provided throughout this handbook should
offer some guidance on how leading healthcare organizations have planned,
adopted and optimized their eHealth systems in order to reap tangible benefits
over time.
In conclusion, the key messages for readers in terms of the future of eHealth
evaluation and its implications within the larger healthcare system are
summarized below.
- eHealth evaluation as an evolving science can advance our understanding and knowledge of eHealth as complex sociotechnical interventions within the larger healthcare system. At the same time, eHealth evaluation as a social practice can generate the empirical evidence needed to link the value of eHealth to the investments made from multiple stakeholder perspectives.
- There is a growing recognition of the need to apply theory-guided, multi-method driven and pragmatic design in eHealth evaluation that is based on best practice principles in order to build on the cumulative knowledge in health informatics.
- There is some evidence to suggest that, under the right conditions, the adoption of eHealth systems is correlated with clinical and health system benefits. Presently this evidence is stronger in care process improvement than in health outcomes, and the positive economic return is based on only a small set of studies. The question now is not whether eHealth can demonstrate benefits, but under what conditions can these benefits be realized and maximized.
References
Catwell, L., & Sheikh, A. (2009). Evaluating eHealth interventions: The need for continuous
systemic evaluation. Public Library of Science Medicine, 6(8), e1000126.
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: new guidance. Swindon, UK: Medical Research Council. Retrieved from
http://www.mrc.ac.uk/documents/pdf/complex-interventions-guidance/
Eisenstein, E. L., Lobach, D. F., Montgomery, P., Kawamoto, K., & Anstrom, K. J. (2007). Evaluating implementation fidelity in health information technology
interventions. In proceedings of the American Medical Informatics Association (AMIA) annual symposium, 2007, Chicago (pp. 211–215). Bethesda, MD: AMIA.
Greenhalgh, T., & Russell, J. (2010). Why do evaluations of eHealth programs fail? An alternative
set of guiding principles. Public Library of Science Medicine, 7(11), e1000360. doi: 10.1371/journal.pmed.1000360
Hillestad, R., Bigelow, J., Bower, A., Girosi, F., Meili, R., Scoville, R., & Taylor, R. (2005). Can electronic medical record systems transform health care?
Potential health benefits, savings, and costs. Health Affairs, 24(5), 1103–1117.
Kaplan, B., & Shaw, N. T. (2004). Future directions in evaluation research: people,
organizational and social issues. Methods of Information in Medicine, 43(3), 215–231.
Kellermann, A. L., & Jones, S. S. (2013). What it will take to achieve the as-yet-unfulfilled
promises of health information technology. Health Affairs, 32(1), 63–68.
Lau, F., Price, M., & Bassi, J. (2014). Toward a coordinated electronic record strategy for Canada.
In A. Cardon, J. Dixon, & K. R. Nossal (Eds.), Toward a healthcare strategy for Canadians (pp. 111–134). Montréal: McGill-Queen’s University Press.
Lilford, R. J., Foster, J., & Pringle, M. (2009). Evaluating eHealth: How to make evaluation more
methodologically robust. Public Library of Science Medicine, 6(11), e1000186.
Liu, J. L. Y., & Wyatt, J. C. (2011). The case for randomized controlled trials to assess the impact of clinical
information systems. Journal of the American Medical Informatics Association, 18(2), 173–180.
Mair, F. S., May, C., O’Donnell, C., Finch, T., Sullivan, F., & Murray, E. (2012). Factors that promote or inhibit the implementation of
e-health systems: an explanatory systematic review. Bulletin of World Health Organization, 90, 257–264. doi: 10.2471/BLT.11.099424
May, C. R., Finch, T., Ballini, L., MacFarlane, A., Mair, F., Murray, E.,
Treweek, S., & Rapley, T. (2011). Evaluating complex interventions and health technologies
using normalization process theory: development of a simplified approach and
web-enabled toolkit. BMC Health Services Research, 11, 245. doi: 10.1186/1472-6963-11-245
Montgomery, P., Underhill, K., Gardner, F., Operario, D., & Mayo-Wilson, E. (2013). The Oxford implementation index: a new tool for
incorporating implementation data into systematic reviews and meta-analyses. Journal of Clinical Epidemiology, 66(8), 874–882.
National Audit Office. (2011). The national programme for IT in the NHS: an update on the delivery of detailed care records systems. London: Author. Retrieved from
https://www.nao.org.uk/report/the-national-programme-for-it-in-the-nhs-an-update-on-the-delivery-of-detailed-care-records-systems/
Normalization Process Theory (NPT). (n.d.). Implementing and evaluating complex interventions. Retrieved from http://www.normalizationprocess.org/
Payne, T. H., Bates, D. W., Berner, E. S., Berstam, E. V., Covvey, H. D.,
Frisse, M. E., … Ozbolt, J. (2013). Healthcare information technology and economics. Journal of American Medical Informatics Association, 20(2), 212–217.
Poon, E. G., Cusack, C. M., & McGowan, J. J. (2009). Evaluating healthcare information technology outside of
academia: observations from the National Resource Centre for healthcare
information technology at the Agency for Healthcare Research and Quality. Journal of American Medical Informatics Association, 16(5), 631–636.
Shcherbatykh, I., Holbrook, A., Thabane, L., & Dolovich, L. (2008). Methodologic issues in health informatics trials: the
complexities of complex interventions. Journal of American Medical Informatics Association, 15(5), 575–580.
Zimlichman, E., Rozenblum, R., Salzberg, C. A., Jang, Y., Tamblyn, M., Tamblyn,
R., & Bates, D. W. (2012). Lessons from the Canadian national health information
technology plan for the United States: opinions of key Canadian experts. Journal of the American Medical Informatics Association,19(3), 453–459.
Appendix
NPT Survey Statements
(Source: May et al., 2011, pp. 8–9)
- Participants distinguish the intervention from current ways of working
- Participants collectively agree about the purpose of the intervention
- Participants individually understand what the intervention requires of them
- Participants construct potential value of the intervention for their work
- Key individuals drive the intervention forward
- Participants agree that the intervention should be part of their work
- Participants buy into the intervention
- Participants continue to support the intervention
- Participants perform the tasks required by the intervention
- Participants maintain their trust in each other’s work and expertise through the intervention
- The work of the intervention is allocated appropriately to participants
- The intervention is adequately supported by its host organization
- Participants access information about the effects of the intervention
- Participants collectively assess the intervention as worthwhile
- Participants individually assess the intervention as worthwhile
- Participants modify their work in response to their appraisal of the intervention
EPUB
Manifold uses cookies
We use cookies to analyze our traffic. Please decide if you are willing to accept cookies from our website. You can change this setting anytime in Privacy Settings.