“Chapter 15: Methods for Modelling and Simulation Studies” in “Handbook of eHealth Evaluation: An Evidence-based Approach”

Chapter 15

Methods for Modelling and Simulation Studies

James G. Anderson, Rong Fu

15.1 Introduction

Evaluation of the implementation and use of an eHealth System such as electronic health records (EHR), decision support systems, computerized provider order entry and telehealth frequently require the use of methods other than traditional randomized control trials. Moehr (2002) points out some of the problems involved in evaluating eHealth applications. He suggests that these evaluations need to include the dynamic process of adaptation of the system and its environment rather than just its technical features. Conventional evaluation methods do not adequately describe the dynamic nature of eHealth systems.

Regression analysis, network analysis and computer simulation provide alternative methodologies that help investigators better understand the impact of these systems on workflow, cost, effectiveness, and quality of healthcare delivery. The analytical approaches described below focus on different aspects of eHealth systems. Regression analysis examines attribute data to answer such questions as which physician characteristics predict EHR use. Network analysis explores relationships among members of a network, such as a medical practice, to determine how communication among members affects use of clinical practice guidelines. The focus of simulation is on the system level to explore issues such as how to alleviate crowding in the Emergency Department (ED). In general, the chapter is aimed at practitioners with little or no experience in designing and implementing these methods.

15.2 Regression Analysis

Regression analysis is a statistical tool that attempts to estimate an outcome (also known as the response variable) based on a set of predictors (also known as the explanatory variables). Specifically, a regression model explores how the typical value of the response variable changes given different values of the explanatory variable(s). There are several regression methods used widely in quantitative research: linear, logistic, and multivariate regression models. Apart from these common regression models, time series regression and structural equation modelling are relatively new regression tools in eHealth studies.

Regression models allow explanations and predictions of past, present, or future events with information obtained from internal or external sources. Regression analysis can be performed with both cross-sectional data and panel data. Cross-sectional data are collected by observing many subjects (e.g., individuals, hospitals) at a particular point in time. Panel data, also called longitudinal data or cross-sectional time series data, are collected by observing the same subjects at two or more time periods. In order to build a regression model, one needs to determine the response variable(s), the explanatory variable(s), the time frame, and the specific analytical model.

15.2.1 Types of Regression Models

Linear regression is the most basic and commonly used technique for determining how the response variable is affected by changes in one or more explanatory variables. Whereas a simple linear regression model predicts the outcome based on a single explanatory variable, a multiple linear regression model uses two or more explanatory variables to predict the response variable. In linear regression analysis, the relationship between the predictor(s) and the outcome is typically plotted as a straight line that best approximates all the individual data points. A possible research question that can be answered using linear regression is the following: What is the association between eHealth literacy (the ability to seek and understand health information from electronic sources) and colorectal cancer knowledge (see Mitsutake, Shibata, Ishii, & Oka, 2012)?

Logistic regression is an extension of linear regression that allows one to predict categorical outcomes based on predictors. A categorical outcome is one that takes on one of a fixed number of possible values (e.g., the blood type of a person has four categories: A, B, AB or O). In eHealth evaluation, a logistic regression model is commonly used to model the linear relationship between a binary outcome variable (a categorical variable with only two values) and one or more predictors. The binary outcome variable usually takes the value of 0 or 1 to indicate the absence or presence of an outcome (e.g., 0 = survival, 1 = death). Thereby, logistic regression models are widely used to predict the odds of the presence of the outcome based on the values of the predictors. A possible research question that can be answered using logistic regression is the following: Do eHealth literacy and patient-centred communication affect the odds of post-visit online health information seeking (see Li, Orrange, Kravitz, & Bell, 2014)?

A multivariate regression model estimates more than one outcome based on a set of predictors. This model attempts to determine a formula that describes how elements in a set of variables respond simultaneously to changes in others. The main characteristic that distinguishes multivariate regression from multiple regression is the use of multiple outcomes. A possible research question that can be answered using multivariate regression is the following: What is the relationship between basic electronic medical records and outcomes such as having a patient safety event, impatient death, and hospital readmission (see Encinosa & Bae, 2011)?

A time series regression model predicts a future outcome based on the outcome history and the transfer of dynamics from a series of predictors. In order to use this model, one needs to have measurements that are taken from the same subjects at successive time points (e.g., hospital readmission rates in five separate years). A possible research question that can be answered using time series regression analysis on panel data is the following: How do hospital information technologies affect hospital operating expenses across three years (see Bardhan & Thouin, 2013)?

Structural equation modelling (SEM) is a family of related statistical procedures designed to determine and validate a proposed process and/or a theoretical model. SEM can be used to examine research questions involving the indirect or direct observation of one or more predictors and/or one or more outcomes. Some common SEM methods include confirmatory factor analysis, path analysis, and latent growth modelling (Kline, 2010).

Confirmatory factor analysis is a multivariate statistical procedure used to verify the hypothesized relationship between observed variables and their underlying latent constructs. The eHealth literacy study presented by Neter and Brainin (2012) is a good example of confirmatory factor analysis. Path analysis is an extension of multiple regression that evaluates causal models by examining the relationship between one or more explanatory and response variables. A case in point is that Cho, Park, and Lee (2014) used a path analysis to examine the effects of several cognitive factors (e.g., health consciousness, health information orientation) on the extent of health-app use. Latent growth modelling is a longitudinal analysis technique that can estimate growth over a period of time. Anderson, Ramanujam, Hensel, and Sirio (2010) used latent growth curve analysis to examine longitudinal trends in the quarterly number of errors and associated corrective actions reported by 25 hospitals.

15.2.2 Evaluating Electronic Medical Records using Multivariate Regression

Encinosa and Bae (2011) used multivariate regression models to examine whether electronic medical records (EMRs) contain costs in the Patient Protection (PP) and Affordable Care Act (ACA) reforms to reduce patient safety events. In this study, data were obtained from the 2007 MarketScan Commercial Claims and Encounter Database, the 2007 American Hospital Association Annual Survey and its Information Technology Supplement. The methodological components for this study are summarized as follow:

Research question #1 – What is the relationship between basic EMRs and the probability that a surgery will have a patient safety event?

Outcome #1 – Patient safety event, measured by “surgical-related patient safety events” with 12 indicators, “nursing-related patient safety events” with 5 indicators, and other “likely preventable patient safety events” with 7 indicators.

Research question #2 – What is the relationship between basic EMRs and the probability of inpatient death within 90 days following surgery?

Outcome #2 – Death, measured by any inpatient hospital death occurring within 90 days following surgery.

Research question #3 – What is the relationship between basic EMRs and the probability of a 90-day readmission for surgeries?

Outcome #3 – Readmission, measured by any overnight stays at an inpatient hospital within 90 days following surgery.

Research question #4 – What is the relationship between basic EMRs and total 90-day hospital expenditures?

Outcome #4 – Hospital expenditures, measured by transacted prices including all inpatient hospital, physician, drug, and lab payments for any inpatient stay occurring up to 90 days following surgery.

Analytical model – Multivariate regression models.
Time frame – A cross-sectional design where the data were collected all at the same time or within a short time frame.
Predictors – Basic EMRs, a binary variable (1 = having basic EMRs; 0 = no basic EMRs) measured by whether a hospital has the following eight basic EMR functionalities in at least one major clinical unit: demographic characteristics of patients, problem lists, medication lists, discharge summaries, laboratory reports, radiologic reports, diagnostic test results, and computerized provider order entry for medications.
Covariates – Age, sex, suffering from hypertension, suffering from diabetes, suffering from liver disease, suffering from depression, obesity, etc.

The study findings showed that EMRs did not reduce the rate of patent safety events. However, once a patient safety event occurs, EMRs reduced death by 34%, readmissions by 39%, and hospital expenditures by $4,840 (16%). These results were obtained by examining the relationships between multiple outcomes and predictors in multivariate regression models after controlling for covariates. Taken together, the findings of this study indicate that EMRs contain costs in the PP and ACA reforms by better coordinating care to rescue patients from medical errors once a patient safety event occurs.

15.2.3 Evaluating Health Information Technologies using Time Series Regression on Panel Data

Bardhan and Thouin (2013) applied time series regression models to panel data to estimate the impact of health information technologies (HIT) on hospital operating expenses and the quality of healthcare delivery during the three-year period. In this study, data on hospital information technologies usage was obtained from the Dorenfest Institute for Health Information Technology Research database. Data on hospital process quality measures was obtained from the U.S. Department of Health and Human Services (HHS) Hospital Compare Program. Data on hospital operating expenses was obtained using publicly available data from the U.S. Center for Medicare and Medicaid Services. The methodological components of this study are summarized as follow:

Research question #1 – What is the relationship between implementation of HIT and the quality of healthcare delivery indicated by levels of conformance to evidence-based best practices?

Outcome #1 – Acute myocardial infarction, with eight process quality measures.
Outcome #2 – Heart failure, with four process quality measures.
Outcome #3 – Pneumonia, with seven process quality measures.
Outcome #4 – Surgical infection prevention, with two process quality measures.

Research question #2 – What is the relationship between implementation of HIT and hospital operating expenses?

Outcome #5 – Operating expense per bed, measured by dividing the hospitals’ operating costs for providing healthcare services by the total number of beds in use.

Analytical model – Time series regression on panel data.
Time frame – A three-year longitudinal design where data were collected each year from 2004 to 2006.
Predictors – Clinical systems (six factors), financial systems (four factors), scheduling systems (one factor), and human resources systems (two factors).
Covariates – Hospital type, hospital size, hospital case mix index, hospital location, and teaching status.

The study findings indicated that usage of clinical information systems and patient scheduling applications was associated with greater conformance with best practices for treatment of heart attacks, heart failures, and pneumonia. Whereas financial and human resource management systems were associated with lower hospital operating expenses, implementation of clinical information systems and scheduling systems was associated with higher operating expenses. Taken together, the findings of this study suggest that investments in HIT have a positive impact on the overall quality of healthcare delivery. However, the effect of HIT implementation on hospital operating expenses is mixed and needs to be factored into consideration when making implementation decisions.

15.3 Social Network Analysis

Social network analysis comprises a set of methods that can be used to investigate patterns of relationships among individuals, departments, organizations, etc. These relationships affect behaviour such as adoption and use of electronic medical records, decision support systems, and telehealth (Anderson, 2002a).

15.3.1 Social Networks and Physician Adoption of Electronic Health Records

Zheng, Padman, Krackhardt, Johnson, and Diamond (2010) studied how social interactions influence physician adoption of EHRs. A survey was used to identify social interactions among 40 residents and 15 attending physicians in an ambulatory care primary care practice. Social network analysis was used to determine the relation of the structure of interactions to physicians’ rates of utilization of the EHR.

Objective – To examine how social influences affect physician EHR adoption.
Research Hypothesis #1 – The level of EHR adoption can be predicted by cohesion over the professional network, the friendship network and the perceived influence network among physicians. Cohesion reflects how well physicians were connected to each other and whether key individuals possess pivotal positions in the network.
Research Hypothesis #2 – The level of EHR adoption can be predicted by structural equivalence of the professional network, the friendship network and the perceived influence network. Structural equivalence measures the similarity in interaction patterns in the three types of networks.
Data Collection – A social network survey was administered to 55 physicians affiliated with an outpatient primary care practice associated with a 512-bed tertiary care medical facility. The survey asked physicians: (a) to name their colleagues that they consulted with on patient care issues; (b) which colleagues they considered to be personal friends; and (c) which colleagues influenced them to use the EHR. A second survey assessed personal characteristics such as gender, work experience, computer literacy, attitude toward use of the EHR, etc.
Outcome Measures – Rates of EHR usage for patient data documentation or retrieval of patient data were calculated for each physician.
Analysis – The analysis assessed the influence of the social structure and structural equivalence on rates of EHR system usage.

Results of the analysis indicated that several physicians provided the bulk of information concerning patient care in the professional network. In contrast, analysis of the perceived influence network suggested that influence over adoption of the EHR rarely occurred in the clinic. Analysis of the friendship network indicated that residents who had named the same attending physician as a personal friend exhibited comparable EHR adoption behaviour.

The results of this study suggest that identifying opinion leaders who developed friendships with many other members of a medical practice can be used to promote the diffusion of innovations like EHRs.

15.3.2 The Use of Social Networks to Study Outbreaks of Hospital-acquired Infections

Cusumano-Towner, Li, Tuo, Krishnan, and Maslove (2013) used social network analysis to study outbreaks of nosocomial infections among hospital patients. EMR data were used to model contacts among patients through shared rooms and contact with healthcare workers. The social networks were used to conduct probabilistic simulations of outbreaks of Methicillin-resistant Staphylococcus aureus (MRSA) and influenza.

Objective – The objectives of this study were: (a) to create a social network of hospital patients using data from an EMR; (b) to use the network to simulate nosocomial outbreaks of MRSA and influenza; and (c) to identify potential interventions.
Data – EMR data were extracted from a clinical data warehouse covering hospital admissions over a 70-day period. Data from days 35 to 45 were used in the simulation. Shared contact with healthcare workers was determined from metadata contained in clinical documents.
Analysis – The data files were used to construct networks of pairwise connections between individual patients based on sharing of rooms and shared contact with healthcare workers. The two networks were combined into a graph of epidemiologic links that change over time. This social network was used to develop a probabilistic model of the spread of infection through the hospital. The probabilistic model was used to simulate outbreaks of MRSA and influenza and to test the potential effects of infection control measures. Infections originating in the ED, a medical step-down unit, and a psychiatry unit were simulated.

The results indicated that the risk of spreading influenza between wards was greatest between two psychiatric units, and between the cardiac unit and coronary care unit. The ED and operating areas had low levels of incoming infection. Its simulations predicted that vaccination of the staff could markedly decrease the spread of influenza. Simulation of outbreaks of MRSA predicted that an infection originating in the medical step-down unit spread to the ICU, the neurosurgical, orthopedic, and cardiac units. The risk of transmission of MRSA was substantially mitigated by a 50% increase in hand hygiene compliance. The benefits of the approach used in this study are: First, it used existing data collected during clinical care and stored in an EMR to construct patient networks; second, these data reflect local staffing and patient flow patterns unique to the hospital under investigation; third, this approach allows for real-time updating of the patient networks; and fourth, social networks can be used to model the effects of infection control interventions such as patient isolation, hand hygiene, and staff vaccination.

15.4 Simulation Modelling

The development of a computer simulation model begins with a system analysis. Important elements of the system and relationships among them are identified. Data used in defining the system may be obtained from system logs, interviews, questionnaires, work sampling and expert judgment (Anderson, 2002b, 2002c).

There are several types of simulation: discrete event, continuous, and agent-based. In a discrete event model, items (e.g., patients, medical orders, etc.) flow through a network of components. Each component performs a function (e.g., MRI) before the item (e.g., patient) moves on to the next component (e.g., service). For a discrete event simulation of a computerized physician order entry system, see Anderson et al. (1988).

Continuous simulation is used when an eHealth system involves a continuous flow of information, patients, material, or other resources. The model is comprised of state variables (e.g., the number of patients in the system at any time), rates of flow (e.g., entry of new patients and exit of existing patients), and control variables that affect the flow rates. For a model of a drug ordering and delivery system of a hospital, see Anderson, Jay, Anderson, and Hunt (2002). Continuous simulation models such as systems dynamics are comprised of a set of differential equations representing feedback loops among state variables that represent the system under investigation. This feedback structure is what makes the system adapt over time.

Agent-based models are used to determine the global consequences of interactions among individual agents. Agents generate emergent behaviour by interacting with one another according to a small set of rules. Interactions among agents give rise to the system’s behaviour. For an agent-based model of the healthcare system of a refugee community, see Anderson, Chaturvedi, and Cibulskis (2007).

Once a simulation model has been constructed, it is validated against historical data that describes the behaviour of the system over time. A major advantage of simulation is that the model can be used to make modifications (e.g., the number of RNs or MDs in the ER) and predict effects on the system’s performance. Such computer experiments can be performed without disrupting the practice setting.

15.4.1 Forecasting Emergency Department Crowding using Discrete Event Simulation

Hoot et al. (2009) applied discrete event simulation to forecasting emergency department crowding. The growing problem of crowding in emergency departments is resulting in delayed treatment, prolonged transport, increased mortality, and financial burdens on hospitals. This study developed and validated a method of forecasting future emergency department crowding using discrete event simulation.

Objective – Implement and validate a simulation model to be used in forecasting future crowding in emergency departments.
Research question #1 – Could a simulation model accurately predict future crowding based on existing data from emergency department information systems?
Research question #2 – How well does the model predict future values of several crowding measures in a real operational setting?
Methods – A discrete event simulation model was constructed and validated based on data from an adult ED in a tertiary care, urban Level 1 trauma center. The model describes patient arrivals, evaluation, treatment and potential hospital admissions.
Input variables – The following data were collected in an adult emergency department of a tertiary care medical centre during a three-month period:

Time of initial registration in the ED.
Time placed in an ED bed.
Time of request for a hospital bed.
Time of discharge from the ED.
Patient’s triage category.
Whether the patient left the ED without being seen.

Outcome measures – The model forecasts the following crowding measures:

Number of patients in the waiting room.
Average waiting time.
Occupancy – total number of patients in ED beds.
Length of stay in the ED.
Number of patients awaiting hospital admission.
Average time patients waited for hospital admission.
Probability of ambulance diversion due to ED crowding.

The simulation model provides accurate real-time forecasts of inputs, throughputs and output measures of crowding up to eight hours in the future. The tool could be used in other EDs that have information systems that provide the six patient-level variables.

15.4.2 Preventing Adverse Drug Events using Continuous Simulation

Anderson, Jay, Anderson, and Hunt (2002) developed a computer simulation model to evaluate information technology applications designed to detect and prevent hospital medication errors that may result in adverse drug events. Model parameters were estimated from a study of prescription errors on two hospital medical/surgical units and used for the baseline simulation. The study evaluated five prevention strategies.

Objective – To develop a model that can be used to evaluate the effectiveness of IT applications designed to prevent medical errors that may result in adverse drug events (ADEs) in hospitals.
Research Question #1 – How effective are each of five interventions in reducing ADEs in a hospital?
Research Question #2 – How effective are each of the interventions in reducing additional days of hospitalization that result from ADEs?
Research Question #3 – How effective are each of the five interventions in reducing the cost resulting from ADEs?
Methods – A computer simulation model was constructed to represent the medication delivery system in a hospital. STELLA, a continuous simulation software package, was used to construct the model. Parameters of the model were estimated from a study of prescription errors on two hospital medical/surgical units.
Input Variables – The following variables were obtained from a study of two hospital units:

Number of medication orders entered into the hospital information system.
Number and type of errors made in writing prescriptions.
Severity of medication errors.
Rates of ADEs resulting from medication errors.
Rates of errors committed during the dispensing and administration of medications were based on published studies.

Interventions – The model was used to evaluate the following interventions:

Provision of drug information by the Hospital Information System when prescriptions are written.
Adoption of physician computer order entry.
Implementation of a unit dosing system in the hospital pharmacy.
Implementation of a barcoding system for medications dispensed in the hospital pharmacy.
Implementation of a comprehensive medication delivery system that includes all four interventions.

Outcome measures – The model was used to estimate the following measures for each intervention:

Number of errors for each stage of the delivery system (i.e., prescription, transcription, dispensation, administration, and total errors).
Rates of medication errors.
Rates of ADEs.
ADEs by intervention.
Additional days of hospitalization resulting from ADEs by intervention.
Additional hospital costs resulting from ADEs

The model simulates the four stages of a hospital medication delivery system. The results indicate that clinical information systems are potentially a cost-effective means of preventing ADEs in hospitals. The results of this study indicate that an integrated medication delivery system could save up to 1,226 days of excess hospitalization and $1.4 million in associated costs in a large tertiary care hospital.

15.4.3 An Agent-based Simulation Designed to Model Events in Hospital Patient Transfers that may lead to Adverse Events

Dunn and colleagues (2011) used agent-based simulation to analyze risk associated with hospital inpatient transfers of patients. The model simulates the possible trajectories routine processes may take that deviate from prescribed work practice. The analysis helps to determine which deviations may lead to adverse events and estimates how often these deviations result in adverse events. The two adverse events that are analyzed are misidentification of a patient and compromised infection control.

Objective – The aim of this study was to develop a model that can be used for risk assessment of hospital inpatient transfers.
Research Question #1 – Identify the variety of possible trajectories in hospital patient transfers that deviate from prescribed work practice.
Research Question #2 – To calculate the probability of adverse events resulting from the deviation in work practices.
Methods – An agent-based simulation model was designed to represent the chain of common violations of work practices that may lead to adverse events during hospital patient transfers. Clinicians and hospital information systems were represented as interacting agents. The model simulates the inpatient transfer process using four human agents, six objects and 186 activities. Model parameters were estimated from data obtained from 101 patient transfers. Two situations were modelled: patient misidentification and violations of infection control.
Input Variables – Transfers of 101 inpatients were observed. The likelihood of violations such as failure to perform patient identification checks and failure to use adequate infection control precautions were estimated from these data.
Outcome Measures – Repeated simulations were run to determine the range of potential chains of events that evolve due to individual violations by interacting agents in the hospital. The likelihood of a risk of an adverse event occurring by the end of the chain of events was calculated for patient misidentification and for violations of infection control procedures.

The analysis found that 95% of simulations of patient misidentification and infection control violations were unique. This finding suggests that the process of inpatient transfer deviates from prescribed work practices in a wide variety of ways. The risk of adverse events occurring was estimated to be 8% for misidentification and 24% for violations of infection control. The value of this simulation approach over more traditional risk analysis methods is that it permits the user to quantitatively examine how individual violations of prescribed work practices combine to create risk.

15.5 Implications

The applicability of the methods described in this chapter depends upon the nature of the eHealth application, the availability of data, and the assumptions upon which the analytic approach is based. Regression analysis is used to predict one or more outcome measures based on a set of predictor variables. The purpose is to make inferences to a population from which the sample of data is drawn. The data must meet certain assumptions such as: (a) the sample of data must accurately represent the population from which it is drawn; (b) the variables are accurately measured; and (c) the relationship between the dependent variables and independent variables is correctly specified. However, there are alternative ways of estimating the equations’ parameters in the event that some of these assumptions are not met.

Network analysis takes a different approach. It is used to study relationships between individuals, objects, or events, such as communication or professional ties. The nature of the relations among actors in the network may affect an actor’s perceptions or actions. Data in this instance is collected on the relations among a set of actors who make up the network (e.g., a medical practice). The analysis tries to uncover significant and influential positions in the network such as opinion leaders.

Simulation involves building a dynamic model that represents a system (e.g., the emergency department of a hospital). The model involves inputs (e.g., patient arrivals) and outputs (e.g., average time to process a patient). Simulation runs are made and the behaviour of the system is observed (e.g., crowding in the ED).

The three methods can also be used in conjunction with one another. A study of the cost-effectiveness of coronary bypass graft operations by Anderson, Harshbarger, Weng, and Anderson (2002) utilized SEM to estimate parameters of a computer simulation model. Cusumano-Towner and colleagues (2013) used social network analysis and computer simulation to study outbreaks of nosocomial infections among hospital patients.

15.6 Summary

This chapter describes three different analytic approaches to the evaluation of eHealth systems. These methods are regression analysis, network analysis, and computer simulation. Case studies are provided as examples of these approaches.

References

Anderson, J. G. (2002a). Evaluation in health informatics: Social network analysis. Computers in Biology and Medicine, 32(3), 179–193.

Anderson, J. G. (2002b). Evaluation in health informatics: Computer simulation. Computers in Biology and Medicine,32(3), 151–164.

Anderson, J. G. (2002c). A focus on simulation in medical informatics. Journal of the American Medical Informatics Association, 9(5), 554–556.

Anderson, J. G., Chaturvedi, A. R., & Cibulskis, M. (2007). Simulation tools for developing policies for complex systems: Modeling the health and safety of refugee communities. Health Care Management Science,10(4), 331–339.

Anderson, J. G, Harshbarger, W., Weng, W. C., & Anderson, M. M. (2002). Modeling the costs and outcomes of cardiovascular surgery. Health Care Management Science, 5(2), 103–111.

Anderson, J. G., Jay, S. J., Anderson, M., & Hunt, T. N. J. (2002). Evaluating the capability of information technology to prevent adverse drug events: A computer simulation approach. Journal of the American Medical Informatics Association,9(5), 479–490.

Anderson, J. G., Jay, S. J., Cathcart, L., Clevenger, S. J., Perry, J., & Anderson, M. M. (1988). Physician use of clinical information systems: A computer simulation model. In System science in health care, vol. 1: Information in health care systems (pp. 421–424). Paris: Masson.

Anderson, J. G., Ramanujam, R., Hensel, D., & Sirio, C. (2010). Reporting trends in a regional medication error data-sharing system. Healthcare Management Science,13(1), 65–73.

Bardhan, I. R., & Thouin, M. F. (2013). Health information technology and its impact on the quality and cost of healthcare delivery. Decision Support Systems,55(2), 438–449.

Cho, J., Park, D., & Lee, H. E. (2014). Cognitive factors of using health apps: Systematic analysis of relationships among health consciousness, health information orientation, eHealth literacy, and health app use efficacy. Journal of Medical Internet Research,16(5), e125. doi: ⁠10.2196/jmir.3283

Cusumano-Towner, M., Li, D. Y., Tuo, S., Krishnan, G., & Maslove, D. M. (2013). A social network of hospital acquired infection built from electronic medical record data. Journal of the American Medical Informatics Association,20(3), 427–434.

Dunn, A. G., Ong, M. S., Westbrook, J., Magrabi, F., Coiera, E., & Wobcke, W. (2011). A simulation framework for mapping risks in clinical processes: The case of in-patient transfers. Journal of the American Medical Informatics Association,18(3), 259–266.

Encinosa, W. E., & Bae, J. (2011). Health information technology and its effects on hospital costs, outcomes, and patient safety. Inquiry,48(4), 288–303.

Hoot, N. R., LeBlanc, L. J., Jones, I., Scotty, R. L., Zhou, C., Gadd, C. S., & Aronsky, D. (2009). Forecasting emergency department crowding: A prospective real-time evaluation. Journal of the American Medical Informatics Association,16(3), 338–345.

Kline, R. B. (2010). Principles and practice of structural equation modeling (3rd ed.). New York: Guilford Press.

Li, N., Orrange, S., Kravitz, R. L., & Bell, R. A. (2014). Reasons for and predictors of patients’ online health information seeking following a medical appointment. Family Practice,31(5), 550–556. doi: ⁠10.1093/fampra/cmu034

Mitsutake, S., Shibata, A., Ishii, K., & Oka, K. (2012). Association of eHealth literacy with colorectal cancer knowledge and screening practice among internet users in Japan. Journal of Medical Internet Research, 14(6), e153. doi: ⁠10. 2196/jmir.1927

Moehr, J. R. (2002). Evaluation of health information systems: Beyond efficiency and effectiveness. Computers in Biology and Medicine,32(3), 111–112.

Neter, E., & Brainin, E. (2012). eHealth literacy: Extending the digital divide to the realm of health information. Journal of Medical Internet Research,14(1), e19. doi: ⁠10.2196/jmir.1619

Zheng, K., Padman, R., Krackhardt, D., Johnson, M. P., & Diamond, H. S. (2010). Social networks and physician adoption of electronic health records: Insights from an empirical study. Journal of the American Medical Informatics Association,17(3), 328–336.

Chapter 16: Methods for Data Quality Studies

EPUB