“Chapter 16: Methods for Data Quality Studies” in “Handbook of eHealth Evaluation: An Evidence-based Approach”

Chapter 16

Methods for Data Quality Studies

Francis Lau

---

16.1 Introduction

The proliferation of eHealth systems has led to a dramatic increase in the volume of electronic health data being collected. Such data are often collected as part of direct patient care delivery to document the patient’s conditions and the care being provided. When the collected health data are used as intended, it is referred to as primary use. Once collected the health data can be used for other purposes such as clinical quality improvement, population health surveillance, health systems planning and research. These are referred to as secondary uses (Safran et al., 2007). In Canada, a further distinction is made where all secondary uses except for research are labelled as health system use (Canadian Institute for Health Information [CIHI], 2013).

The quality of routinely collected eHealth data is a major issue for healthcare organizations. To illustrate, a systematic review by Thiru, Hassey, and Sullivan (2003) on EHR data quality in primary care found a great deal of variability depending on the type of data collected. In 10 EHR studies on sensitivities they found data completeness ranged from 93% to 100% for prescriptions, 40% to 100% for diagnoses, 37% to 97% for lifestyle in alcohol use and smoking, to 25% for socio-economic data. A 2010 review by Chan, Fowles, and Weiner (2010) showed that the variability in the quality of EHR data is an ongoing issue especially with problem lists and medications. Iron and Manuel (2007) reported in their environmental scan of administrative health data quality assessment that: the concepts of accuracy and validity are often confused; there are no standard methods for measuring data quality; and the notion of data quality depends on the purpose for which the data are used. These findings suggest data quality can affect the performance of the eHealth systems and care delivery within organizations.

In this chapter we describe approaches to eHealth data quality assessment that are relevant to healthcare organizations. The approaches cover concepts, practice and implications. Concepts refer to dimensions, measures and methods of data quality assessment. Practice refers to how data quality assessment is done in different settings as illustrated through case examples. Implications refer to guidance and issues in eHealth data quality assessment to be considered by healthcare organizations.

16.2 Concepts of Data Quality

In this section we describe the key concepts in eHealth data quality. These are the conceptual quality dimensions, measures used to assess the quality dimensions, and methods of assessment. These concepts are described below.

16.2.1 Data Quality Dimensions

An overriding consideration when defining data quality concepts is “fitness for use”. This suggests data quality is a relative construct that is dependent on the intended use of the data collected. Different terms have been used to describe the conceptual dimensions of data quality, with no agreement on which should be the standard. Sometimes the meanings of these terms overlap or conflict with each other. Drawing on the studies by Weiskopf and Weng (2013) and Bowen and Lau (2012), we arrived at the following five commonly cited terms for this chapter:

• Correctness – Reflects the true state of a patient, also known as accuracy. An example is whether a high blood pressure value for a patient is true or not.
• Completeness – Covers all truths on a patient, also known as comprehensiveness. An example is the blood pressure measurement that contains the systolic and diastolic pressures, method of assessment, and date/time of assessment.
• Concordance – Agreement of the data with other elements or sources, also known as reliability, consistency and comparability. An example is the use of metformin as a diabetic medication in the presence of a diabetes diagnosis.
• Plausibility – Does the data make sense in what is being measured given what is known from other elements? This is also known as validity, believability and trustworthiness. An example is the presence of a hypertension diagnosis in the presence of recent abnormal blood pressure measurements.
• Currency – Reflects the true state of a patient at a given point in time, also known as timeliness. An example is the presence of a recent blood pressure measurement when considering a hypertensive condition.

In the literature review by Weiskopf and Weng (2013), completeness, correctness and concordance were the most common dimensions assessed. Other less common data quality dimensions described in the literature (Bowen & Lau, 2012) include comprehensibility, informative sufficiency, and consistency of capture and form. These terms are defined below.

• Comprehensibility – The extent to which the data can be understood by the intended user.
• Informative sufficiency – The extent to which the data support an inference on the true state of condition.
• Consistency of capture – The extent to which the data can be recorded reliably without variation by users.
• Consistency of form – The extent to which the data can be recorded reliably in the same medium by users.

16.2.2 Data Quality Measures

The dimensions of correctness and completeness can be quantified through such measures as sensitivity, specificity, positive predictive value and negative predictive value. Quantifying these data quality measures requires some type of reference standard to compare the data under consideration. Using a health condition example such as diabetes, we can take a group of patients where the presence or absence of their condition is known, and compare with their charts to see if the condition is recorded as present or absent. For instance, if the patient is known to have diabetes and it is also recorded in his chart then the condition is true. The comparison can lead to different results as listed below.

• Sensitivity – The percentage of patients recorded as having the condition among those with the condition.
• Specificity – The percentage of patients recorded as not having the condition among those without the condition.
• Positive predictive value – The percentage of patients with the condition among those recorded as having the condition (i.e., condition present).
• Negative predictive value – The percentage of patients without the condition among those recorded as not having the condition (i.e., condition absent).
• Correctness – The percentage of patients with the condition among those recorded as having the condition. It can also be the percentage of patients without the condition among those recorded as not having the condition. These are also known as the positive predictive value and negative predictive value, respectively. Often only positive predict value is used to reflect correctness.
• Completeness – The percentage of patients recorded as having the condition among those with the condition. It can also be the percentage of patients recorded as not having the condition among those without the condition. These are also known as sensitivity and specificity, respectively. Often only sensitivity is used to reflect completeness.

The comparison of the patients’ actual condition against the recorded condition can be enumerated in a 2x2 table (see Table 16.1). The actual condition represents the true state of the patient, and is also known as the reference standard.

Note. From“Defining and evaluating electronic medical record data quality within the Canadian context,” by M. Bowen and F. Lau, 2012, ElectronicHealthcare, 11(1), e5–e13.

16.2.3 Data Quality Methods

Different data quality assessment methods have been described in the literature. Some methods are focused on ways to measure different dimensions of data quality such as correctness and completeness of the data in an eHealth system. Others are concerned with the means of carrying out and reporting data quality assessment studies. There are also methods that apply predefined criteria to identify and validate specific health conditions recorded in the eHealth system. The types of methods covered in this chapter are defined below and elaborated in the next section.

• Validation of data from single and multiple sources – The use of predefined knowledge and query rules to validate the integrity of the data in one or more eHealth systems and/or databases.
• Designing, conducting and reporting data quality studies – The use of a systematic process to carry out data quality assessment studies.
• Identification and validation of health conditions – The use of predefined criteria to identify and validate specific health conditions in an eHealth system or database. The process is also known as case definition or case finding, and the criteria may be from evidence-based guidelines or empirically derived with expert consensus.

16.3 Methods of Data Quality Assessment

This section examines the three types of data quality assessment methods defined in section 16.2.3. Most of the methods were developed as part of data quality assessment studies or as validation of previously developed methods. The analysis in these methods typically involves the use of frequency distributions, cross-tabulations, descriptive statistics and comparison with a reference source for anomalies. These methods are described below.

16.3.1 Validation of Data from Single and Multiple Sources

Brown and Warmington (2002) introduced Data Quality Probe (DQP) as a method to assess the quality of encounter-driven clinical information systems. The principle behind DQP is that predefined queries can be created from clinical knowledge and guidelines to run against the system such as an EHR as measures of its quality. Typically the DQPs examine two or more data recordings that should or should not appear together in the patient record. The most common DQPs involve checking for the presence of a clinical measurement that either should always or never be associated with a diagnosis, and a therapy that either should always or never be accompanied by a diagnosis or clinical measurement. In an ideal system there should be no data inconsistencies detected when the queries are run. Examples are the absence of Hemoglobin A1c (HbA1c) test results on patients with diabetes and prescriptions of penicillin on patients with a penicillin allergy. Two types of errors can be detected: failure to record the data or error of omission, and suboptimal clinical judgment or error of commission. Once detected, these errors should be reported, investigated and corrected in a timely fashion. To be effective, the DQPs should be run periodically with reports of any inconsistencies shared with providers at the individual and/or aggregate level for purposes of education or action.

Kahn, Raebel, Glanz, Riedlinger, and Steiner (2012) proposed a two-stage data quality assessment approach for EHR-based clinical effectiveness research that involves single and multiple study sites. In stage-1, source datasets from each site are evaluated using five types of data quality rules adapted from Maydanchik (2007). In stage-2, datasets from multiple sites are combined, with additional data quality rules applied, to compare the individual datasets with each other. Such multisite comparisons can reveal anomalies that may not be apparent when examining the datasets from one site alone. The five types of stage-1 data quality rules are outlined below (for details, see Kahn et al., 2012, p. S26, Table 3).

• Attribute domain constraints – Rules that restrict allowable values in individual data elements using assessment methods of attribute profiling, optionality, format, precision and valid values to find out-of-range, incorrect format or precision, missing or unusual data values. An example is a birthdate that is missing, unlikely, or in the wrong format.
• Relational integrity rules – Rules that ensure correct values and relationships are maintained between data elements in the same table or across different tables. An example is the use of diagnostic codes that should exist in the master reference table.
• Historical data rules – Rules that ensure correct values and relationships are maintained with data that are collected over time. For example, the recording of HbA1c results over time should correspond to the requested dates/times, follow an expected pattern, and be in a consistent format and unit.
• State-dependent objects rules – Rules that ensure correct values are maintained on data that have expected life cycle transitions. An example is a hospital discharge event should always be preceded by an admission or transfer.
• Attribute dependency rules – Rules that ensure the consistency and plausibility of related data on an entity. An example is the birthdate of a patient should not change over time; neither should a test be ordered on a deceased patient.

For stage-2 data quality rules, the focus is on semantic consistency to ensure data from different sites have the same definitions so they can be aggregated and analyzed meaningfully. The rules typically compare frequency distributions, expected event rates, time trends, missing data and descriptive statistics (e.g., mean, median, standard deviation) of the respective datasets to detect patterns of anomalies between sites. An example is the need to distinguish random versus fasting glucose tests or dramatic differences in the prevalence of diabetes between sites.

In both stage-1 and stage-2 data quality assessment, documentation is needed to articulate the rationale, methods and results of the assessments done. Often there can be hundreds of data quality rules depending on the complexity of the eHealth system and databases involved. Therefore some type of prioritization is needed on the key data elements and assessments to be included. The outputs generated can be daunting especially if every error encountered on each record is reported. The detailed errors should be grouped into categories and summarized into key areas with select performance indicators to report on the overall quality of the system or database, such as the percentage of records that pass all the data quality tests.

16.3.2 Designing, Conducting and Reporting Data Quality Studies

Bowen and Lau (2012, p. e10) published a 10-step method for conducting a context-sensitive data quality evaluation. These steps provide a systematic means of planning, conducting and reporting a data quality evaluation study that takes into account the intended use of the data and the organizational context. The 10 steps are:

• Focus on an activity that requires the use of the data being evaluated.
• Determine the context in which the activity is carried out, including the intent, tasks, people and results.
• Identify the tools/resources needed to evaluate the quality of the data and their alignment with the activity.
• Determine the degree of fitness between the activity and data being evaluated and the acceptable level of fitness.
• Select an appropriate data quality measurement method for the chosen fitness dimension being evaluated.
• Adapt the measurement method depending on the context and use data quality probes to aid in the evaluation.
• Apply the tools/resources identified in step-3 to evaluate the quality of the data being measured.
• Document the output of the fitness evaluation for each data being measured.
• Describe the overall fitness of the important data with the activity in context.
• Present data quality evaluation findings and provide feedback on the quality/utility of the data and improvement.

16.3.3 Identification and Validation of Health Conditions

Wright and colleagues (2011, pp. 2–6) developed and validated a set of rules for inferring 17 patient problems from medications, laboratory results, billing codes and vital signs found in the EHR system. A six-step process was used to develop and validate the rules. Additional analyses were done to adjust rule performance based on different rule options. These steps are listed below:

• Automated identification of problem associations – based on a previous study that identified lab-problem and medication-problem associations against gold standard clinical references and significant co-occurring statistics.
• Selection of problems of interest – an initial list of problems ranked according to three criteria: related pay-for-performance initiatives; existing decision support rules in EHR; strength of identified associations.
• Development of initial rules – confirmed relevant lab tests, medications and billing codes with medical references, added relevant free-text entries, then drafted initial rules which were reviewed by expert clinicians.
• Characterization of initial rules and alternatives – focused on patients with at least one relevant medication, lab test, billing code and vital sign but without the problem recorded in EHR; applied initial rules to identify rule-positive and rule-negative patients, then conducted chart review on a sample of patients to see if they have the problem (i.e., chart-positive, chart-negative); derived sensitivities, specificities, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) of initial rule options by varying their thresholds such as lab values, drugs and counts.
• Selection of the final rule – had expert clinicians review different rule options for each problem with varying sensitivities, specificities, PPV and NPV; selected final rules with high PPV over specificity.
• Validation of the final rule – repeated above steps using an independent patient dataset from the same population.
• Additional analyses – derived sensitivity, specificity, PPV and NPV with coded problems then billing data only to adjust final set of rules based on F-measure for higher PPV over sensitivity (false negatives versus false positives).

An example of the final rules for diabetes is shown below (Wright et al., 2011, supplementary data):

• Rule 0: code or free-text problem on problem list for diabetes.
• Rule 1: any HbA1c result greater than or equal to 7.
• Rule 2: 2 or more ICD-9 billing codes in diabetes (250, 250.0, 250.00, 250.01, 250.02, 250.03, 250.1, 250.10, 250.11, 250.12, 250.13, 250.2, 250.20, 250.21, 250.22, 250.23, 250.3, 250.30, 250.31, 250.32, 250.33, 250.4, 250.41, 250.42, 250.43, 250.5, 250.50, 250.51, 250.52, 250.53, 250.6, 250.60, 250.61, 250.62, 250.63, 250.7, 250.71, 250.72, 250.73, 250.8, 250.80, 250.81, 250.82, 250.83, 250.9, 250.91, 250.92, 250.93).
• Rule 3: at least one medication in injectable anti-diabetic agents or oral anti-diabetic agents.

16.4 Case Examples

This section includes two published examples of eHealth data quality studies: one is on multisite data quality assessment while the other is on primary care EMRs.

16.4.1 Multisite Data Quality Assessment

Brown, Kahn, and Toh (2013) reviewed multisite data quality checking approaches that have been field-tested in distributed networks for comparative effectiveness research such as the Observational Medical Outcomes Partnership in the United States. Typically these networks employ a common data model and different types or levels of data quality checks for cross-site analysis as described in Kahn’s 2-stage data quality assessment approach (Kahn et al., 2012). These data quality-checking approaches are:

• Common data model adherence – These are checks on extracted data against the common data model dictionary for consistency and adherence to the model. They are: (a) syntactic correctness on transformed variable names, values, lengths and format meeting data model specifications; (b) table structure and row definition correctness; and (c) cross-table variable relationships for consistency. Examples are valid codes for sex and diagnosis, linkable tables by person or encounter identifier, and presence of valid enrolment for all prescription records.
• Data domain review – These are checks on the frequency and proportion of categorical variables, distribution and extreme values for continuous variables, missing and out-of-range values, expected relationships between variables, normalized rates and temporal trends. The domains may cover enrolment, demographics, medication dispensing, prescribing, medication utilization, laboratory results and vital signs. Examples of checks are enrolment periods per member, age/sex distribution, dispensing/prescriptions per user per month, diagnoses/procedures per encounter, weight differences between men and women, and number of tests conducted per month.
• Review of expected clinical relationships with respect to anomalies, errors and plausibility – Within and cross-site co-occurrence of specific clinical variables should be assessed, such as the rate of hip fractures in 60- to 65-year-old females, male pregnancy and female prostate cancer.
• Member/study-specific checks – These are checks to ensure proprietary and privacy-related policies and regulations for specific members are protected, such as the inclusion of unique product formulary status, clinical/procedure codes, patient level information and tables with low frequency counts; and to detect data variability across study sites such as the exposure, outcome and covariates under investigation.

16.4.2 Improving Data Quality in Primary Care EMRs

The Canadian Primary Care Sentinel Surveillance Network (CPCSSN) is a pan-Canadian practice-based research network made up of 10 regional networks involving more than 1,000 primary health care providers in eight provinces and territories (CPCSSN, n.d.). Its mission is to improve primary health care delivery and outcomes, epidemiological surveillance, research excellence, and knowledge translation. The effort involves the extraction and use of EMR data from community-based primary health care practices to inform and improve the management of the most common chronic diseases in Canada. Here we describe the CPCSSN Data Presentation Tool (DPT) that has been developed to improve the management of individual and groups of patients within and across practices (Moeinedin & Greiver, 2013). In particular, we emphasize the effort undertaken to improve the quality of the EMR data and its impact in the DPT initiative.

• DPT purpose and features – The DPT is an interactive software developed as a quality dashboard to generate custom reports at the provider, office and organizational levels. It uses EMR data that have been de-identified, cleaned and standardized through a systematic process which are then returned to the providers for use in quality improvement purposes. These include the ability to improve data quality at the practice level, re-identify at-risk patients for tracking and follow-up, and produce custom reports such as prescribing patterns and comorbidities in specific chronic diseases (Williamson, Natarajan, Barber, Jackson, & Greiver, 2013; Moeinedin & Greiver, 2013).
• DPT study design – The DPT was implemented and evaluated as a quality improvement study in a family health team in Ontario. The study used mixed methods to examine practice change before and after DPT implementation from May to August 2013. Sixty-one primary care providers took part in the study. The qualitative component included field notes, observations, key informant interviews and a survey. The quantitative component measured the change in data quality during that period {Moeinedin & Greiver, 2013; Greiver et al., 2015).
• Data quality tasks – CPCSSN has developed an automated approach to cleaning EMR data. The data cleaning algorithms are used to identify missing data, correct erroneous entries, de-identify patients, and standardize terms. In particular, the standardization process can reduce the various ways of describing the same item into one term only. Examples are the use of kilograms for weights, one term only for HA1c, and three terms only for smoking status (i.e., current smoker, ex-smoker, never smoked). The cleaned data are then returned to the providers, allowing them to assess the data cleaning needed at the local level within their EMRs. To ensure transparency, CPCSSN has published its data cleaning algorithms in peer-reviewed journals and on its website (Greiver et al., 2012; Keshavjee et al., 2014).
• Key findings – The family health team in the DPT study was able to identify missing and non-standardized data in its EMRs. The DPT was used to produce quality reports such as the prevalence of hypertension and dementia in the region, re-identification of high-risk patients for follow-up, and specific medication recall. Overall, the updating and standardization of the EMR data led to a 22% improvement in the coding of five chronic conditions and the creation of registries for these conditions (Moeinedin & Greiver, 2013; Greiver et al., 2015).

 16.5 Implications

As healthcare organizations become more dependent on eHealth systems for their day-to-day operations, the issue of eHealth data quality becomes even more prominent for the providers, administrators and patients involved. The consequence of poor-quality data can be catastrophic especially if the care provided is based on incomplete, inaccurate, inaccessible or outdated information from the eHealth systems. The data quality assessment approaches described in this chapter are empirically-derived pragmatic ways for organizations to improve the quality and performance of their eHealth systems. To do so, there are a number of policy and practice implications to be considered.

For policy implications, healthcare organizations need to be aware of the task-dependent nature of data quality, or fitness for use, in order to embark on data quality policies that are most appropriate for their needs. An important first step is to adopt a consistent set of eHealth data quality concepts with clearly defined evaluation dimensions, measures and methods. More importantly, it should be recognized that data quality evaluation is only a means to an end. Once the state of eHealth data quality has been identified, there must be remedial actions with engaged owners and users of the data to rectify the situation. Last, organizational leaders should foster a data quality culture that is based on established best practices.

For practice implications, healthcare organizations need to dedicate sufficient resources with the right expertise to tackle data quality as a routine practice. Data quality evaluation is a tedious endeavour requiring attention to detail that includes meticulous investigation into the root causes of the data quality issues identified. There should be detailed documentation on all of the data quality issues found and remedial actions taken to provide a clear audit trail for references. Last, since providers are responsible for a substantial portion of the routine clinical data being collected, they need to be convinced of the value in having high-quality data as part of patient care delivery.

16.6 Summary

This chapter described eHealth data quality assessment approaches in terms of the key concepts involved, which are the data quality assessment dimensions, measures and methods used and reported. Also included in this chapter are two examples of data quality assessment studies in different settings, and related implications for healthcare organizations.

References

Bowen, M., & Lau, F. (2012). Defining and evaluating electronic medical record data quality within the Canadian context. ElectronicHealthcare, 11(1), e5–e13.

Brown, J., & Warmington, V. (2002). Data quality probes – exploiting and improving the quality of electronic patient record data and patient care. International Journal of Medical Informatics, 68(1–3), 91–98.

Brown, J., Kahn, M., & Toh, S. (2013). Data quality assessment for comparative effectiveness research in distributed data networks. Medical Care, 51(8 suppl 3), S22–S29.

Canadian Institute for Health Information (CIHI). (2013). Better information for improved health: A vision for health system use of data in Canada. Ottawa, ON: Author.

Chan, K. S., Fowles, J. B., & Weiner, J. P. (2010). Electronic health records and the reliability and validity of quality measures: A review of the literature. Medical Care Research and Review, 67(5), 503–527.

Canadian Primary Care Sentinel Surveillance Network (CPCSSN). (n.d.). Canadian primary care sentinel surveillance network (website). Retrieved from http://cpcssn.ca/

Greiver, M., Drummond, N., Birwhistle, R., Queenan, J., Lambert-Lanning, A., & Jackson, D. (2015). Using EMRs to fuel quality improvement. Canadian Family Physician, 61(1), 92.

Greiver, M., Keshavjee, K., Jackson, D., Forst, B., Martin, K., & Aliarzadeh, B. (2012). Sentinel feedback: Path to meaningful use of EMRs. Canadian Family Physician, 58(10), 1168.

Kahn, M. G., Raebel, M. A., Glanz, J. M., Riedlinger, K., & Steiner, J. F. (2012). A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Medical Care, 50(7), S21–S28.

Keshavjee, K., Williamson, T., Martin, K., Truant, R., Aliarzadeh, B., Ghany, A., & Greiver, M. (2014). Getting to usable EMR data. Canadian Family Physician, 60(4), 392.

Iron, K., & Manuel, D. G. (2007, July). Quality assessment of administrative data (Quaad): An opportunity for enhancing Ontario’s health data (ICES Investigative Report). Toronto: Institute for Clinical Evaluative Sciences.

Maydanchik, A. (2007). On hunting mammoths and measuring data quality. Data Management Review, 6, 14–15, 41.

Moeinedin, M., & Greiver, M. (2013). Implementation of the data presentation tool in a primary care organization. Presentation to Trillium Research Day, Toronto, June 19, 2013. Retrieved from http://www.trilliumresearchday.com/documents/2013_Moeinedin_DPT_implementation_Final_June192013.pdf

Safran, C., Bloomrosen, M., Hammond, W. E., Labkoff, S., Markel-Fox, S., Tang, P. C., & Detmer, D. (2007). Toward a national framework for the secondary use of health data: An American Medical Informatics Association white paper. Journal of American Medical Informatics Association, 14(1), 1–9.

Thiru, K., Hassey, A., & Sullivan, F. (2003). Systematic review of scope and quality of electronic patient record data in primary care. British Medical Journal, 326(7398), 1070–1072.

Weiskopf, N. G., & Weng, C. (2013). Methods and dimensions of electronic health record data quality assessment: Enabling reuse for clinical research. Journal of American Medical Informatics Association, 20(1), 144–151.

Williamson, T., Natarajan, N., Barber, D., Jackson, D., & Greiver, M. (2013). Caring for the whole practice — the future of primary care. Canadian Family Physician, 59(7), 800.

Wright, A., Pang, J., Feblowitz, J. C., Maloney, F. L., Wilcox, A. R., Ramelson, H. Z., Schneider, L. I., & Bates, D. W. (2011). A method and knowledge base for automated inference of patient problems from structured data in an electronic medical record. Journal of American Medical Informatics Association, 18(6), 859–867.