Comparison of Information Content of Structured and Narrative Text Data Sources on the Example of Medication Intensification

Objective

To compare information obtained from narrative and structured electronic sources using anti-hypertensive medication intensification as an example clinical issue of interest.

Design

A retrospective cohort study of 5,634 hypertensive patients with diabetes from 2000 to 2005.

Measurements

The authors determined the fraction of medication intensification events documented in both narrative and structured data in the electronic medical record. The authors analyzed the relationship between provider characteristics and concordance between intensifications in narrative and structured data. As there is no gold standard data source for medication information, the authors clinically validated medication intensification information by assessing the relationship between documented medication intensification and the patients' blood pressure in univariate and multivariate models.

Results

Overall, 5,627 (30.9%) of 18,185 medication intensification events were documented in both sources. For a medication intensification event documented in narrative notes the probability of a concordant entry in structured records increased by 11% for each study year (p < 0.0001) and decreased by 19% for each decade of provider age (p = 0.035). In a multivariate model that adjusted for patient demographics and intraphysician correlations, an increase of one medication intensification per month documented in either narrative or structured data were associated with a 5-8 mm Hg monthly decrease in systolic and 1.5-4 mm Hg decrease in diastolic blood pressure (p < 0.0001 for all).

Conclusion

Narrative and structured electronic data sources provide complementary information on anti-hypertensive medication intensification. Clinical validity of information in both sources was demonstrated by correlation with changes in blood pressure.     

Evaluation of a Physician Informatics Tool to Improve Patient Handoffs

Objective

To facilitate patient handoffs between physicians, the computerized patient handoff tool (PHT) extracts information from the electronic health record to populate a form that is printed and given to the cross-cover physician. Objectives were to: (1) evaluate the rate at which data elements of interest were extracted from the electronic health record into the PHT, (2) assess the frequency for needing information beyond that contained in the PHT and where obtained, (3) assess physician's perceptions of the PHT, (4) identify opportunities for improvement.

Design

Observational study.

Measurements

This multi-method study included content coding of PHT forms, end of shift surveys of cross-cover resident physicians, and semi-structured interviews to identify opportunities for improvement. Thirty-five of 42 internal medicine resident physicians participated. Measures included: 1264 PHT forms coded for type of information, 63 end-of-shift surveys of cross-cover residents (residents could participate 2 times), and 18 semi-structured interviews.

Results

For objective 1, patient identifiers and medications were reliably extracted (>98%). Other types of information—allergies and code status—were more variable (<50%). For objective 2, nearly a quarter of respondents required information from physician notes not available in the PHT. For objective 3, respondents found that the PHT supported handoffs but indicated that it often excluded the assessment and plan. For objective 4, residents suggested including treatment plans.

Conclusions

The PHT reliably extracts information from the electronic health record. Respondents found the PHT to be suitable, although opportunities for improvement were identified.     

Influenza surveillance using electronic health records in the American Indian and Alaska Native population

Objective

Increasing use of electronic health records (EHRs) provides new opportunities for public health surveillance. During the 2009 influenza A (H1N1) virus pandemic, we developed a new EHR-based influenza-like illness (ILI) surveillance system designed to be resource sparing, rapidly scalable, and flexible. 4 weeks after the first pandemic case, ILI data from Indian Health Service (IHS) facilities were being analyzed.

Materials and methods

The system defines ILI as a patient visit containing either an influenza-specific International Classification of Disease, V.9 (ICD-9) code or one or more of 24 ILI-related ICD-9 codes plus a documented temperature ≥100°F. EHR-based data are uploaded nightly. To validate results, ILI visits identified by the new system were compared to ILI visits found by medical record review, and the new system''s results were compared with those of the traditional US ILI Surveillance Network.

Results

The system monitored ILI activity at an average of 60% of the 269 IHS electronic health databases. EHR-based surveillance detected ILI visits with a sensitivity of 96.4% and a specificity of 97.8% based on chart review (N=2375) of visits at two facilities in September 2009. At the peak of the pandemic (week 41, October 17, 2009), the median time from an ILI visit to data transmission was 6 days, with a mode of 1 day.

Discussion

EHR-based ILI surveillance was accurate, timely, occurred at the majority of IHS facilities nationwide, and provided useful information for decision makers. EHRs thus offer the opportunity to transform public health surveillance.     

Forecasting Emergency Department Crowding: A Prospective, Real-time Evaluation

Objective

Emergency department crowding threatens quality and access to health care, and a method of accurately forecasting near-future crowding should enable novel ways to alleviate the problem. The authors sought to implement and validate the previously developed ForecastED discrete event simulation for real-time forecasting of emergency department crowding.

Design and Measurements

The authors conducted a prospective observational study during a three-month period (5/1/07-8/1/07) in the adult emergency department of a tertiary care medical center. The authors connected the forecasting tool to existing information systems to obtain real-time forecasts of operational data, updated every 10 minutes. The outcome measures included the emergency department waiting count, waiting time, occupancy level, length of stay, boarding count, boarding time, and ambulance diversion; each forecast 2, 4, 6, and 8 hours into the future.

Results

The authors obtained crowding forecasts at 13,239 10-minute intervals, out of 13,248 possible (99.9%). The R² values for predicting operational data 8 hours into the future, with 95% confidence intervals, were 0.27 (0.26, 0.29) for waiting count, 0.11 (0.10, 0.12) for waiting time, 0.57 (0.55, 0.58) for occupancy level, 0.69 (0.68, 0.70) for length of stay, 0.61 (0.59, 0.62) for boarding count, and 0.53 (0.51, 0.54) for boarding time. The area under the receiver operating characteristic curve for predicting ambulance diversion 8 hours into the future, with 95% confidence intervals, was 0.85 (0.84, 0.86).

Conclusions

The ForecastED tool provides accurate forecasts of several input, throughput, and output measures of crowding up to 8 hours into the future. The real-time deployment of the system should be feasible at other emergency departments that have six patient-level variables available through information systems.     

Seeking Health Information Online: Does Wikipedia Matter?

Objective

To determine the significance of the English Wikipedia as a source of online health information.

Design

The authors measured Wikipedia's ranking on general Internet search engines by entering keywords from MedlinePlus, NHS Direct Online, and the National Organization of Rare Diseases as queries into search engine optimization software. We assessed whether article quality influenced this ranking. The authors tested whether traffic to Wikipedia coincided with epidemiological trends and news of emerging health concerns, and how it compares to MedlinePlus.

Measurements

Cumulative incidence and average position of Wikipedia^® compared to other Web sites among the first 20 results on general Internet search engines (Google^®, Google UK^®, Yahoo^®, and MSN^®), and page view statistics for selected Wikipedia articles and MedlinePlus pages.

Results

Wikipedia ranked among the first ten results in 71-85% of search engines and keywords tested. Wikipedia surpassed MedlinePlus and NHS Direct Online (except for queries from the latter on Google UK), and ranked higher with quality articles. Wikipedia ranked highest for rare diseases, although its incidence in several categories decreased. Page views increased parallel to the occurrence of 20 seasonal disorders and news of three emerging health concerns. Wikipedia articles were viewed more often than MedlinePlus Topic (p = 0.001) but for MedlinePlus Encyclopedia pages, the trend was not significant (p = 0.07-0.10).

Conclusions

Based on its search engine ranking and page view statistics, the English Wikipedia is a prominent source of online health information compared to the other online health information providers studied.     

Clinical Decision Support Capabilities of Commercially-available Clinical Information Systems

Background

The most effective decision support systems are integrated with clinical information systems, such as inpatient and outpatient electronic health records (EHRs) and computerized provider order entry (CPOE) systems.

Purpose

The goal of this project was to describe and quantify the results of a study of decision support capabilities in Certification Commission for Health Information Technology (CCHIT) certified electronic health record systems.

Methods

The authors conducted a series of interviews with representatives of nine commercially available clinical information systems, evaluating their capabilities against 42 different clinical decision support features.

Results

Six of the nine reviewed systems offered all the applicable event-driven, action-oriented, real-time clinical decision support triggers required for initiating clinical decision support interventions. Five of the nine systems could access all the patient-specific data items identified as necessary. Six of the nine systems supported all the intervention types identified as necessary to allow clinical information systems to tailor their interventions based on the severity of the clinical situation and the user's workflow. Only one system supported all the offered choices identified as key to allowing physicians to take action directly from within the alert.

Discussion

The principal finding relates to system-by-system variability. The best system in our analysis had only a single missing feature (from 42 total) while the worst had eighteen.This dramatic variability in CDS capability among commercially available systems was unexpected and is a cause for concern.

Conclusions

These findings have implications for four distinct constituencies: purchasers of clinical information systems, developers of clinical decision support, vendors of clinical information systems and certification bodies.     

Prediction of Chronic Obstructive Pulmonary Disease (COPD) in Asthma Patients Using Electronic Medical Records

Objective

Identify clinical factors that modulate the risk of progression to COPD among asthma patients using data extracted from electronic medical records.

Design

Demographic information and comorbidities from adult asthma patients who were observed for at least 5 years with initial observation dates between 1988 and 1998, were extracted from electronic medical records of the Partners Healthcare System using tools of the National Center for Biomedical Computing “Informatics for Integrating Biology to the Bedside” (i2b2).

Measurements

A predictive model of COPD was constructed from a set of 9,349 patients (843 cases, 8,506 controls) using Bayesian networks. The model's predictive accuracy was tested using it to predict COPD in a future independent set of asthma patients (992 patients; 46 cases, 946 controls), who had initial observation dates between 1999 and 2002.

Results

A Bayesian network model composed of age, sex, race, smoking history, and 8 comorbidity variables is able to predict COPD in the independent set of patients with an accuracy of 83.3%, computed as the area under the Receiver Operating Characteristic curve (AUROC).

Conclusions

Our results demonstrate that data extracted from electronic medical records can be used to create predictive models. With improvements in data extraction and inclusion of more variables, such models may prove to be clinically useful.     

An Electronic Health Record Based on Structured Narrative

Objective

To develop an electronic health record that facilitates rapid capture of detailed narrative observations from clinicians, with partial structuring of narrative information for integration and reuse.

Design

We propose a design in which unstructured text and coded data are fused into a single model called structured narrative. Each major clinical event (e.g., encounter or procedure) is represented as a document that is marked up to identify gross structure (sections, fields, paragraphs, lists) as well as fine structure within sentences (concepts, modifiers, relationships). Marked up items are associated with standardized codes that enable linkage to other events, as well as efficient reuse of information, which can speed up data entry by clinicians. Natural language processing is used to identify fine structure, which can reduce the need for form-based entry.

Validation

The model is validated through an example of use by a clinician, with discussion of relevant aspects of the user interface, data structures and processing rules.

Discussion

The proposed model represents all patient information as documents with standardized gross structure (templates). Clinicians enter their data as free text, which is coded by natural language processing in real time making it immediately usable for other computation, such as alerts or critiques. In addition, the narrative data annotates and augments structured data with temporal relations, severity and degree modifiers, causal connections, clinical explanations and rationale.

Conclusion

Structured narrative has potential to facilitate capture of data directly from clinicians by allowing freedom of expression, giving immediate feedback, supporting reuse of clinical information and structuring data for subsequent processing, such as quality assurance and clinical research.     

Handheld vs. Laptop Computers for Electronic Data Collection in Clinical Research: A Crossover Randomized Trial

Objective

To compare users' speed, number of entry errors and satisfaction in using two current devices for electronic data collection in clinical research: handheld and laptop computers.

Design

The authors performed a randomized cross-over trial using 160 different paper-based questionnaires and representing altogether 45,440 variables. Four data coders were instructed to record, according to a random predefined and equally balanced sequence, the content of these questionnaires either on a laptop or on a handheld computer. Instructions on the kind of device to be used were provided to data-coders in individual sealed and opaque envelopes. Study conditions were controlled and the data entry process performed in a quiet environment.

Measurements

The authors compared the duration of the data recording process, the number of errors and users' satisfaction with the two devices. The authors divided errors into two separate categories, typing and missing data errors. The original paper-based questionnaire was used as a gold-standard.

Results

The overall duration of the recording process was significantly reduced (2.0 versus 3.3 min) when data were recorded on the laptop computer (p < 0.001). Data accuracy also improved. There were 5.8 typing errors per 1,000 entries with the laptop compared to 8.4 per 1,000 with the handheld computer (p < 0.001). The difference was even more important for missing data which decreased from 22.8 to 2.9 per 1,000 entries when a laptop was used (p < 0.001). Users found the laptop easier, faster and more satisfying to use than the handheld computer.

Conclusions

Despite the increasing use of handheld computers for electronic data collection in clinical research, these devices should be used with caution. They double the duration of the data entry process and significantly increase the risk of typing errors and missing data. This may become a particularly crucial issue in studies where these devices are provided to patients or healthcare workers, unfamiliar with Computer Technologies, for self-reporting or research data collection processes.     

Physicians' Use of Key Functions in Electronic Health Records from 2005 to 2007: A Statewide Survey

Objective

Electronic health records (EHRs) have potential to improve quality and safety, but many physicians do not use these systems to full capacity. The objective of this study was to determine whether this usage gap is narrowing over time.

Design

Follow-up mail survey of 1,144 physicians in Massachusetts who completed a 2005 survey.

Measurements

Adoption of EHRs and availability and use of 10 EHR functions.

Results

The response rate was 79.4%. In 2007, 35% of practices had EHRs, up from 23% in 2005. Among practices with EHRs, there was little change between 2005 and 2007 in the availability of nine of ten EHR features; the notable exception was electronic prescribing, reported as available in 44.7% of practices with EHRs in 2005 and 70.8% in 2007. Use of EHR functions changed inconsequentially, with more than one out of five physicians not using each available function regularly in both 2005 and 2007. Only electronic prescribing increased substantially: in 2005, 19.9% of physicians with this function available used it most or all the time, compared with 42.6% in 2007 (p < 0.001).

Conclusions

By 2007, more than one third of practices in Massachusetts reported having EHRs; the availability and use of electronic prescribing within these systems has increased. In contrast, physicians reported little change in the availability and use of other EHR functions. System refinements, certification efforts, and health policies, including standards development, should address the gaps in both EHR adoption and the use of key functions.     

Using Empiric Semantic Correlation to Interpret Temporal Assertions in Clinical Texts

Objective

To measure the uncertainty of temporal assertions like “3 weeks ago” in clinical texts.

Design

Temporal assertions extracted from narrative clinical reports were compared to facts extracted from a structured clinical database for the same patients.

Measurements

The authors correlated the assertions and the facts to determine the dependence of the uncertainty of the assertions on the semantic and lexical properties of the assertions.

Results

The observed deviation between the stated duration and actual duration averaged about 20% of the stated deviation. Linear regression revealed that assertions about events further in the past tend to be more uncertain, smaller numeric values tend to be more uncertain (1 mo v. 30 d), and round numbers tend to be more uncertain (10 versus 11 yrs).

Conclusions

The authors empirically derived semantics behind statements of duration using “ago,” and verified intuitions about how numbers are used.     

Possible impact of global warming on the evolution of hemagglutinins from influenza a viruses

Objective To determine if global warming has an impact on the evolution of hemagglutinins from influenza A viruses, because both global warming and influenza pandemics/epidemics threaten the world. Methods 4 706 hemagglutinins from influenza A viruses sampled from 1956 to 2009 were converted to a time‐series to show their evolutionary process and compared with the global, northern hemisphere and southern hemisphere temperatures, to determine if their trends run in similar or opposite directions. Point‐to‐po...     

Formulation of a model for automating infection surveillance: algorithmic detection of central-line associated bloodstream infection

Objective

To formulate a model for translating manual infection control surveillance methods to automated, algorithmic approaches.

Design

We propose a model for creating electronic surveillance algorithms by translating existing manual surveillance practices into automated electronic methods. Our model suggests that three dimensions of expert knowledge be consulted: clinical, surveillance, and informatics. Once collected, knowledge should be applied through a process of conceptualization, synthesis, programming, and testing.

Results

We applied our framework to central vascular catheter associated bloodstream infection surveillance, a major healthcare performance outcome measure. We found that despite major barriers such as differences in availability of structured data, in types of databases used and in semantic representation of clinical terms, bloodstream infection detection algorithms could be deployed at four very diverse medical centers.

Conclusions

We present a framework that translates existing practice—manual infection detection—to an automated process for surveillance. Our experience details barriers and solutions discovered during development of electronic surveillance for central vascular catheter associated bloodstream infections at four hospitals in a variety of data environments. Moving electronic surveillance to the next level—availability at a majority of acute care hospitals nationwide—would be hastened by the incorporation of necessary data elements, vocabularies and standards into commercially available electronic health records.     

Using Computerized Provider Order Entry and Clinical Decision Support to Improve Referring Physicians' Implementation of Consultants' Medical Recommendations

Objectives

Only half of consultants' medical recommendations are implemented. We created a tool that lets referring providers review and implement electronic recommendations made by consultants, with the hypothesis that facilitation with our tool could improve implementation.

Measurements

The tool was piloted among geriatrics consultants and hospitalists. Pre-post evaluation was done with control (before pilot; N = 20) and intervention (after pilot; N = 20) patients. Consultants wrote notes containing recommendations for all study patients, and entered electronic recommendations only for intervention patients. We analyzed all recommendations and surveyed hospitalists.

Results

A total of 249 recommendations were made for intervention patients versus 192 for controls (p < 0.05). Of all recommendations about intervention patients, 78% were implemented, compared to 59% for controls (p = 0.01). Of the intervention recommendations, 77% were entered electronically using our tool; of these, 86% were implemented. All 24 survey respondents indicated that the system improved quality, saved time, and should be expanded.

Conclusion

Consultant recommendations were implemented 30% more often when there was electronic facilitation of recommendations.     

An Empiric Modification to the Probabilistic Record Linkage Algorithm Using Frequency-Based Weight Scaling

Objective

To incorporate value-based weight scaling into the Fellegi-Sunter (F-S) maximum likelihood linkage algorithm and evaluate the performance of the modified algorithm.

Background

Because healthcare data are fragmented across many healthcare systems, record linkage is a key component of fully functional health information exchanges. Probabilistic linkage methods produce more accurate, dynamic, and robust matching results than rule-based approaches, particularly when matching patient records that lack unique identifiers. Theoretically, the relative frequency of specific data elements can enhance the F-S method, including minimizing the false-positive or false-negative matches. However, to our knowledge, no frequency-based weight scaling modification to the F-S method has been implemented and specifically evaluated using real-world clinical data.

Methods

The authors implemented a value-based weight scaling modification using an information theoretical model, and formally evaluated the effectiveness of this modification by linking 51,361 records from Indiana statewide newborn screening data to 80,089 HL7 registration messages from the Indiana Network for Patient Care, an operational health information exchange. In addition to applying the weight scaling modification to all fields, we examined the effect of selectively scaling common or uncommon field-specific values.

Results

The sensitivity, specificity, and positive predictive value for applying weight scaling to all field-specific values were 95.4, 98.8, and 99.9%, respectively. Compared with nonweight scaling, the modified F-S algorithm demonstrated a 10% increase in specificity with a 3% decrease in sensitivity.

Conclusion

By eliminating false-positive matches, the value-based weight modification can enhance the specificity of the F-S method with minimal decrease in sensitivity.     

Disparities in Use of a Personal Health Record in a Managed Care Organization

Objective

Personal health records (PHRs) can increase patient access to health care information. However, use of PHRs may be unequal by race/ethnicity.

Design

The authors conducted a 2-year cohort study (2005-2007) assessing differences in rates of registration with KP.org, a component of the Kaiser Permanente electronic health record (EHR).

Measurements

At baseline, 1,777 25-59 year old Kaiser Permanente Georgia enrollees, who had not registered with KP.org, responded to a mixed mode (written or Internet) survey. Baseline, EHR, and KP.org data were linked. Time to KP.org registration by race from 10/1/05 (with censoring for disenrollment from Kaiser Permanente) was adjusted for baseline education, comorbidity, patient activation, and completion of the baseline survey online vs. by paper using Cox proportional hazards.

Results

Of 1,777, 34.7% (616) registered with KP.org between Oct 2005 and Nov 2007. Median time to registering a KP.org account was 409 days. Among African Americans, 30.1% registered, compared with 41.7% of whites (p < 0.01). In the hazards model, African Americans were again less likely to register than whites (hazard ratio [HR] = 0.652, 95% CI: 0.549-0.776) despite adjustment. Those with baseline Internet access were more likely to register (HR = 1.629, 95% CI: 1.294-2.050), and a significant educational gradient was also observed (more likely registration with higher educational levels).

Conclusions

Differences in education, income, and Internet access did not account for the disparities in PHR registration by race. In the short-term, attempts to improve patient access to health care with PHRs may not ameliorate prevailing disparities between African Americans and whites.     

A Globally Optimal k-Anonymity Method for the De-Identification of Health Data

Background

Explicit patient consent requirements in privacy laws can have a negative impact on health research, leading to selection bias and reduced recruitment. Often legislative requirements to obtain consent are waived if the information collected or disclosed is de-identified.

Objective

The authors developed and empirically evaluated a new globally optimal de-identification algorithm that satisfies the k-anonymity criterion and that is suitable for health datasets.

Design

Authors compared OLA (Optimal Lattice Anonymization) empirically to three existing k-anonymity algorithms, Datafly, Samarati, and Incognito, on six public, hospital, and registry datasets for different values of k and suppression limits.

Measurement

Three information loss metrics were used for the comparison: precision, discernability metric, and non-uniform entropy. Each algorithm's performance speed was also evaluated.

Results

The Datafly and Samarati algorithms had higher information loss than OLA and Incognito; OLA was consistently faster than Incognito in finding the globally optimal de-identification solution.

Conclusions

For the de-identification of health datasets, OLA is an improvement on existing k-anonymity algorithms in terms of information loss and performance.     

Automated Semantic Indexing of Figure Captions to Improve Radiology Image Retrieval

Objective

We explored automated concept-based indexing of unstructured figure captions to improve retrieval of images from radiology journals.

Design

The MetaMap Transfer program (MMTx) was used to map the text of 84,846 figure captions from 9,004 peer-reviewed, English-language articles to concepts in three controlled vocabularies from the UMLS Metathesaurus, version 2006AA. Sampling procedures were used to estimate the standard information-retrieval metrics of precision and recall, and to evaluate the degree to which concept-based retrieval improved image retrieval.

Measurements

Precision was estimated based on a sample of 250 concepts. Recall was estimated based on a sample of 40 concepts. The authors measured the impact of concept-based retrieval to improve upon keyword-based retrieval in a random sample of 10,000 search queries issued by users of a radiology image search engine.

Results

Estimated precision was 0.897 (95% confidence interval, 0.857-0.937). Estimated recall was 0.930 (95% confidence interval, 0.838-1.000). In 5,535 of 10,000 search queries (55%), concept-based retrieval found results not identified by simple keyword matching; in 2,086 searches (21%), more than 75% of the results were found by concept-based search alone.

Conclusion

Concept-based indexing of radiology journal figure captions achieved very high precision and recall, and significantly improved image retrieval.     

Validation Study of an Automated Electronic Acute Lung Injury Screening Tool

Objective

The authors designed an automated electronic system that incorporates data from multiple hospital information systems to screen for acute lung injury (ALI) in mechanically ventilated patients. The authors evaluated the accuracy of this system in diagnosing ALI in a cohort of patients with major trauma, but excluding patients with congestive heart failure (CHF).

Design

Single-center validation study. Arterial blood gas (ABG) data and chest radiograph (CXR) reports for a cohort of intensive care unit (ICU) patients with major trauma but excluding patients with CHF were screened prospectively for ALI requiring intubation by an automated electronic system. The system was compared to a reference standard established through consensus of two blinded physician reviewers who independently screened the same population for ALI using all available ABG data and CXR images. The system's performance was evaluated (1) by measuring the sensitivity and overall accuracy, and (2) by measuring concordance with respect to the date of ALI identification (vs. reference standard).

Measurements

One hundred ninety-nine trauma patients admitted to our level 1 trauma center with an initial injury severity score (ISS) ≥ 16 were evaluated for development of ALI in the first five days in an ICU after trauma.

Main Results

The system demonstrated 87% sensitivity (95% confidence interval [CI] 82.3-91.7) and 89% specificity (95% CI 84.7-93.4). It identified ALI before or within the 24-hour period during which ALI was identified by the two reviewers in 87% of cases.

Conclusions

An automated electronic system that screens intubated ICU trauma patients, excluding patients with CHF, for ALI based on CXR reports and results of ABGs is sufficiently accurate to identify many early cases of ALI.