Can Surveillance Systems Identify and Avert Adverse Drug Events? A Prospective Evaluation of a Commercial Application

Objective

Computerized monitors can effectively detect and potentially prevent adverse drug events (ADEs). Most monitors have been developed in large academic hospitals and are not readily usable in other settings. We assessed the ability of a commercial program to identify and prevent ADEs in a community hospital.

Design and Measurement

We prospectively evaluated the commercial application in a community-based hospital. We examined the frequency and types of alerts produced, how often they were associated with ADEs and potential ADEs, and the potential financial impact of monitoring for ADEs.

Results

Among 2,407 patients screened, the application generated 516 high priority alerts. We were able to review 266 alerts at the time they were generated and among these, 30 (11.3%) were considered substantially important to warrant contacting the physician caring for the patient. These 30 alerts were associated with 4 ADEs and 11 potential ADEs. In all 15 cases, the responsible physician was unaware of the event, leading to a change in clinical care in 14 cases. Overall, 23% of high priority alerts were associated with an ADE (95% confidence interval [CI] 12% to 34%) and another 15% were associated with a potential ADE (95% CI 6% to 24%). Active surveillance used approximately 1.5 hours of pharmacist time daily.

Conclusions

A commercially available, computer-based ADE detection tool was effective at identifying ADEs. When used as part of an active surveillance program, it can have an impact on preventing or ameliorating ADEs.     

Participatory surveillance of diabetes device safety: a social media-based complement to traditional FDA reporting

Background and objective

Malfunctions or poor usability of devices measuring glucose or delivering insulin are reportable to the FDA. Manufacturers submit 99.9% of these reports. We test online social networks as a complementary source to traditional FDA reporting of device-related adverse events.

Methods

Participatory surveillance of members of a non-profit online social network, TuDiabetes.org, from October 2011 to September 2012. Subjects were volunteers from a group within TuDiabetes, actively engaged online in participatory surveillance. They used the free TuAnalyze app, a privacy-preserving method to report detailed clinical information, available through the network. Network members were polled about finger-stick blood glucose monitors, continuous glucose monitors, and insulin delivery devices, including insulin pumps and insulin pens.

Results

Of 549 participants, 75 reported device-related adverse events, nearly half (48.0%) requiring intervention from another person to manage the event. Only three (4.0%) of these were reported by participants to the FDA. All TuAnalyze reports contained outcome information compared with 22% of reports to the FDA. Hypoglycemia and hyperglycemia were experienced by 48.0% and 49.3% of participants, respectively.

Discussion

Members of an online community readily engaged in participatory surveillance. While polling distributed online populations does not yield generalizable, denominator-based rates, this approach can characterize risk within online communities using a bidirectional communication channel that enables reach-back and intervention.

Conclusions

Engagement of distributed communities in social networks is a viable complementary approach to traditional public health surveillance for adverse events related to medical devices.     

Research and applications: Phenotyping for patient safety: algorithm development for electronic health record based automated adverse event and medical error detection in neonatal intensive care

Background

Although electronic health records (EHRs) have the potential to provide a foundation for quality and safety algorithms, few studies have measured their impact on automated adverse event (AE) and medical error (ME) detection within the neonatal intensive care unit (NICU) environment.

Objective

This paper presents two phenotyping AE and ME detection algorithms (ie, IV infiltrations, narcotic medication oversedation and dosing errors) and describes manual annotation of airway management and medication/fluid AEs from NICU EHRs.

Methods

From 753 NICU patient EHRs from 2011, we developed two automatic AE/ME detection algorithms, and manually annotated 11 classes of AEs in 3263 clinical notes. Performance of the automatic AE/ME detection algorithms was compared to trigger tool and voluntary incident reporting results. AEs in clinical notes were double annotated and consensus achieved under neonatologist supervision. Sensitivity, positive predictive value (PPV), and specificity are reported.

Results

Twelve severe IV infiltrates were detected. The algorithm identified one more infiltrate than the trigger tool and eight more than incident reporting. One narcotic oversedation was detected demonstrating 100% agreement with the trigger tool. Additionally, 17 narcotic medication MEs were detected, an increase of 16 cases over voluntary incident reporting.

Conclusions

Automated AE/ME detection algorithms provide higher sensitivity and PPV than currently used trigger tools or voluntary incident-reporting systems, including identification of potential dosing and frequency errors that current methods are unequipped to detect.     

A systematic review to evaluate the accuracy of electronic adverse drug event detection

Objective

Adverse drug events (ADEs), defined as adverse patient outcomes caused by medications, are common and difficult to detect. Electronic detection of ADEs is a promising method to identify ADEs. We performed this systematic review to characterize established electronic detection systems and their accuracy.

Methods

We identified studies evaluating electronic ADE detection from the MEDLINE and EMBASE databases. We included studies if they contained original data and involved detection of electronic triggers using information systems. We abstracted data regarding rule characteristics including type, accuracy, and rationale.

Results

Forty-eight studies met our inclusion criteria. Twenty-four (50%) studies reported rule accuracy but only 9 (18.8%) utilized a proper gold standard (chart review in all patients). Rule accuracy was variable and often poor (range of sensitivity: 40%–94%; specificity: 1.4%–89.8%; positive predictive value: 0.9%–64%). 5 (10.4%) studies derived or used detection rules that were defined by clinical need or the underlying ADE prevalence. Detection rules in 8 (16.7%) studies detected specific types of ADEs.

Conclusion

Several factors led to inaccurate ADE detection algorithms, including immature underlying information systems, non-standard event definitions, and variable methods for detection rule validation. Few ADE detection algorithms considered clinical priorities. To enhance the utility of electronic detection systems, there is a need to systematically address these factors.     

Data use and effectiveness in electronic surveillance of healthcare associated infections in the 21st century: a systematic review

Objective

As more electronic health records have become available during the last decade, we aimed to uncover recent trends in use of electronically available patient data by electronic surveillance systems for healthcare associated infections (HAIs) and identify consequences for system effectiveness.

Methods

A systematic review of published literature evaluating electronic HAI surveillance systems was performed. The PubMed service was used to retrieve publications between January 2001 and December 2011. Studies were included in the review if they accurately described what electronic data were used and if system effectiveness was evaluated using sensitivity, specificity, positive predictive value, or negative predictive value. Trends were identified by analyzing changes in the number and types of electronic data sources used.

Results

26 publications comprising discussions on 27 electronic systems met the eligibility criteria. Trend analysis showed that systems use an increasing number of data sources which are either medico-administrative or clinical and laboratory-based data. Trends on the use of individual types of electronic data confirmed the paramount role of microbiology data in HAI detection, but also showed increased use of biochemistry and pharmacy data, and the limited adoption of clinical data and physician narratives. System effectiveness assessments indicate that the use of heterogeneous data sources results in higher system sensitivity at the expense of specificity.

Conclusions

Driven by the increased availability of electronic patient data, electronic HAI surveillance systems use more data, making systems more sensitive yet less specific, but also allow systems to be tailored to the needs of healthcare institutes’ surveillance programs.     

Guided medication dosing for elderly emergency patients using real-time, computerized decision support

Objective

To evaluate the impact of a real-time computerized decision support tool in the emergency department that guides medication dosing for the elderly on physician ordering behavior and on adverse drug events (ADEs).

Design

A prospective controlled trial was conducted over 26 weeks. The status of the decision support tool alternated OFF (7/17/06–8/29/06), ON (8/29/06–10/10/06), OFF (10/10/06–11/28/06), and ON (11/28/06–1/16/07) in consecutive blocks during the study period. In patients ≥65 who were ordered certain benzodiazepines, opiates, non-steroidals, or sedative-hypnotics, the computer application either adjusted the dosing or suggested a different medication. Physicians could accept or reject recommendations.

Measurements

The primary outcome compared medication ordering consistent with recommendations during ON versus OFF periods. Secondary outcomes included the admission rate, emergency department length of stay for discharged patients, 10-fold dosing orders, use of a second drug to reverse the original medication, and rate of ADEs using previously validated explicit chart review.

Results

2398 orders were placed for 1407 patients over 1548 visits. The majority (49/53; 92.5%) of recommendations for alternate medications were declined. More orders were consistent with dosing recommendations during ON (403/1283; 31.4%) than OFF (256/1115; 23%) periods (p≤0.0001). 673 (43%) visits were reviewed for ADEs. The rate of ADEs was lower during ON (8/237; 3.4%) compared with OFF (31/436; 7.1%) periods (p=0.02). The remaining secondary outcomes showed no difference.

Limitations

Single institution study, retrospective chart review for ADEs.

Conclusion

Though overall agreement with recommendations was low, real-time computerized decision support resulted in greater acceptance of medication recommendations. Fewer ADEs were observed when computerized decision support was active.     

A Systematic Review of the Performance Characteristics of Clinical Event Monitor Signals Used to Detect Adverse Drug Events in the Hospital Setting

Objective

We conducted a systematic review of pharmacy and laboratory signals used by clinical event monitor systems to detect adverse drug events (ADEs) in adult hospitals.

Design and Measurements

We searched the MEDLINE, CINHAL, and EMBASE databases for the years 1985–2006, and found 12 studies describing 36 unique ADE signals (10 medication levels, 19 laboratory values, and 7 antidotes). We were able to calculate positive predictive values (PPVs) and 95% confidence intervals (CIs) for 15 signals.

Results

We found that PPVs ranged from 0.03 (95% CI, 0.03–0.03) for hypokalemia, to 0.50 (95% CI, 0.39–0.61) for supratherapeutic quinidine level. In general, antidotes (range = 0.09–0.11) had the lowest PPVs, followed by laboratory values (range = 0.03–0.27) and medication levels (range = 0.03–0.50).

Conclusion

Data from this study should help clinical information system and computerized decision support producers develop or improve existing clinical event monitor systems to detect ADEs in their own hospitals by prioritizing those signals with the highest PPVs.     

A novel signal detection algorithm for identifying hidden drug-drug interactions in adverse event reports

Objective

Adverse drug events (ADEs) are common and account for 770 000 injuries and deaths each year and drug interactions account for as much as 30% of these ADEs. Spontaneous reporting systems routinely collect ADEs from patients on complex combinations of medications and provide an opportunity to discover unexpected drug interactions. Unfortunately, current algorithms for such “signal detection” are limited by underreporting of interactions that are not expected. We present a novel method to identify latent drug interaction signals in the case of underreporting.

Materials and Methods

We identified eight clinically significant adverse events. We used the FDA''s Adverse Event Reporting System to build profiles for these adverse events based on the side effects of drugs known to produce them. We then looked for pairs of drugs that match these single-drug profiles in order to predict potential interactions. We evaluated these interactions in two independent data sets and also through a retrospective analysis of the Stanford Hospital electronic medical records.

Results

We identified 171 novel drug interactions (for eight adverse event categories) that are significantly enriched for known drug interactions (p=0.0009) and used the electronic medical record for independently testing drug interaction hypotheses using multivariate statistical models with covariates.

Conclusion

Our method provides an option for detecting hidden interactions in spontaneous reporting systems by using side effect profiles to infer the presence of unreported adverse events.     

Formulation of a model for automating infection surveillance: algorithmic detection of central-line associated bloodstream infection

Objective

To formulate a model for translating manual infection control surveillance methods to automated, algorithmic approaches.

Design

We propose a model for creating electronic surveillance algorithms by translating existing manual surveillance practices into automated electronic methods. Our model suggests that three dimensions of expert knowledge be consulted: clinical, surveillance, and informatics. Once collected, knowledge should be applied through a process of conceptualization, synthesis, programming, and testing.

Results

We applied our framework to central vascular catheter associated bloodstream infection surveillance, a major healthcare performance outcome measure. We found that despite major barriers such as differences in availability of structured data, in types of databases used and in semantic representation of clinical terms, bloodstream infection detection algorithms could be deployed at four very diverse medical centers.

Conclusions

We present a framework that translates existing practice—manual infection detection—to an automated process for surveillance. Our experience details barriers and solutions discovered during development of electronic surveillance for central vascular catheter associated bloodstream infections at four hospitals in a variety of data environments. Moving electronic surveillance to the next level—availability at a majority of acute care hospitals nationwide—would be hastened by the incorporation of necessary data elements, vocabularies and standards into commercially available electronic health records.     

Intensified surveillance after surgery for colorectal cancer significantly improves survival

Background

Postoperative surveillance after curative resection for colorectal cancer has been demostrated to improve survival. It remains unknown however, whether intensified surveillance provides a significant benefit regarding outcome and survival. This study was aimed at comparing different surveillance strategies regarding their effect on long-term outcome.

Methods

Between 1990 and 2006, all curative resections for colorectal cancer were selected from our prospective colorectal cancer database. All patients were offered to follow our institution''s surveillance programm according to the ASCO guidelines. We defined surveillance as "intensive" in cases where > 70% appointments were attended and the program was completed. As "minimal" we defined surveillance with < 70% of the appointments attended and an incomplete program. As "none" we defined the group which did not take part in any surveillance.

Results

Out of 1469 patients 858 patients underwent "intensive", 297 "minimal" and 314 "none" surveillance. The three groups were well balanced regarding biographical data and tumor characteristics. The 5-year survival rates were 79% (intensive), 76% (minimal) and 54% (none) (OR 1.480, (95% CI 1.135-1.929); p < 0.0001), respectively. The 10-year survival rates were 65% (intensive), 50% (minimal) and 31% (none) (p < 0.0001), respectively. With a median follow-up of 70 months the median time of survival was 191 months (intensive), 116 months (minimal) and 66 months (none) (p < 0.0001). After recurrence, the 5-year survival rates were 32% (intensive, p = 0.034), 13% (minimal, p = 0.001) and 19% (none, p = 0.614). The median time of survival after recurrence was 31 months (intensive, p < 0.0001), 21 months (minimal, p < 0.0001) and 16 month (none, p < 0.0001) respectively.

Conclusion

Intensive surveillance after curative resection of colorectal cancer improves survival. In cases of recurrent disease, intensive surveillance has a positive impact on patients'' prognosis. Large randomized, multicenter trials are needed to substantiate these results.     

Combining rules and machine learning for extraction of temporal expressions and events from clinical narratives

Objective

Identification of clinical events (eg, problems, tests, treatments) and associated temporal expressions (eg, dates and times) are key tasks in extracting and managing data from electronic health records. As part of the i2b2 2012 Natural Language Processing for Clinical Data challenge, we developed and evaluated a system to automatically extract temporal expressions and events from clinical narratives. The extracted temporal expressions were additionally normalized by assigning type, value, and modifier.

Materials and methods

The system combines rule-based and machine learning approaches that rely on morphological, lexical, syntactic, semantic, and domain-specific features. Rule-based components were designed to handle the recognition and normalization of temporal expressions, while conditional random fields models were trained for event and temporal recognition.

Results

The system achieved micro F scores of 90% for the extraction of temporal expressions and 87% for clinical event extraction. The normalization component for temporal expressions achieved accuracies of 84.73% (expression''s type), 70.44% (value), and 82.75% (modifier).

Discussion

Compared to the initial agreement between human annotators (87–89%), the system provided comparable performance for both event and temporal expression mining. While (lenient) identification of such mentions is achievable, finding the exact boundaries proved challenging.

Conclusions

The system provides a state-of-the-art method that can be used to support automated identification of mentions of clinical events and temporal expressions in narratives either to support the manual review process or as a part of a large-scale processing of electronic health databases.     

Substrate reduction therapy with miglustat for type 1 Gaucher disease: A retrospective analysis from a single institution

Introduction.

Gaucher disease (GD) is an infrequent progressive multisystem lysosomal storage disorder caused by the deficient activity of the lysosomal enzyme, glucocerebrosidase. A retrospective, single-center analysis of the clinical experience concerning the use of miglustat (N-butyldeoxynojirimycin), an oral inhibitor of glucosylceramide synthase, in type 1 Gaucher disease (GD1) was conducted to evaluate the efficacy, adverse events (AE), and outcome of miglustat therapy.

Patients and methods.

Six adult Caucasian patients with GD1 (two women and four men), aged 21–81 years (median age 59 years), were treated with miglustat between October 2005 and April 2011. All but one patient (83%) carried at least one allele with c.1226A>G (N370S) mutation in the GBA1 gene.

Results.

Weight loss, diarrhea, poor appetite, and tremor were frequently reported AE by the patients. All of them experienced at least 2 AE, and three patients (50%) experienced at least 4 AE. Only two out of six patients (33%) have used miglustat longer than 12 months, of which only one used it longer than 15 months.

Conclusions.

The major obstacle to successful miglustat therapy in GD1 was the high proportion of patients discontinuing their treatment due to the AE and the worsened quality of life. Further efforts are needed to improve tolerability of miglustat and, in consequence, compliance of patients treated with this orphan drug.     

Banding pattern indicative of echinococcosis in a commercial cysticercosis western blot

Objective

A commercial cysticercosis Western blot was evaluated for serological cross-reactivity of sera from patients with alveolar (AE) and cystic echinococcosis (CE).

Methods

A total of 161 sera were examined, including 31 sera from AE-patients, 11 sera from CE-patients, 9 sera from patients with other parasitic diseases and 109 sera from patients with unrelated medical conditions. All AE-and CE-sera were also examined by the echinococcosis Western blot.

Results

More sera from patients with AE than with CE showed cross-reactivity in the form of ladder-like patterns ("Mikado aspect") and untypical bands at 6-8 kDa (71% and 77.4% versus 27.3% and 45.5%, respectively). In contrast, triplets of bands in the area above 50 kDa and between 24 and 39-42 kDa were more frequent in CE than in AE sera. The fuzzy band at 50-55 kDa typical for cysticercosis was absent in all AE and CE sera.

Conclusions

Atypical banding patterns in the cysticercosis Western blot should raise the suspicion of a metacestode infection different from Taenia solium, i.e. Echinococcus multilocularis or E. granulosus, especially when the Mikado aspect and an altered 6-8 kDa band is visible in the absence of a fuzzy 50-55 kDa band.     

The Evaluation of a Temporal Reasoning System in Processing Clinical Discharge Summaries

Context

TimeText is a temporal reasoning system designed to represent, extract, and reason about temporal information in clinical text.

Objective

To measure the accuracy of the TimeText for processing clinical discharge summaries.

Design

Six physicians with biomedical informatics training served as domain experts. Twenty discharge summaries were randomly selected for the evaluation. For each of the first 14 reports, 5 to 8 clinically important medical events were chosen. The temporal reasoning system generated temporal relations about the endpoints (start or finish) of pairs of medical events. Two experts (subjects) manually generated temporal relations for these medical events. The system and expert-generated results were assessed by four other experts (raters). All of the twenty discharge summaries were used to assess the system’s accuracy in answering time-oriented clinical questions. For each report, five to ten clinically plausible temporal questions about events were generated. Two experts generated answers to the questions to serve as the gold standard. We wrote queries to retrieve answers from system’s output.

Measurements

Correctness of generated temporal relations, recall of clinically important relations, and accuracy in answering temporal questions.

Results

The raters determined that 97% of subjects’ 295 generated temporal relations were correct and that 96.5% of the system’s 995 generated temporal relations were correct. The system captured 79% of 307 temporal relations determined to be clinically important by the subjects and raters. The system answered 84% of the temporal questions correctly.

Conclusion

The system encoded the majority of information identified by experts, and was able to answer simple temporal questions.     

Comparison of a basic and an advanced pharmacotherapy-related clinical decision support system in a hospital care setting in the Netherlands

Objective

To compare the clinical relevance of medication alerts in a basic and in an advanced clinical decision support system (CDSS).

Design

A prospective observational study.

Materials and methods

We collected 4023 medication orders in a hospital for independent evaluation in two pharmacotherapy-related decision support systems. Only the more advanced system considered patient characteristics and laboratory test results in its algorithms. Two pharmacists assessed the clinical relevance of the medication alerts produced. The alert was considered relevant if the pharmacist would undertake action (eg, contact the physician or the nurse). The primary analysis concerned the positive predictive value (PPV) for clinically relevant medication alerts in both systems.

Results

The PPV was significantly higher in the advanced system (5.8% vs 17.0%; p<0.05). Significant differences were found in the alert categories: drug–(drug) interaction (9.9% vs 14.8%; p<0.05), drug–age interaction (2.9% vs 73.3%; p<0.05), and dosing guidance (5.6% vs 16.9%; p<0.05). Including laboratory values and other patient characteristics resulted in a significantly higher PPV for the advanced CDSS compared to the basic medication alerts (12.2% vs 23.3%; p<0.05).

Conclusion

The advanced CDSS produced a higher proportion of clinically relevant medication alerts, but the number of irrelevant alerts remained high. To improve the PPV of the advanced CDSS, the algorithms should be optimized by identifying additional risk modifiers and more data should be made electronically available to improve the performance of the algorithms. Our study illustrates and corroborates the need for cyclic testing of technical improvements in information technology in circumstances representative of daily clinical practice.     

Adoption of Order Entry with Decision Support for Chronic Care by Physician Organizations

Objective

This study sought to explore physician organizations’ adoption of chronic care guidelines in order entry systems and to investigate the organizational and market-related factors associated with this adoption.

Design

A quantitative nationwide survey of all primary care medical groups in the United States with 20 or more physicians; data were collected on 1,104 physician organizations, representing a 70% response rate.

Measurements

Measurements were the presence of an asthma, diabetes, or congestive heart failure guideline in a physician organization’s order entry system; size; age of the organization; number of clinic locations; type of ownership; health maintenance organization market penetration; urban/rural location; and presence of external incentives to improve quality of care.

Results

Only 27% of organizations reported access to order entry with decision support for chronic disease care. External incentives for quality is the only factor significantly associated with adoption of these tools. Organizations experiencing greater external incentives for quality are more likely to adopt order entry with decision support.

Conclusion

Because external incentives are strong drivers of adoption, policies requiring reporting of chronic care measurements and rewarding improvement as well as financial incentives for use of specific information technology tools are likely to accelerate adoption of order entry with decision support.     

The Utility of Adding Retrospective Medication Profiling to Computerized Provider Order Entry in an Ambulatory Care Population

Background

We assessed whether medication safety improved when a medication profiling program was added to a computerized provider order entry system.

Design

Between June 2001 and January 2002 we profiled outpatients with potential prescribing errors using computerized retrospective drug utilization software. We focused primarily on drug interactions. Patients were randomly assigned either to Provider Feedback or to Usual Care. Subsequent adverse drug event (ADE) incidence and other outcomes, including ADE preventability and severity, occurring up to 1 year following the last profiling date were evaluated retrospectively by a pharmacist blinded to patient assignment.

Measurements

Data were abstracted using a study-designed instrument. An ADE was defined by an Adverse Drug Reaction Probability scale score of 1 or more. Statistical analyses included negative binomial regression for comparing ADE incidence.

Results

Of 913 patients in the analytic sample, 371 patients (41%) had one or more ADEs. Incidence, by individual, was not significantly different between Usual Care and Provider Feedback groups (37% vs. 45%; p = 0.06; Coefficient, 0.19; 95% CI: −0.008, 0.390). ADE severity was also similar. For example, 51% of ADEs in the Usual Care and 58% in the Provider Feedback groups involved symptoms that were not serious (95% CI for the difference, −15%, 2%). Finally, ADE preventability did not differ. For example, 16% in the Usual Care group and 17% in the Provider Feedback group had an associated warning (95% CI for the difference, −7 to 5%; p = 0.79).

Conclusion

Medications safety did not improve with the addition of a medication profiling program to an electronic prescribing system.