Comparing the accuracy of syndrome surveillance systems in detecting influenza-like illness: GUARDIAN vs. RODS vs. electronic medical record reports

BackgroundA highly sensitive real-time syndrome surveillance system is critical to detect, monitor, and control infectious disease outbreaks, such as influenza. Direct comparisons of diagnostic accuracy of various surveillance systems are scarce.ObjectiveTo statistically compare sensitivity and specificity of multiple proprietary and open source syndrome surveillance systems to detect influenza-like illness (ILI).MethodsA retrospective, cross-sectional study was conducted utilizing data from 1122 patients seen during November 1–7, 2009 in the emergency department of a single urban academic medical center. The study compared the Geographic Utilization of Artificial Intelligence in Real-time for Disease Identification and Alert Notification (GUARDIAN) system to the Complaint Coder (CoCo) of the Real-time Outbreak Detection System (RODS), the Symptom Coder (SyCo) of RODS, and to a standardized report generated via a proprietary electronic medical record (EMR) system. Sensitivity, specificity, and accuracy of each classifier's ability to identify ILI cases were calculated and compared to a manual review by a board-certified emergency physician. Chi-square and McNemar's tests were used to evaluate the statistical difference between the various surveillance systems.ResultsThe performance of GUARDIAN in detecting ILI in terms of sensitivity, specificity, and accuracy, as compared to a physician chart review, was 95.5%, 97.6%, and 97.1%, respectively. The EMR-generated reports were the next best system at identifying disease activity with a sensitivity, specificity, and accuracy of 36.7%, 99.3%, and 83.2%, respectively. RODS (CoCo and SyCo) had similar sensitivity (35.3%) but slightly different specificity (CoCo = 98.9%; SyCo = 99.3%). The GUARDIAN surveillance system with its multiple data sources performed significantly better compared to CoCo (χ² = 130.6, p < 0.05), SyCo (χ² = 125.2, p < 0.05), and EMR-based reports (χ² = 121.3, p < 0.05). In addition, similar significant improvements in the accuracy (>12%) and sensitivity (>47%) were observed for GUARDIAN with only chief complaint data as compared to RODS (CoCo and SyCo) and EMR-based reports.ConclusionIn our study population, the GUARDIAN surveillance system, with its ability to utilize multiple data sources from patient encounters and real-time automaticity, demonstrated a more robust performance when compared to standard EMR-based reports and the RODS systems in detecting ILI. More large-scale studies are needed to validate the study findings, and to compare the performance of GUARDIAN in detecting other infectious diseases.     

Modeling Baseline Shifts in Multivariate Disease Outbreak Detection

Objective

Outbreak detection algorithms monitoring only disease-relevant data streams may be prone to false alarms due to baseline shifts. In this paper, we propose a Multinomial-Generalized-Dirichlet (MGD) model to adjust for baseline shifts.

Introduction

Population surges or large events may cause shift of data collected by biosurveillance systems [1]. For example, the Cherry Blossom Festival brings hundreds of thousands of people to DC every year, which results in simultaneous elevations in multiple data streams (Fig. 1). In this paper, we propose an MGD model to accommodate the needs of dealing with baseline shifts.Open in a separate windowFig. 1Eight data streams of NRDM categories collected by RODS system (Anti-Diarrhea, Anti-Fever Adult, Chest Rubs, Cough/Cold, Baby/Child Electrolytes, Nasal Products, Rash and Thermometers) between Apr. 3, 2011 and Apr. 8, 2011 in Washington DC.

Methods

Existing multivariate algorithms only model disease-relevant data streams (e.g., anti-fever medication sales or patient visits with constitutional syndrome for detection of flu outbreak). On the contrary, we also incorporate a non-disease-relevant data stream as a control factor.We assume that the counts from all data streams follow a Multinomial distribution. Given this distribution, the expected value of the distribution parameter is not subject to change during a baseline shift; however, it has to change in order to model an outbreak. Therefore, this distribution inherently adjusts for the baseline shifts. In addition, we use the generalized Dirichlet (GD) distribution to model the parameter, since GD distribution is one of the conjugate prior of Multinomial [2]. We call this model the Multinomial-Generalized-Dirichlet (MGD) model.

Results

We applied MGD model in our previous proposed Rank-Based Spatial Clustering (MRSC) algorithm [3]. We simulated both outbreak cases and baseline shift phenomena. The experiment includes two groups of data sets. The first includes the data sets only injected with outbreak cases, and the second includes the ones with both outbreak cases and baseline shifts. We apply MRSC algorithm and a reference method, the Multivariate Bayesian Scan Statistic (MBSS) algorithm (which only analyzes the disease-relevant data streams) [4], to both data sets. Fig. 2 shows the performance of outbreak detection: the ROC curves and AMOC curves of analyzing the data sets with baseline shifts (solid lines) and without (dashed lines). We can see from Fig. 2 that the performance of MBSS dropped much more significantly than MRSC when analyzing the data sets with baseline shifts.Open in a separate windowFig. 2ROC and AMOC curves of MRSC (red) and MBSS (blue). The solid lines represent the two algorithms applied on the data sets injected with both outbreak cases and baseline shift phenomena. The dashed lines represent the two algorithms applied on the data sets injected with outbreak cases only.

Conclusions

The MGD model can be a good supplement model used to detect disease outbreaks in order to achieve both better sensitivity and better specificity especially when baseline shifts are present in the data.     

2007年北京市居民母乳中二噁英类化合物负荷水平调查

目的 调查北京地区居民母乳中多氯代苯并二噁英和多氯代苯并呋喃(PCDD/Fs)和二噁英样多氯联苯(dl-PCBs)的污染水平,评价一般人群PCDD/Fs和dl-PCBs的机体负荷状况.方法 2007年在北京市11个区、县中采集母乳样品110份,制成11个混合样品后,采用同位素稀释技术以高分辨气相色谱-高分辨质谱(HRGC-HRMS)测定母乳样品中的PCDD/Fs和dl-PCBs.结果 北京市母乳样品中二噁英类化合物含量最高的组分为八氯代二苯并二噁英(OCDD)和多氯联苯(PCB)-118、PCB-105,其含量中位数分别为20.6 pg/g脂肪、4.07和1.63 ng/g脂肪.按毒性当量(TEQ)计,北京市11个母乳混合样品中总二噁英类化合物含量中位数为7.4 pg TEQ/g脂肪;最高的为通州区,含量中位数为13.5 pg TEQ/g脂肪;最低的为平谷区,为4.3 pg TEQ/g脂肪.结论 目前北京市母乳中PCDD/Fs和dl-PCBs处于较低水平,但随着我国的快速工业化,人群此类物质的机体负荷水平有可能会上升,因此有必要持续跟踪监测.     

Surge Capacity for Health Care Systems: Early Detection, Methodologies, and Process

Excessive demand on hospital services from large‐scale emergencies is something that every emergency department health care provider and hospital administrator knows could happen at any time. Nowhere in this country have we recently faced a disaster of the magnitude of concern we now face involving agents of mass destruction or social disruption, especially those in the area of infectious diseases and radiological materials. The war on terrorism is not a conventional war, and terrorists may use any means of convenience to carry out their objectives in an unpredictable time line. Have we adequately prepared for the potentially excessive surge in demand for medical services that a large‐scale event could bring to our medical care system? Are our emergency departments ready for such events? Surveillance systems, such as BioWatch, BioSense, the National Biosurveillance Integration System, and the countermeasure program BioShield, offer hope that we will be able to meet these new challenges.