Practical comparison of aberration detection algorithms for biosurveillance systems

National syndromic surveillance systems require optimal anomaly detection methods. For method performance comparison, we injected multi-day signals stochastically drawn from lognormal distributions into time series of aggregated daily visit counts from the U.S. Centers for Disease Control and Prevention’s BioSense syndromic surveillance system. The time series corresponded to three different syndrome groups: rash, upper respiratory infection, and gastrointestinal illness. We included a sample of facilities with data reported every day and with median daily syndromic counts ⩾1 over the entire study period. We compared anomaly detection methods of five control chart adaptations, a linear regression model and a Poisson regression model. We assessed sensitivity and timeliness of these methods for detection of multi-day signals. At a daily background alert rate of 1% and 2%, the sensitivities and timeliness ranged from 24 to 77% and 3.3 to 6.1 days, respectively. The overall sensitivity and timeliness increased substantially after stratification by weekday versus weekend and holiday. Adjusting the baseline syndromic count by the total number of facility visits gave consistently improved sensitivity and timeliness without stratification, but it provided better performance when combined with stratification. The daily syndrome/total-visit proportion method did not improve the performance. In general, alerting based on linear regression outperformed control chart based methods. A Poisson regression model obtained the best sensitivity in the series with high-count data.     

SMART Platforms: Building the App Store for Biosurveillance

Objective

To enable public health departments to develop “apps” to run on electronic health records (EHRs) for (1) biosurveillance and case reporting and (2) delivering alerts to the point of care. We describe a novel health information technology platform with substitutable apps constructed around core services enabling EHRs to function as iPhone-like platforms.

Introduction

Health care information is a fundamental source of data for biosurveillance, yet configuring EHRs to report relevant data to health departments is technically challenging, labor intensive, and often requires custom solutions for each installation. Public health agencies wishing to deliver alerts to clinicians also must engage in an endless array of one-off systems integrations.Despite a $48B investment in HIT, and meaningful use criteria requiring reporting to biosurveillance systems, most vendor electronic health records are architected monolithically, making modification difficult for hospitals and physician practices. An alternative approach is to reimagine EHRs as iPhone-like platforms supporting substitutable apps-based functionality. Substitutability is the capability inherent in a system of replacing one application with another of similar functionality.

Methods

Substitutability requires that the purchaser of an app can replace one application with another without being technically expert, without requiring re-engineering other applications that they are using, and without having to consult or require assistance of any of the vendors of previously installed or currently installed applications. Apps necessarily compete with each other promoting progress and adaptability.The Substitutable Medical Applications, Reusable Technologies (SMART) Platforms project is funded by a $15M grant from Office of the National Coordinator of Health Information Technology’s Strategic Health IT Advanced Research Projects (SHARP) Program. All SMART standards are open and the core software is open source.The SMART project promotes substitutability through an application programming interface (API) that can be adopted as part of a “container” built around by a wide variety of HIT, providing readonly access to the underlying data model and a software development toolkit to readily create apps. SMART containers are HIT systems, that have implemented the SMART API or a portion of it. Containers marshal data sources and present them consistently across the SMART API. SMART applications consume the API and are substitutable.

Results

SMART provides a common platform supporting an “app store for biosurveillance” as an approach to enabling one stop shopping for public health departments—to create an app once, and distribute it everywhere.Further, such apps can be readily updated or created—for example, in the case of an emerging infection, an app may be designed to collect additional data at emergency department triage. Or a public health department may widely distribute an app, interoperable with any SMART-enabled EMR, that delivers contextualized alerts when patient electronic records are opened, or through background processes.SMART has sparked an ecosystem of apps developers and attracted existing health information technology platforms to adopt the SMART API—including, traditional, open source, and next generation EHRs, patient-facing platforms and health information exchanges. SMART-enabled platforms to date include the Cerner EMR, the WorldVista EHR, the OpenMRS EHR, the i2b2 analytic platform, and the Indivo X personal health record. The SMART team is working with the Mirth Corporation, to SMART-enable the HealthBridge and Redwood MedNet Health Information Exchanges. We have demonstrated that a single SMART app can run, unmodified, in all of these environments, as long as the underlying platform collects the required data types. Major EHR vendors are currently adapting the SMART API for their products.

Conclusions

The SMART system enables nimble customization of any electronic health record system to create either a reporting function (outgoing communication) or an alerting function (incoming communication) establishing a technology for a robust linkage between public health and clinical environments.     

Statistical Models for Biosurveillance of Multiple Organisms

Objective

To look at the diversity of the patterns displayed by a range of organisms, and to seek a simple family of models that adequately describes all organisms, rather than a well-fitting model for any particular organism.

Introduction

There has been much research on statistical methods of prospective outbreak detection that are aimed at identifying unusual clusters of one syndrome or disease, and some work on multivariate surveillance methods (1). In England and Wales, automated laboratory surveillance of infectious diseases has been undertaken since the early 1990’s. The statistical methodology of this automated system is described in (2). However, there has been little research on outbreak detection methods that are suited to large, multiple surveillance systems involving thousands of different organisms.

Methods

We obtained twenty years’ data on weekly counts of all infectious disease organisms reported to the UK’s Health Protection Agency. We summarized the mean frequencies, trends and seasonality of each organism using log-linear models. To identify a simple family of models which adequately represents all organisms, the Poisson model, the quasi-Poisson model and the negative binomial model were investigated (3,4). Formal goodness-of-fit tests were not used as they can be unreliable with sparse data. Adequacy of the models was empirically studied using the relationships between the mean, variance and skewness. For this purpose, each data series was first subdivided into 41 half-years and de-seasonalized.

Results

Trends and seasonality were summarized by plotting the distribution of estimated linear trend parameters for 2250 organisms, and modal seasonal period for 2254 organisms, including those organisms for which the seasonal effect is statistically significant.Relationships between mean and variance were summarized as given in Figure 1.Open in a separate windowFigure 1Relationships between mean and variance. (top) Histogram of the slopes of the best fit lines for 1001 organisms; the value 1 corresponds to the quasi-Poisson model; (bottom) log of variance plotted against log of mean for one organism. The full line is the best fit to the points; the dashed line corresponds to the quasi-Poisson model; the dotted line corresponds to the Poisson model.Similar plots were used to summarize the relationships between mean and skewness.

Conclusions

Statistical outbreak detection models must be able to cope with seasonality and trends. The data analyses suggest that the great majority of organisms can adequately – though far from perfectly – be represented by a statistical model in which the variance is proportional to the mean, such as the quasi-Poisson or negative binomial models.     

National Collaborative for Bio-Preparedness

Objective

Demonstrate the functionality of the National Collaborative for Bio-Preparedness system.

Introduction

The National Collaborative for Bio-Preparedness (NCB-Prepared) was established in 2010 to create a biosurveillance resource to enhance situational awareness and emergency preparedness. This joint-institutional effort has drawn on expertise from the University of North Carolina- Chapel Hill, North Carolina State University, and SAS Institute, leveraging North Carolina’s role as a leader in syndromic surveillance, technology development and health data standards. As an unprecedented public/private alliance, they bring the flexibility of the private sector to support the public sector. The project has developed a functioning prototype system for multiple states that will be scaled and made more robust for national adoption.

Methods

NCB-Prepared recognizes that the capability of any biosurveillance system is a function of the data is analyzes. NCB-Prepared is designed to provide information services that analyze and integrate national data across a variety of domains, such as human clinical, veterinary and physical data. In addition to this one-health approach to surveillance, a primary objective of NCB-Prepared is to gather data that is closer in time to the event of interest. NCB-Prepared has validated the usefulness of North Carolina emergency medical services data for the purposes of biosurveillance of both acute outbreaks and seasonal epidemics (1).A unique model of user-driven valuing of data-providing value back to the provider in their terms-motivates collaboration from potential data providers, along with timely and complete data. NCB-Prepared approaches potential data providers, partners and users with the proposition that enhanced data quality and analysis is valuable to them individually and that an integrated information system can be valuable to all. With the onboarding of new data sources, NCB-Prepared implements a formal process of data discovery and integration. The goal of this process is three-fold: 1) to develop recommendations to enhance data quality going forward, 2) to integrate information across data sources, and 3) to develop novel analytic techniques for detecting health threats. NCB-Prepared is committed to both utilizing standard methods for event detection and to developing innovative analytics for biosurveillance such as the Text Analytics and Proportional charts method (TAP). The sophisticated analytic functionality of the system, including improved time to detection, query reporting, alert detection, forecasting and predictive modeling, can be attributed to collaboration between analysts from private industry, academia and public health.NCB-Prepared followed the formal software development process known as agile development to create the user interface of the system. This method is based on iterative cycles wherein requirements evolve from regular sessions between user groups and developers. The result of agile development and collaborative relationships is a system which visualizes signals and diverse data sources across time and place while providing information services across all levels of users and stakeholders.

Conclusions

Lessons Learned:

1. Understand the functionality of new biosurveillance system, NCB-Prepared
2. Identify the benefit of creating collaborative relationships with data providers and users
3. Appreciate the value of a public/private partnership for biosurveillance and bio-preparedness