Evaluating Biosurveillance System Components using Multi-Criteria Decision Analysis

Objective

The use of Multi-Criteria Decision Analysis (MCDA) has traditionally been limited to the field of operations research, however many of the tools and methods developed for MCDA can also be applied to biosurveillance. Our project demonstrates the utility of MCDA for this purpose by applying it to the evaluation of data streams for use in an integrated, global biosurveillance system.

Introduction

The evaluation of biosurveillance system components is a complex, multi-objective decision that requires consideration of a variety of factors. Multi-Criteria Decision Analysis provides a methodology to assist in the objective analysis of these types of evaluation by creating a mathematical model that can simulate decisions. This model can utilize many types of data, both quantitative and qualitative, that can accurately describe components. The decision-maker can use this model to determine which of the system components best accomplish the goals being evaluated. Before MCDA can be utilized effectively, an evaluation framework needs to be developed. We built a robust framework that identified unique metrics, surveillance goals, and priorities for metrics. Using this framework, we were able to use MCDA to assist in the evaluation of data streams and to determine which types would be of most use within a global biosurveillance system.

Methods

MCDA was implemented using the Logical Decisions® software. The construction of the evaluation framework was carried out in several steps: identification and definition of data streams, metrics and surveillance goals, and the determination of the relative importance of each metric to the respective surveillance goal being evaluated. Sixteen data streams types were defined and identified for evaluation from a survey we conducted that collected over 200 surveillance products. A subject matter expert (SME) panel was assembled to help identify the biosurveillance goals and metrics in which to evaluate the data streams. To assign values for the metrics, we referenced properties of data streams used in currently operational systems.

Results

Our survey identified sixteen different classes of data streams: Ambulance Records, Clinic/Health Care Provider Records, ED/Hospital Records, Employment/School Records, Established Databases, Financial Records, Help Lines, Internet Search Queries, Laboraotry Records, News Aggregators, Official Reports, Police/Fire Department Records, Personal Communication, Prediction Markets, Sales, and Social Media.Four biosurveillance goals were identified: Early Warning of Health Threats, Early Detection of Health Events, Situational Awareness, and Consequence Management.Eleven metrics were identified: Accessibility, Cost, Credibility, Flexibility, Integrability, Geographic/Population Coverage, Granularity, Specificity of Detection, Sustainability, Time to Indication, and Timeliness.Using the framework, it was possible to use MCDA to rank the utility of each data stream for each goal.

Conclusions

The results suggest that a “one size fits all” approach does not work and that there is no ideal data stream that is most useful for each goal. Data streams that scored more highly for speed tended to rank more highly when the biosurveillance goal is early warning or early detection, whereas data streams that scored more highly for data credibility and geographic/population coverage ranked highly when the goal was situational awareness or consequence management. However, there are several data streams that rank consistently within the top 5 for each goal: Internet Search Queries, News Aggregators, Clinic/Health Care Provider records, ED/Hospital Records, and Laboratory Records and may be considered useful for integrated, global biosurveillance for infectious disease.

Table 1

Early Warning of Health Threats	Early Detection of Health Events	Situational Awareness	Consequence Management
1. Internet Search Queries	1. News Aggregators	1. Laboratory Records	1. ED/Hospital Records
2. News Aggregators	2. Internet Search Queries	2. ED/Hospital Records	1. Clinic/Healthcare Provider Records
3. Social Media	3. Social Media	2. Clinic/Healthcare provider Records	2. Laboratory Records
4. Laboratory Records	4 ED/Hospital Records	3. News Aggregators	3 Internet Search Queries
5. ED/Hospital Records	4. Clinic/Healthcare Provider Records	4. Internet Search Queries	4. News Aggregators
5. Clinic/Healthcare Provider Records	5. Laboratory Records	5. Official Reports	5. Official Reports
6, Help Lines	6. Help Lines	6. Employment/School Records	6. Ambulance Records
7. Ambulance Records	7. Amublance Records	7. Social Media	7. Employment/School Records
8. Employment/School Records	8. Employment/School records	8. Ambulance Records	8. Social Media
9. Sales	9. Official Reports	9. Personal Communication	9. Established Databases
10. Crowd Sourcing	9. Sales	10. Established Databases	10. Personal Communication
11. Official Reports	10. Crowd Sourcing	11. Help Lines	11. Sales
12. Personal Communication	11. Personal Communication	12. Prediction Markets	12. Help Lines
13. Financial Records	12. Financial Records	13. Financial Records	13. Financial Records
14. Established Databases	13. Prediction Markets	14. Sales	14. Police/Fire Department records
15. Police/Fire Department Records	14. Police/Fire Department Records	15. Police/Fire Department Records	15. Prediction Markets
16. Prediction Markets	15. Established Databases	16. Crowd Sourcing	16. Crowd Sourcing

Open in a separate window