Using Computer‐Extracted Data from Electronic Health Records to Measure the Quality of Adolescent Well‐Care

Objective

To determine whether quality measures based on computer-extracted EHR data can reproduce findings based on data manually extracted by reviewers.

Data Sources

We studied 12 measures of care indicated for adolescent well-care visits for 597 patients in three pediatric health systems.

Study Design

Observational study.

Data Collection/Extraction Methods

Manual reviewers collected quality data from the EHR. Site personnel programmed their EHR systems to extract the same data from structured fields in the EHR according to national health IT standards.

Principal Findings

Overall performance measured via computer-extracted data was 21.9 percent, compared with 53.2 percent for manual data. Agreement measures were high for immunizations. Otherwise, agreement between computer extraction and manual review was modest (Kappa = 0.36) because computer-extracted data frequently missed care events (sensitivity = 39.5 percent). Measure validity varied by health care domain and setting. A limitation of our findings is that we studied only three domains and three sites.

Conclusions

The accuracy of computer-extracted EHR quality reporting depends on the use of structured data fields, with the highest agreement found for measures and in the setting that had the greatest concentration of structured fields. We need to improve documentation of care, data extraction, and adaptation of EHR systems to practice workflow.