Use of a support vector machine for categorizing free-text notes: assessment of accuracy across two institutions

Background

Electronic health record (EHR) users must regularly review large amounts of data in order to make informed clinical decisions, and such review is time-consuming and often overwhelming. Technologies like automated summarization tools, EHR search engines and natural language processing have been shown to help clinicians manage this information.

Objective

To develop a support vector machine (SVM)-based system for identifying EHR progress notes pertaining to diabetes, and to validate it at two institutions.

Materials and methods

We retrieved 2000 EHR progress notes from patients with diabetes at the Brigham and Women''s Hospital (1000 for training and 1000 for testing) and another 1000 notes from the University of Texas Physicians (for validation). We manually annotated all notes and trained a SVM using a bag of words approach. We then used the SVM on the testing and validation sets and evaluated its performance with the area under the curve (AUC) and F statistics.

Results

The model accurately identified diabetes-related notes in both the Brigham and Women''s Hospital testing set (AUC=0.956, F=0.934) and the external University of Texas Faculty Physicians validation set (AUC=0.947, F=0.935).

Discussion

Overall, the model we developed was quite accurate. Furthermore, it generalized, without loss of accuracy, to another institution with a different EHR and a distinct patient and provider population.

Conclusions

It is possible to use a SVM-based classifier to identify EHR progress notes pertaining to diabetes, and the model generalizes well.     

Sequencing of EHR adoption among US hospitals and the impact of meaningful use

Objective

To examine whether there is a common sequence of adoption of electronic health record (EHR) functions among US hospitals, identify differences by hospital type, and assess the impact of meaningful use.

Materials and methods

Using 2008 American Hospital Association (AHA) Information Technology (IT) Supplement data, we calculate adoption rates of individual EHR functions, along with Loevinger homogeneity (H) coefficients, to assess the sequence of EHR adoption across hospitals. We compare adoption rates and Loevinger H coefficients for hospitals of different types to assess variation in sequencing. We qualitatively assess whether stage 1 meaningful use functions are those adopted early in the sequence.

Results

There is a common sequence of EHR adoption across hospitals, with moderate-to-strong homogeneity. Patient demographic and ancillary results functions are consistently adopted first, while physician notes, clinical reminders, and guidelines are adopted last. Small hospitals exhibited greater homogeneity than larger hospitals. Rural hospitals and non-teaching hospitals exhibited greater homogeneity than urban and teaching hospitals. EHR functions emphasized in stage 1 meaningful use are spread throughout the scale.

Discussion

Stronger homogeneity among small, rural, and non-teaching hospitals may be driven by greater reliance on vendors and less variation in the types of care they deliver. Stage 1 meaningful use is likely changing how hospitals sequence EHR adoption—in particular, by moving clinical guidelines and medication computerized provider order entry ahead in sequence.

Conclusions

While there is a common sequence underlying adoption of EHR functions, the degree of adherence to the sequence varies by key hospital characteristics. Stage 1 meaningful use likely alters the sequence.     

Quantifying clinical narrative redundancy in an electronic health record

Objective

Although electronic notes have advantages compared to handwritten notes, they take longer to write and promote information redundancy in electronic health records (EHRs). We sought to quantify redundancy in clinical documentation by studying collections of physician notes in an EHR.

Design and methods

We implemented a retrospective design to gather all electronic admission, progress, resident signout and discharge summary notes written during 100 randomly selected patient admissions within a 6 month period. We modified and applied a Levenshtein edit-distance algorithm to align and compare the documents written for each of the 100 admissions. We then identified and measured the amount of text duplicated from previous notes. Finally, we manually reviewed the content that was conserved between note types in a subsample of notes.

Measurements

We measured the amount of new information in a document, which was calculated as the number of words that did not match with previous documents divided by the length, in words, of the document. Results are reported as the percentage of information in a document that had been duplicated from previously written documents.

Results

Signout and progress notes proved to be particularly redundant, with an average of 78% and 54% information duplicated from previous documents respectively. There was also significant information duplication between document types (eg, from an admission note to a progress note).

Conclusion

The study established the feasibility of exploring redundancy in the narrative record with a known sequence alignment algorithm used frequently in the field of bioinformatics. The findings provide a foundation for studying the usefulness and risks of redundancy in the EHR.     

Toward personalizing treatment for depression: predicting diagnosis and severity

Objective

Depression is a prevalent disorder difficult to diagnose and treat. In particular, depressed patients exhibit largely unpredictable responses to treatment. Toward the goal of personalizing treatment for depression, we develop and evaluate computational models that use electronic health record (EHR) data for predicting the diagnosis and severity of depression, and response to treatment.

Materials and methods

We develop regression-based models for predicting depression, its severity, and response to treatment from EHR data, using structured diagnosis and medication codes as well as free-text clinical reports. We used two datasets: 35 000 patients (5000 depressed) from the Palo Alto Medical Foundation and 5651 patients treated for depression from the Group Health Research Institute.

Results

Our models are able to predict a future diagnosis of depression up to 12 months in advance (area under the receiver operating characteristic curve (AUC) 0.70–0.80). We can differentiate patients with severe baseline depression from those with minimal or mild baseline depression (AUC 0.72). Baseline depression severity was the strongest predictor of treatment response for medication and psychotherapy.

Conclusions

It is possible to use EHR data to predict a diagnosis of depression up to 12 months in advance and to differentiate between extreme baseline levels of depression. The models use commonly available data on diagnosis, medication, and clinical progress notes, making them easily portable. The ability to automatically determine severity can facilitate assembly of large patient cohorts with similar severity from multiple sites, which may enable elucidation of the moderators of treatment response in the future.     

Under-documentation of chronic kidney disease in the electronic health record in outpatients

Objective

To ascertain if outpatients with moderate chronic kidney disease (CKD) had their condition documented in their notes in the electronic health record (EHR).

Design

Outpatients with CKD were selected based on a reduced estimated glomerular filtration rate and their notes extracted from the Columbia University data warehouse. Two lexical-based classification tools (classifier and word-counter) were developed to identify documentation of CKD in electronic notes.

Measurements

The tools categorized patients'' individual notes on the basis of the presence of CKD-related terms. Patients were categorized as appropriately documented if their notes contained reference to CKD when CKD was present.

Results

The sensitivities of the classifier and word-count methods were 95.4% and 99.8%, respectively. The specificity of both was 99.8%. Categorization of individual patients as appropriately documented was 96.9% accurate. Of 107 patients with manually verified moderate CKD, 32 (22%) lacked appropriate documentation. Patients whose CKD had not been appropriately documented were significantly less likely to be on renin-angiotensin system inhibitors or have urine protein quantified, and had the illness for half as long (15.1 vs 30.7 months; p<0.01) compared to patients with documentation.

Conclusion

Our studies show that lexical-based classification tools can accurately ascertain if appropriate documentation of CKD is present in a EHR. Using this method, we demonstrated under-documentation of patients with moderate CKD. Under-documented patients were less likely to receive CKD guideline recommended care. A tool that prompts providers to document CKD might shorten the time to implementing guideline-based recommendations.     

Implementation and Use of an Electronic Health Record within the Indian Health Service

Objectives

There are limited data regarding implementing electronic health records (EHR) in underserved settings. We evaluated the implementation of an EHR within the Indian Health Service (IHS), a federally funded health system for Native Americans.

Design

We surveyed 223 primary care clinicians practicing at 26 IHS health centers that implemented an EHR between 2003 and 2005.

Methods

The survey instrument assessed clinician attitudes regarding EHR implementation, current utilization of individual EHR functions, and attitudes regarding the use of information technology to improve quality of care in underserved settings. We fit a multivariable logistic regression model to identify correlates of increased utilization of the EHR.

Results

The overall response rate was 56%. Of responding clinicians, 66% felt that the EHR implementation process was positive. One-third (35%) believed that the EHR improved overall quality of care, with many (39%) feeling that it decreased the quality of the patient–doctor interaction. One-third of clinicians (34%) reported consistent use of electronic reminders, and self-report that EHRs improve quality was strongly associated with increased utilization of the EHR (odds ratio 3.03, 95% confidence interval 1.05–8.8). The majority (87%) of clinicians felt that information technology could potentially improve quality of care in rural and underserved settings through the use of tools such as online information sources, telemedicine programs, and electronic health records.

Conclusions

Clinicians support the use of information technology to improve quality in underserved settings, but many felt that it was not currently fulfilling its potential in the IHS, potentially due to limited use of key functions within the EHR.     

A Comparative Evaluation of Full-text,Concept-based,and Context-sensitive Search

Objectives

Study comparatively (1) concept-based search, using documents pre-indexed by a conceptual hierarchy; (2) context-sensitive search, using structured, labeled documents; and (3) traditional full-text search. Hypotheses were: (1) more contexts lead to better retrieval accuracy; and (2) adding concept-based search to the other searches would improve upon their baseline performances.

Design

Use our Vaidurya architecture, for search and retrieval evaluation, of structured documents classified by a conceptual hierarchy, on a clinical guidelines test collection.

Measurements

Precision computed at different levels of recall to assess the contribution of the retrieval methods. Comparisons of precisions done with recall set at 0.5, using t-tests.

Results

Performance increased monotonically with the number of query context elements. Adding context-sensitive elements, mean improvement was 11.1% at recall 0.5. With three contexts, mean query precision was 42% ± 17% (95% confidence interval [CI], 31% to 53%); with two contexts, 32% ± 13% (95% CI, 27% to 38%); and one context, 20% ± 9% (95% CI, 15% to 24%). Adding context-based queries to full-text queries monotonically improved precision beyond the 0.4 level of recall. Mean improvement was 4.5% at recall 0.5. Adding concept-based search to full-text search improved precision to 19.4% at recall 0.5.

Conclusions

The study demonstrated usefulness of concept-based and context-sensitive queries for enhancing the precision of retrieval from a digital library of semi-structured clinical guideline documents. Concept-based searches outperformed free-text queries, especially when baseline precision was low. In general, the more ontological elements used in the query, the greater the resulting precision.     

Area-level factors associated with electronic health record adoption and meaningful use in the Regional Extension Center Program

Objective

To identify area-level correlates of electronic health record (EHR) adoption and meaningful use (MU) among primary care providers (PCPs) enrolled in the Regional Extension Center (REC) Program.

Materials and methods

County-level data on 2013 EHR adoption and MU among REC-enrolled PCPs were obtained from the Office of the National Coordinator for Health Information Technology and linked with other county-level data sources including the Area Resource File, American Community Survey, and Federal Communications Commission''s broadband availability database. Hierarchical models with random intercepts for RECs were employed to assess associations between a broad set of area-level factors and county-level rates of EHR adoption and MU.

Results

Among the 2715 counties examined, the average county-level EHR adoption and MU rates for REC-enrolled PCPs were 87.5% and 54.2%, respectively. Community health center presence and Medicaid enrollment concentration were positively associated with EHR adoption, while metropolitan status and Medicare Advantage enrollment concentration were positively associated with MU. Health professional shortage area status and minority concentration were negatively associated with EHR adoption and MU.

Discussion

Increased financial incentives in areas with greater concentrations of Medicaid and Medicare enrollees may be encouraging EHR adoption and MU among REC-enrolled PCPs. Disparities in EHR adoption and MU in some low-resource and underserved areas remain a concern.

Conclusions

Federal efforts to spur EHR adoption and MU have demonstrated some early success; however, some geographic variations in EHR diffusion indicate that greater attention needs to be paid to ensuring equitable uptake and use of EHRs throughout the US.     

Proof-of-concept Design and Development of an EN13606-based Electronic Health Care Record Service

Objective

The authors present an Electronic Healthcare Record (EHR) server, designed and developed as a proof of concept of the revised prEN13606:2005 European standard concerning EHR communications.

Methods

The development of the server includes five modules: the libraries for the management of the standard reference model, for the demographic package and for the data types; the permanent storage module, built on a relational database; two communication interfaces through which the clients can send information or make queries; the XML (eXtensible Markup Language) process module; and the tools for the validation of the extracts managed, implemented on a defined XML-Schema.

Results

The server was subjected to four phases of trials, the first three with ad hoc test data and processes to ensure that each of the modules complied with its specifications and that the interaction between them provided the expected functionalities. The fourth used real extracts generated by other research groups for the additional purpose of testing the validity of the standard in real-world scenarios.

Conclusion

The acceptable performance of the server has made it possible to include it as a middleware service in a platform for the out-of-hospital follow-up and monitoring of patients with chronic heart disease which, at the present time, supports pilot projects and clinical trials for the evaluation of eHealth services.     

How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use

Background

Studies of the effects of electronic health records (EHRs) have had mixed findings, which may be attributable to unmeasured confounders such as individual variability in use of EHR features.

Objective

To capture physician-level variations in use of EHR features, associations with other predictors, and usage intensity over time.

Methods

Retrospective cohort study of primary care providers eligible for meaningful use at a network of federally qualified health centers, using commercial EHR data from January 2010 through June 2013, a period during which the organization was preparing for and in the early stages of meaningful use.

Results

Data were analyzed for 112 physicians and nurse practitioners, consisting of 430 803 encounters with 99 649 patients. EHR usage metrics were developed to capture how providers accessed and added to patient data (eg, problem list updates), used clinical decision support (eg, responses to alerts), communicated (eg, printing after-visit summaries), and used panel management options (eg, viewed panel reports). Provider-level variability was high: for example, the annual average proportion of encounters with problem lists updated ranged from 5% to 60% per provider. Some metrics were associated with provider, patient, or encounter characteristics. For example, problem list updates were more likely for new patients than established ones, and alert acceptance was negatively correlated with alert frequency.

Conclusions

Providers using the same EHR developed personalized patterns of use of EHR features. We conclude that physician-level usage of EHR features may be a valuable additional predictor in research on the effects of EHRs on healthcare quality and costs.     

Research and applications: A comprehensive study of named entity recognition in Chinese clinical text

Objective

Named entity recognition (NER) is one of the fundamental tasks in natural language processing. In the medical domain, there have been a number of studies on NER in English clinical notes; however, very limited NER research has been carried out on clinical notes written in Chinese. The goal of this study was to systematically investigate features and machine learning algorithms for NER in Chinese clinical text.

Materials and methods

We randomly selected 400 admission notes and 400 discharge summaries from Peking Union Medical College Hospital in China. For each note, four types of entity—clinical problems, procedures, laboratory test, and medications—were annotated according to a predefined guideline. Two-thirds of the 400 notes were used to train the NER systems and one-third for testing. We investigated the effects of different types of feature including bag-of-characters, word segmentation, part-of-speech, and section information, and different machine learning algorithms including conditional random fields (CRF), support vector machines (SVM), maximum entropy (ME), and structural SVM (SSVM) on the Chinese clinical NER task. All classifiers were trained on the training dataset and evaluated on the test set, and micro-averaged precision, recall, and F-measure were reported.

Results

Our evaluation on the independent test set showed that most types of feature were beneficial to Chinese NER systems, although the improvements were limited. The system achieved the highest performance by combining word segmentation and section information, indicating that these two types of feature complement each other. When the same types of optimized feature were used, CRF and SSVM outperformed SVM and ME. More specifically, SSVM achieved the highest performance of the four algorithms, with F-measures of 93.51% and 90.01% for admission notes and discharge summaries, respectively.     

Automated identification of drug and food allergies entered using non-standard terminology

Objective

An accurate computable representation of food and drug allergy is essential for safe healthcare. Our goal was to develop a high-performance, easily maintained algorithm to identify medication and food allergies and sensitivities from unstructured allergy entries in electronic health record (EHR) systems.

Materials and methods

An algorithm was developed in Transact-SQL to identify ingredients to which patients had allergies in a perioperative information management system. The algorithm used RxNorm and natural language processing techniques developed on a training set of 24 599 entries from 9445 records. Accuracy, specificity, precision, recall, and F-measure were determined for the training dataset and repeated for the testing dataset (24 857 entries from 9430 records).

Results

Accuracy, precision, recall, and F-measure for medication allergy matches were all above 98% in the training dataset and above 97% in the testing dataset for all allergy entries. Corresponding values for food allergy matches were above 97% and above 93%, respectively. Specificities of the algorithm were 90.3% and 85.0% for drug matches and 100% and 88.9% for food matches in the training and testing datasets, respectively.

Discussion

The algorithm had high performance for identification of medication and food allergies. Maintenance is practical, as updates are managed through upload of new RxNorm versions and additions to companion database tables. However, direct entry of codified allergy information by providers (through autocompleters or drop lists) is still preferred to post-hoc encoding of the data. Data tables used in the algorithm are available for download.

Conclusions

A high performing, easily maintained algorithm can successfully identify medication and food allergies from free text entries in EHR systems.     

Patient characteristics associated with venous thromboembolic events: a cohort study using pooled electronic health record data

Objective

To demonstrate the potential of de-identified clinical data from multiple healthcare systems using different electronic health records (EHR) to be efficiently used for very large retrospective cohort studies.

Materials and methods

Data of 959 030 patients, pooled from multiple different healthcare systems with distinct EHR, were obtained. Data were standardized and normalized using common ontologies, searchable through a HIPAA-compliant, patient de-identified web application (Explore; Explorys Inc). Patients were 26 years or older seen in multiple healthcare systems from 1999 to 2011 with data from EHR.

Results

Comparing obese, tall subjects with normal body mass index, short subjects, the venous thromboembolic events (VTE) OR was 1.83 (95% CI 1.76 to 1.91) for women and 1.21 (1.10 to 1.32) for men. Weight had more effect then height on VTE. Compared with Caucasian, Hispanic/Latino subjects had a much lower risk of VTE (female OR 0.47, 0.41 to 0.55; male OR 0.24, 0.20 to 0.28) and African-Americans a substantially higher risk (female OR 1.83, 1.76 to 1.91; male OR 1.58, 1.50 to 1.66). This 13-year retrospective study of almost one million patients was performed over approximately 125 h in 11 weeks, part time by the five authors.

Discussion

As research informatics tools develop and more clinical data become available in EHR, it is important to study and understand unique opportunities for clinical research informatics to transform the scale and resources needed to perform certain types of clinical research.

Conclusions

With the right clinical research informatics tools and EHR data, some types of very large cohort studies can be completed with minimal resources.     

Openness of patients' reporting with use of electronic records: psychiatric clinicians' views

Objectives

Improvements in electronic health record (EHR) system development will require an understanding of psychiatric clinicians'' views on EHR system acceptability, including effects on psychotherapy communications, data-recording behaviors, data accessibility versus security and privacy, data quality and clarity, communications with medical colleagues, and stigma.

Design

Multidisciplinary development of a survey instrument targeting psychiatric clinicians who recently switched to EHR system use, focus group testing, data analysis, and data reliability testing.

Measurements

Survey of 120 university-based, outpatient mental health clinicians, with 56 (47%) responding, conducted 18 months after transition from a paper to an EHR system.

Results

Factor analysis gave nine item groupings that overlapped strongly with five a priori domains. Respondents both praised and criticized the EHR system. A strong majority (81%) felt that open therapeutic communications were preserved. Regarding data quality, content, and privacy, clinicians (63%) were less willing to record highly confidential information and disagreed (83%) with including their own psychiatric records among routinely accessed EHR systems.

Limitations

single time point; single academic medical center clinic setting; modest sample size; lack of prior instrument validation; survey conducted in 2005.

Conclusions

In an academic medical center clinic, the presence of electronic records was not seen as a dramatic impediment to therapeutic communications. Concerns regarding privacy and data security were significant, and may contribute to reluctances to adopt electronic records in other settings. Further study of clinicians'' views and use patterns may be helpful in guiding development and deployment of electronic records systems.     

The SMART Platform: early experience enabling substitutable applications for electronic health records

Objective

The Substitutable Medical Applications, Reusable Technologies (SMART) Platforms project seeks to develop a health information technology platform with substitutable applications (apps) constructed around core services. The authors believe this is a promising approach to driving down healthcare costs, supporting standards evolution, accommodating differences in care workflow, fostering competition in the market, and accelerating innovation.

Materials and methods

The Office of the National Coordinator for Health Information Technology, through the Strategic Health IT Advanced Research Projects (SHARP) Program, funds the project. The SMART team has focused on enabling the property of substitutability through an app programming interface leveraging web standards, presenting predictable data payloads, and abstracting away many details of enterprise health information technology systems. Containers—health information technology systems, such as electronic health records (EHR), personally controlled health records, and health information exchanges that use the SMART app programming interface or a portion of it—marshal data sources and present data simply, reliably, and consistently to apps.

Results

The SMART team has completed the first phase of the project (a) defining an app programming interface, (b) developing containers, and (c) producing a set of charter apps that showcase the system capabilities. A focal point of this phase was the SMART Apps Challenge, publicized by the White House, using http://www.challenge.gov website, and generating 15 app submissions with diverse functionality.

Conclusion

Key strategic decisions must be made about the most effective market for further disseminating SMART: existing market-leading EHR vendors, new entrants into the EHR market, or other stakeholders such as health information exchanges.     

Electronic health records in small physician practices: availability,use, and perceived benefits

Objective

To examine variation in the adoption of electronic health record (EHR) functionalities and their use patterns, barriers to adoption, and perceived benefits by physician practice size.

Design

Mailed survey of a nationally representative random sample of practicing physicians identified from the Physician Masterfile of the American Medical Association.

Measurements

We measured, stratified by practice size: (1) availability of EHR functionalities, (2) functionality use, (3) barriers to the adoption and use of EHR, and (4) impact of the EHR on the practice and quality of patient care.

Results

With a response rate of 62%, we found that <2% of physicians in solo or two-physician (small) practices reported a fully functional EHR and 5% reported a basic EHR compared with 13% of physicians from 11+ group (largest group) practices with a fully functional system and 26% with a basic system. Between groups, a 21–46% difference in specific functionalities available was reported. Among adopters there were moderate to large differences in the use of the EHR systems. Financial barriers were more likely to be reported by smaller practices, along with concerns about future obsolescence. These differences were sizable (13–16%) and statistically significant (p<0.001). All adopters reported similar benefits.

Limitations

Although we have adjusted for response bias, influences may still exist.

Conclusion

Our study found that physicians in small practices have lower levels of EHR adoption and that these providers were less likely to use these systems. Ensuring that unique barriers are addressed will be critical to the widespread meaningful use of EHR systems among small practices.     

Harmonized patient-reported data elements in the electronic health record: supporting meaningful use by primary care action on health behaviors and key psychosocial factors

Background

Electronic health records (EHR) have the potential to improve patient care through efficient access to complete patient health information. This potential may not be reached because many of the most important determinants of health outcome are rarely included. Successful health promotion and disease prevention requires patient-reported data reflecting health behaviors and psychosocial issues. Furthermore, there is a need to harmonize this information across different EHR systems.

Methods

To fill this gap a three-phased process was used to conceptualize, identify and recommend patient-reported data elements on health behaviors and psychosocial factors for the EHR. Expert panels (n=13) identified candidate measures (phase 1) that were reviewed and rated by a wide range of health professionals (n=93) using the grid-enabled measures wiki social media platform (phase 2). Recommendations were finalized through a town hall meeting with key stakeholders including patients, providers, researchers, policy makers, and representatives from healthcare settings (phase 3).

Results

Nine key elements from three areas emerged as the initial critical patient-reported elements to incorporate systematically into EHR—health behaviors (eg, exercise), psychosocial issues (eg, distress), and patient-centered factors (eg, demographics). Recommendations were also made regarding the frequency of collection ranging from a single assessment (eg, demographic characteristics), to annual assessment (eg, health behaviors), or more frequent (eg, patient goals).

Conclusions

There was strong stakeholder support for this initiative reflecting the perceived value of incorporating patient-reported elements into EHR. The next steps will include testing the feasibility of incorporating these elements into the EHR across diverse primary care settings.     

semCDI: A Query Formulation for Semantic Data Integration in caBIG

Objectives

To develop mechanisms to formulate queries over the semantic representation of cancer-related data services available through the cancer Biomedical Informatics Grid (caBIG).

Design

The semCDI query formulation uses a view of caBIG semantic concepts, metadata, and data as an ontology, and defines a methodology to specify queries using the SPARQL query language, extended with Horn rules. semCDI enables the joining of data that represent different concepts through associations modeled as object properties, and the merging of data representing the same concept in different sources through Common Data Elements (CDE) modeled as datatype properties, using Horn rules to specify additional semantics indicating conditions for merging data.

Validation

In order to validate this formulation, a prototype has been constructed, and two queries have been executed against currently available caBIG data services.

Discussion

The semCDI query formulation uses the rich semantic metadata available in caBIG to build queries and integrate data from multiple sources. Its promise will be further enhanced as more data services are registered in caBIG, and as more linkages can be achieved between the knowledge contained within caBIG''s NCI Thesaurus and the data contained in the Data Services.

Conclusion

semCDI provides a formulation for the creation of queries on the semantic representation of caBIG. This constitutes the foundation to build a semantic data integration system for more efficient and effective querying and exploratory searching of cancer-related data.     

Integrating existing natural language processing tools for medication extraction from discharge summaries

Objective

To develop an automated system to extract medications and related information from discharge summaries as part of the 2009 i2b2 natural language processing (NLP) challenge. This task required accurate recognition of medication name, dosage, mode, frequency, duration, and reason for drug administration.

Design

We developed an integrated system using several existing NLP components developed at Vanderbilt University Medical Center, which included MedEx (to extract medication information), SecTag (a section identification system for clinical notes), a sentence splitter, and a spell checker for drug names. Our goal was to achieve good performance with minimal to no specific training for this document corpus; thus, evaluating the portability of those NLP tools beyond their home institution. The integrated system was developed using 17 notes that were annotated by the organizers and evaluated using 251 notes that were annotated by participating teams.

Measurements

The i2b2 challenge used standard measures, including precision, recall, and F-measure, to evaluate the performance of participating systems. There were two ways to determine whether an extracted textual finding is correct or not: exact matching or inexact matching. The overall performance for all six types of medication-related findings across 251 annotated notes was considered as the primary metric in the challenge.

Results

Our system achieved an overall F-measure of 0.821 for exact matching (0.839 precision; 0.803 recall) and 0.822 for inexact matching (0.866 precision; 0.782 recall). The system ranked second out of 20 participating teams on overall performance at extracting medications and related information.

Conclusions

The results show that the existing MedEx system, together with other NLP components, can extract medication information in clinical text from institutions other than the site of algorithm development with reasonable performance.