Identifying Goals of Care Conversations in the Electronic Health Record Using Natural Language Processing and Machine Learning

ContextGoals-of-care discussions are an important quality metric in palliative care. However, goals-of-care discussions are often documented as free text in diverse locations. It is difficult to identify these discussions in the electronic health record (EHR) efficiently.ObjectivesTo develop, train, and test an automated approach to identifying goals-of-care discussions in the EHR, using natural language processing (NLP) and machine learning (ML).MethodsFrom the electronic health records of an academic health system, we collected a purposive sample of 3183 EHR notes (1435 inpatient notes and 1748 outpatient notes) from 1426 patients with serious illness over 2008–2016, and manually reviewed each note for documentation of goals-of-care discussions. Separately, we developed a program to identify notes containing documentation of goals-of-care discussions using NLP and supervised ML. We estimated the performance characteristics of the NLP/ML program across 100 pairs of randomly partitioned training and test sets. We repeated these methods for inpatient-only and outpatient-only subsets.ResultsOf 3183 notes, 689 contained documentation of goals-of-care discussions. The mean sensitivity of the NLP/ML program was 82.3% (SD 3.2%), and the mean specificity was 97.4% (SD 0.7%). NLP/ML results had a median positive likelihood ratio of 32.2 (IQR 27.5–39.2) and a median negative likelihood ratio of 0.18 (IQR 0.16–0.20). Performance was better in inpatient-only samples than outpatient-only samples.ConclusionUsing NLP and ML techniques, we developed a novel approach to identifying goals-of-care discussions in the EHR. NLP and ML represent a potential approach toward measuring goals-of-care discussions as a research outcome and quality metric.