Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy |
| |
Authors: | Christopher C. Stahl Sarah A. Jung Alexandra A. Rosser Aaron S. Kraut Benjamin H. Schnapp Mary Westergaard Azita G. Hamedani Rebecca M. Minter Jacob A. Greenberg |
| |
Affiliation: | 2. Department of Emergency Medicine, Queen''s University, Kingston, Ontario, Canada;2. Department of Medical Education, Feinberg School of Medicine, Northwestern University, Chicago Illinois;3. Department of Surgical Education, Feinberg School of Medicine, Northwestern University, Chicago Illinois |
| |
Abstract: | BackgroundEntrustable Professional Activities (EPAs) contain narrative ‘entrustment roadmaps’ designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice.MethodsAll text comments associated with EPA microassessments at a single institution were combined. EPA—entrustment level pairs (e.g. Gallbladder Disease—Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters.ResultsOver 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics).ConclusionsLDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps. |
| |
Keywords: | Surgery education Natural language processing Entrustable professional activities Assessment Feedback |
本文献已被 ScienceDirect 等数据库收录! |
|