A research agenda to support the development and implementation of genomics-based clinical informatics tools and resources

ObjectiveThe Genomic Medicine Working Group of the National Advisory Council for Human Genome Research virtually hosted its 13th genomic medicine meeting titled “Developing a Clinical Genomic Informatics Research Agenda”. The meeting’s goal was to articulate a research strategy to develop Genomics-based Clinical Informatics Tools and Resources (GCIT) to improve the detection, treatment, and reporting of genetic disorders in clinical settings.Materials and MethodsExperts from government agencies, the private sector, and academia in genomic medicine and clinical informatics were invited to address the meeting''s goals. Invitees were also asked to complete a survey to assess important considerations needed to develop a genomic-based clinical informatics research strategy.ResultsOutcomes from the meeting included identifying short-term research needs, such as designing and implementing standards-based interfaces between laboratory information systems and electronic health records, as well as long-term projects, such as identifying and addressing barriers related to the establishment and implementation of genomic data exchange systems that, in turn, the research community could help address.DiscussionDiscussions centered on identifying gaps and barriers that impede the use of GCIT in genomic medicine. Emergent themes from the meeting included developing an implementation science framework, defining a value proposition for all stakeholders, fostering engagement with patients and partners to develop applications under patient control, promoting the use of relevant clinical workflows in research, and lowering related barriers to regulatory processes. Another key theme was recognizing pervasive biases in data and information systems, algorithms, access, value, and knowledge repositories and identifying ways to resolve them.     

Clinical informatics during the COVID-19 pandemic: Lessons learned and implications for emergency department and inpatient operations

In response to a pandemic, hospital leaders can use clinical informatics to aid clinical decision making, virtualizing medical care, coordinating communication, and defining workflow and compliance. Clinical informatics procedures need to be implemented nimbly, with governance measures in place to properly oversee and guide novel patient care pathways, diagnostic and treatment workflows, and provider education and communication. The authors’ experience recommends (1) creating flexible order sets that adapt to evolving guidelines that meet needs across specialties, (2) enhancing and supporting inherent telemedicine capability, (3) electronically enabling novel workflows quickly and suspending noncritical administrative or billing functions in the electronic health record, and (4) using communication platforms based on tiered urgency that do not compromise security and privacy.     

Building capacity of community health centers to overcome data challenges with the development of an agile COVID-19 public health registry: a multistate quality improvement effort

ObjectiveDuring the coronavirus disease 2019 (COVID-19) pandemic, federally qualified health centers rapidly mobilized to provide SARS-CoV-2 testing, COVID-19 care, and vaccination to populations at increased risk for COVID-19 morbidity and mortality. We describe the development of a reusable public health data analytics system for reuse of clinical data to evaluate the health burden, disparities, and impact of COVID-19 on populations served by health centers.Materials and MethodsThe Multistate Data Strategy engaged project partners to assess public health readiness and COVID-19 data challenges. An infrastructure for data capture and sharing procedures between health centers and public health agencies was developed to support existing capabilities and data capacities to respond to the pandemic.ResultsBetween August 2020 and March 2021, project partners evaluated their data capture and sharing capabilities and reported challenges and preliminary data. Major interoperability challenges included poorly aligned federal, state, and local reporting requirements, lack of unique patient identifiers, lack of access to pharmacy, claims and laboratory data, missing data, and proprietary data standards and extraction methods.DiscussionEfforts to access and align project partners’ existing health systems data infrastructure in the context of the pandemic highlighted complex interoperability challenges. These challenges remain significant barriers to real-time data analytics and efforts to improve health outcomes and mitigate inequities through data-driven responses.ConclusionThe reusable public health data analytics system created in the Multistate Data Strategy can be adapted and scaled for other health center networks to facilitate data aggregation and dashboards for public health, organizational planning, and quality improvement and can inform local, state, and national COVID-19 response efforts.     

A retrospective look at the predictions and recommendations from the 2009 AMIA policy meeting: did we see EHR-related clinician burnout coming?

Clinicians often attribute much of their burnout experience to use of the electronic health record, the adoption of which was greatly accelerated by the Health Information Technology for Economic and Clinical Health Act of 2009. That same year, AMIA’s Policy Meeting focused on possible unintended consequences associated with rapid implementation of electronic health records, generating 17 potential consequences and 15 recommendations to address them. At the 2020 annual meeting of the American College of Medical Informatics (ACMI), ACMI fellows participated in a modified Delphi process to assess the accuracy of the 2009 predictions and the response to the recommendations. Among the findings, the fellows concluded that the degree of clinician burnout and its contributing factors, such as increased documentation requirements, were significantly underestimated. Conversely, problems related to identify theft and fraud were overestimated. Only 3 of the 15 recommendations were adjudged more than half-addressed.     

Development and validation of prediction models for mechanical ventilation,renal replacement therapy,and readmission in COVID-19 patients

ObjectiveCoronavirus disease 2019 (COVID-19) patients are at risk for resource-intensive outcomes including mechanical ventilation (MV), renal replacement therapy (RRT), and readmission. Accurate outcome prognostication could facilitate hospital resource allocation. We develop and validate predictive models for each outcome using retrospective electronic health record data for COVID-19 patients treated between March 2 and May 6, 2020.Materials and MethodsFor each outcome, we trained 3 classes of prediction models using clinical data for a cohort of SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2)–positive patients (n = 2256). Cross-validation was used to select the best-performing models per the areas under the receiver-operating characteristic and precision-recall curves. Models were validated using a held-out cohort (n = 855). We measured each model’s calibration and evaluated feature importances to interpret model output.ResultsThe predictive performance for our selected models on the held-out cohort was as follows: area under the receiver-operating characteristic curve—MV 0.743 (95% CI, 0.682-0.812), RRT 0.847 (95% CI, 0.772-0.936), readmission 0.871 (95% CI, 0.830-0.917); area under the precision-recall curve—MV 0.137 (95% CI, 0.047-0.175), RRT 0.325 (95% CI, 0.117-0.497), readmission 0.504 (95% CI, 0.388-0.604). Predictions were well calibrated, and the most important features within each model were consistent with clinical intuition.DiscussionOur models produce performant, well-calibrated, and interpretable predictions for COVID-19 patients at risk for the target outcomes. They demonstrate the potential to accurately estimate outcome prognosis in resource-constrained care sites managing COVID-19 patients.ConclusionsWe develop and validate prognostic models targeting MV, RRT, and readmission for hospitalized COVID-19 patients which produce accurate, interpretable predictions. Additional external validation studies are needed to further verify the generalizability of our results.     

Inviting patients to identify diagnostic concerns through structured evaluation of their online visit notes

BackgroundThe 21st Century Cures Act mandates patients’ access to their electronic health record (EHR) notes. To our knowledge, no previous work has systematically invited patients to proactively report diagnostic concerns while documenting and tracking their diagnostic experiences through EHR-based clinician note review.ObjectiveTo test if patients can identify concerns about their diagnosis through structured evaluation of their online visit notes.MethodsIn a large integrated health system, patients aged 18–85 years actively using the patient portal and seen between October 2019 and February 2020 were invited to respond to an online questionnaire if an EHR algorithm detected any recent unexpected return visit following an initial primary care consultation (“at-risk” visit). We developed and tested an instrument (Safer Dx Patient Instrument) to help patients identify concerns related to several dimensions of the diagnostic process based on notes review and recall of recent “at-risk” visits. Additional questions assessed patients’ trust in their providers and their general feelings about the visit. The primary outcome was a self-reported diagnostic concern. Multivariate logistic regression tested whether the primary outcome was predicted by instrument variables.ResultsOf 293 566 visits, the algorithm identified 1282 eligible patients, of whom 486 responded. After applying exclusion criteria, 418 patients were included in the analysis. Fifty-one patients (12.2%) identified a diagnostic concern. Patients were more likely to report a concern if they disagreed with statements “the care plan the provider developed for me addressed all my medical concerns” [odds ratio (OR), 2.65; 95% confidence interval [CI], 1.45–4.87) and “I trust the provider that I saw during my visit” (OR, 2.10; 95% CI, 1.19–3.71) and agreed with the statement “I did not have a good feeling about my visit” (OR, 1.48; 95% CI, 1.09–2.01).ConclusionPatients can identify diagnostic concerns based on a proactive online structured evaluation of visit notes. This surveillance strategy could potentially improve transparency in the diagnostic process.     

The national landscape of culminating experiences in master’s programs in health and biomedical informatics

Health and biomedical informatics graduate-level degree programs have proliferated across the United States in the last 10 years. To help inform programs on practices in teaching and learning, a survey of master’s programs in health and biomedical informatics in the United States was conducted to determine the national landscape of culminating experiences including capstone projects, research theses, internships, and practicums. Almost all respondents reported that their programs required a culminating experience (97%). A paper (not a formal thesis), an oral presentation, a formal course, and an internship were required by ≥50% programs. The most commonly reported purposes for the culminating experience were to help students extend and apply the learning and as a bridge to the workplace. The biggest challenges were students’ maturity, difficulty in synthesizing information into a coherent paper, and ability to generate research ideas. The results provide students and program leaders with a summary of pedagogical methods across programs.     

Dynamics of social corrections to peers sharing COVID-19 misinformation on WhatsApp in Brazil

ObjectiveOnline COVID-19 misinformation is a serious concern in Brazil, home to the second-largest WhatsApp user base and the second-highest number of COVID-19 deaths. We examined the extent to which WhatsApp users might be willing to correct their peers who might share COVID-19 misinformation.Materials and MethodsWe conducted a cross-sectional online survey using Qualtrics among 726 Brazilian adults to identify the types of social correction behaviors (SCBs) and health and technological factors that shape the performance of these behaviors.ResultsBrazil’s WhatsApp users expressed medium to high levels of willingness to engage in SCBs. We discovered 3 modes of SCBs: correction to the group, correction to the sender only, and passive or no correction. WhatsApp users with lower levels of educational attainment and from younger age groups were less inclined to provide corrections. Lastly, the perceived severity of COVID-19 and the ability to critically evaluate a message were positively associated with providing corrections to either the group or the sender.DiscussionThe demographic analyses point to the need to strengthen information literacy among population groups that are younger with lower levels of educational attainment. These efforts could facilitate individual-level contributions to the global fight against misinformation by the World Health Organization in collaboration with member states, social media companies, and civil society.ConclusionOur study suggests that Brazil’s WhatsApp users might be willing to actively respond with feedback when exposed to COVID-19 misinformation by their peers on small-world networks like WhatsApp groups.     

Feeling better on hemodialysis: user-centered design requirements for promoting patient involvement in the prevention of treatment complications

ObjectiveHemodialysis patients frequently experience dialysis therapy sessions complicated by intradialytic hypotension (IDH), a major patient safety concern. We investigate user-centered design requirements for a theory-informed, peer mentoring-based, informatics intervention to activate patients toward IDH prevention.MethodsWe conducted observations (156 hours) and interviews (n = 28) with patients in 3 hemodialysis clinics, followed by 9 focus groups (including participatory design activities) with patients (n = 17). Inductive and deductive analyses resulted in themes and design principles linked to constructs from social, cognitive, and self-determination theories.ResultsHemodialysis patients want an informatics intervention for IDH prevention that collapses distance between patients, peers, and family; harnesses patients’ strength of character and resolve in all parts of their life; respects and supports patients’ individual needs, preferences, and choices; and links “feeling better on dialysis” to becoming more involved in IDH prevention. Related design principles included designing for: depth of interpersonal connections; positivity; individual choice and initiative; and comprehension of connections and possible actions.DiscussionFindings advance the design of informatics interventions by presenting design requirements for outpatient safety and addressing key design opportunities for informatics to support patient involvement; these include incorporation of behavior change theories. Results also demonstrate the meaning of design choices for hemodialysis patients in the context of their experiences; this may have applicability to other populations with serious illnesses.ConclusionThe resulting patient-facing informatics intervention will be evaluated in a pragmatic cluster-randomized controlled trial in 28 hemodialysis facilities in 4 US regions.     

An approach to predicting patient experience through machine learning and social network analysis

ObjectiveImproving the patient experience has become an essential component of any healthcare system’s performance metrics portfolio. In this study, we developed a machine learning model to predict a patient’s response to the Hospital Consumer Assessment of Healthcare Providers and Systems survey’s “Doctor Communications” domain questions while simultaneously identifying most impactful providers in a network.Materials and MethodsThis is an observational study of patients admitted to a single tertiary care hospital between 2016 and 2020. Using machine learning algorithms, electronic health record data were used to predict patient responses to Hospital Consumer Assessment of Healthcare Providers and Systems survey questions in the doctor domain, and patients who are at risk for responding negatively were identified. Model performance was assessed by area under receiver-operating characteristic curve. Social network analysis metrics were also used to identify providers most impactful to patient experience.ResultsUsing a random forest algorithm, patients’ responses to the following 3 questions were predicted: “During this hospital stay how often did doctors. 1) treat you with courtesy and respect? 2) explain things in a way that you could understand? 3) listen carefully to you?” with areas under the receiver-operating characteristic curve of 0.876, 0.819, and 0.819, respectively. Social network analysis found that doctors with higher centrality appear to have an outsized influence on patient experience, as measured by rank in the random forest model in the doctor domain.ConclusionsA machine learning algorithm identified patients at risk of a negative experience. Furthermore, a doctor social network framework provides metrics for identifying those providers that are most influential on the patient experience.     

A digital health profile & maturity assessment toolkit: cocreation and testing in the Pacific Islands

IntroductionCountries need to determine their level of digital health capability maturity to assess and mobilize their knowledge, skills, and resources to systematically develop, implement, evaluate, scale up and maintain large-scale implementations of standards-based interoperable digital health tools.ObjectiveDevelop a Digital Health Profile and Maturity Assessment Toolkit (DHPMAT) to assist Pacific Island Countries (PICs) to harness digital tools to support national health priorities.Materials and MethodsA literature review guided the development of the conceptual framework to underpin the DHPMAT. Key informants collaborated to collect key digital health features and indicators to inform their country’s digital health maturity assessment. The DHPMAT was tested with country stakeholders at a Pacific Health Information Network workshop in 2019.ResultsA comprehensive list of indicators to describe country digital health profiles (DHP). A digital health maturity assessment tool that uses criteria codeveloped with country stakeholders to assess essential digital health foundations and quality improvement. DHPs created and maturity assessed and packaged into individualized DHPMATs for 13 PICs. PIC users perceived the DHPMAT as useful, especially the congruence with the 2017 WHO WPRO Regional Strategy but noted a “cognitive overload” from a plethora of complex digital health toolkits.ConclusionsThe cocreation approach optimized currency, accuracy, and appropriateness of information in the DHP, understanding, and use of the DHPMAT to facilitate informed iterative discussion by PICs on their digital health maturity to harness digital tools to strengthen country health systems. The DHPMAT can rationalize the choice and use of existing tools and reduce cognitive overload.     

Physicians’ electronic inbox work patterns and factors associated with high inbox work duration

ObjectivesElectronic health record systems are increasingly used to send messages to physicians, but research on physicians’ inbox use patterns is limited. This study’s aims were to (1) quantify the time primary care physicians (PCPs) spend managing inboxes; (2) describe daily patterns of inbox use; (3) investigate which types of messages consume the most time; and (4) identify factors associated with inbox work duration.Materials and MethodsWe analyzed 1 month of electronic inbox data for 1275 PCPs in a large medical group and linked these data with physicians’ demographic data.ResultsPCPs spent an average of 52 minutes on inbox management on workdays, including 19 minutes (37%) outside work hours. Temporal patterns of electronic inbox use differed from other EHR functions such as charting. Patient-initiated messages (28%) and results (29%) accounted for the most inbox work time. PCPs with higher inbox work duration were more likely to be female (P < .001), have more patient encounters (P < .001), have older patients (P < .001), spend proportionally more time on patient messages (P < .001), and spend more time per message (P < .001). Compared with PCPs with the lowest duration of time on inbox work, PCPs with the highest duration had more message views per workday (200 vs 109; P < .001) and spent more time on the inbox outside work hours (30 minutes vs 9.7 minutes; P < .001).ConclusionsElectronic inbox work by PCPs requires roughly an hour per workday, much of which occurs outside scheduled work hours. Interventions to assist PCPs in handling patient-initiated messages and results may help alleviate inbox workload.     

An interview study with medical scribes on how their work may alleviate clinician burnout through delegated health IT tasks

ObjectivesTo understand how medical scribes’ work may contribute to alleviating clinician burnout attributable directly or indirectly to the use of health IT.Materials and MethodsQualitative analysis of semistructured interviews with 32 participants who had scribing experience in a variety of clinical settings.ResultsWe identified 7 categories of clinical tasks that clinicians commonly choose to offload to medical scribes, many of which involve delegated use of health IT. These range from notes-taking and computerized data entry to foraging, assembling, and tracking information scattered across multiple clinical information systems. Some common characteristics shared among these tasks include: (1) time-consuming to perform; (2) difficult to remember or keep track of; (3) disruptive to clinical workflow, clinicians’ cognitive processes, or patient–provider interactions; (4) perceived to be low-skill “clerical” work; and (5) deemed as adding no value to direct patient care.DiscussionThe fact that clinicians opt to “outsource” certain clinical tasks to medical scribes is a strong indication that performing these tasks is not perceived to be the best use of their time. Given that a vast majority of healthcare practices in the US do not have the luxury of affording medical scribes, the burden would inevitably fall onto clinicians’ shoulders, which could be a major source for clinician burnout.ConclusionsMedical scribes help to offload a substantial amount of burden from clinicians—particularly with tasks that involve onerous interactions with health IT. Developing a better understanding of medical scribes’ work provides useful insights into the sources of clinician burnout and potential solutions to it.     

EHR-based cohort assessment for multicenter RCTs: a fast and flexible model for identifying potential study sites

ObjectiveThe Recruitment Innovation Center (RIC), partnering with the Trial Innovation Network and institutions in the National Institutes of Health-sponsored Clinical and Translational Science Awards (CTSA) Program, aimed to develop a service line to retrieve study population estimates from electronic health record (EHR) systems for use in selecting enrollment sites for multicenter clinical trials. Our goal was to create and field-test a low burden, low tech, and high-yield method.Materials and MethodsIn building this service line, the RIC strove to complement, rather than replace, CTSA hubs’ existing cohort assessment tools. For each new EHR cohort request, we work with the investigator to develop a computable phenotype algorithm that targets the desired population. CTSA hubs run the phenotype query and return results using a standardized survey. We provide a comprehensive report to the investigator to assist in study site selection.ResultsFrom 2017 to 2020, the RIC developed and socialized 36 phenotype-dependent cohort requests on behalf of investigators. The average response rate to these requests was 73%.DiscussionAchieving enrollment goals in a multicenter clinical trial requires that researchers identify study sites that will provide sufficient enrollment. The fast and flexible method the RIC has developed, with CTSA feedback, allows hubs to query their EHR using a generalizable, vetted phenotype algorithm to produce reliable counts of potentially eligible study participants.ConclusionThe RIC’s EHR cohort assessment process for evaluating sites for multicenter trials has been shown to be efficient and helpful. The model may be replicated for use by other programs.     

Developing machine learning models to personalize care levels among emergency room patients for hospital admission

ObjectiveTo develop prediction models for intensive care unit (ICU) vs non-ICU level-of-care need within 24 hours of inpatient admission for emergency department (ED) patients using electronic health record data.Materials and MethodsUsing records of 41 654 ED visits to a tertiary academic center from 2015 to 2019, we tested 4 algorithms—feed-forward neural networks, regularized regression, random forests, and gradient-boosted trees—to predict ICU vs non-ICU level-of-care within 24 hours and at the 24th hour following admission. Simple-feature models included patient demographics, Emergency Severity Index (ESI), and vital sign summary. Complex-feature models added all vital signs, lab results, and counts of diagnosis, imaging, procedures, medications, and lab orders.ResultsThe best-performing model, a gradient-boosted tree using a full feature set, achieved an AUROC of 0.88 (95%CI: 0.87–0.89) and AUPRC of 0.65 (95%CI: 0.63–0.68) for predicting ICU care need within 24 hours of admission. The logistic regression model using ESI achieved an AUROC of 0.67 (95%CI: 0.65–0.70) and AUPRC of 0.37 (95%CI: 0.35–0.40). Using a discrimination threshold, such as 0.6, the positive predictive value, negative predictive value, sensitivity, and specificity were 85%, 89%, 30%, and 99%, respectively. Vital signs were the most important predictors.Discussion and ConclusionsUndertriaging admitted ED patients who subsequently require ICU care is common and associated with poorer outcomes. Machine learning models using readily available electronic health record data predict subsequent need for ICU admission with good discrimination, substantially better than the benchmarking ESI system. The results could be used in a multitiered clinical decision-support system to improve ED triage.     

Rapid deployment of an algorithm to triage dental emergencies during COVID-19 pandemic

COVID-19 quickly immobilized healthcare systems in the United States during the early stages of the outbreak. While much of the ensuing response focused on supporting the medical infrastructure, Columbia University College of Dental Medicine pursued a solution to triage and safely treat patients with dental emergencies amid the pandemic. Considering rapidly changing guidelines from governing bodies, dental infection control protocols, and our clinical faculty''s expertise, we modeled, built, and implemented a screening algorithm, which provides decision support as well as insight into COVID-19 status and clinical comorbidities, within a newly integrated electronic health record (EHR). Once operationalized, we analyzed the data and outcomes of its utilization and found that it had effectively guided providers in triaging patient needs in a standardized methodology. This article describes the algorithm’s rapid development to assist faculty providers in identifying patients with the most urgent needs, thus prioritizing treatment of dental emergencies during the pandemic.     

Towards clinical data-driven eligibility criteria optimization for interventional COVID-19 clinical trials

ObjectiveThis research aims to evaluate the impact of eligibility criteria on recruitment and observable clinical outcomes of COVID-19 clinical trials using electronic health record (EHR) data.Materials and MethodsOn June 18, 2020, we identified frequently used eligibility criteria from all the interventional COVID-19 trials in ClinicalTrials.gov (n = 288), including age, pregnancy, oxygen saturation, alanine/aspartate aminotransferase, platelets, and estimated glomerular filtration rate. We applied the frequently used criteria to the EHR data of COVID-19 patients in Columbia University Irving Medical Center (CUIMC) (March 2020–June 2020) and evaluated their impact on patient accrual and the occurrence of a composite endpoint of mechanical ventilation, tracheostomy, and in-hospital death.ResultsThere were 3251 patients diagnosed with COVID-19 from the CUIMC EHR included in the analysis. The median follow-up period was 10 days (interquartile range 4–28 days). The composite events occurred in 18.1% (n = 587) of the COVID-19 cohort during the follow-up. In a hypothetical trial with common eligibility criteria, 33.6% (690/2051) were eligible among patients with evaluable data and 22.2% (153/690) had the composite event.DiscussionBy adjusting the thresholds of common eligibility criteria based on the characteristics of COVID-19 patients, we could observe more composite events from fewer patients.ConclusionsThis research demonstrated the potential of using the EHR data of COVID-19 patients to inform the selection of eligibility criteria and their thresholds, supporting data-driven optimization of participant selection towards improved statistical power of COVID-19 trials.