Cross-vendor evaluation of key user-defined clinical decision support capabilities: a scenario-based assessment of certified electronic health records with guidelines for future development

Objective Clinical decision support (CDS) is essential for delivery of high-quality, cost-effective, and safe healthcare. The authors sought to evaluate the CDS capabilities across electronic health record (EHR) systems.Methods We evaluated the CDS implementation capabilities of 8 Office of the National Coordinator for Health Information Technology Authorized Certification Body (ONC-ACB)-certified EHRs. Within each EHR, the authors attempted to implement 3 user-defined rules that utilized the various data and logic elements expected of typical EHRs and that represented clinically important evidenced-based care. The rules were: 1) if a patient has amiodarone on his or her active medication list and does not have a thyroid-stimulating hormone (TSH) result recorded in the last 12 months, suggest ordering a TSH; 2) if a patient has a hemoglobin A1c result >7% and does not have diabetes on his or her problem list, suggest adding diabetes to the problem list; and 3) if a patient has coronary artery disease on his or her problem list and does not have aspirin on the active medication list, suggest ordering aspirin.Results Most evaluated EHRs lacked some CDS capabilities; 5 EHRs were able to implement all 3 rules, and the remaining 3 EHRs were unable to implement any of the rules. One of these did not allow users to customize CDS rules at all. The most frequently found shortcomings included the inability to use laboratory test results in rules, limit rules by time, use advanced Boolean logic, perform actions from the alert interface, and adequately test rules.Conclusion Significant improvements in the EHR certification and implementation procedures are necessary.     

A visual representation of microbiological culture data improves comprehension: a randomized controlled trial

ObjectiveWhile the judicious use of antibiotics takes past microbiological culture results into consideration, this data’s typical format in the electronic health record (EHR) may be unwieldy when incorporated into clinical decision-making. We hypothesize that a visual representation of sensitivities may aid in their comprehension.Materials and MethodsA prospective parallel unblinded randomized controlled trial was undertaken at an academic urban tertiary care center. Providers managing emergency department (ED) patients receiving antibiotics and having previous culture sensitivity testing were included. Providers were randomly selected to use standard EHR functionality or a visual representation of patients’ past culture data as they answered questions about previous sensitivities. Concordance between provider responses and past cultures was assessed using the kappa statistic. Providers were surveyed about their decision-making and the usability of the tool using Likert scales.Results518 ED encounters were screened from 3/5/2018 to 9/30/18, with providers from 144 visits enrolled and analyzed in the intervention arm and 129 in the control arm. Providers using the visualization tool had a kappa of 0.69 (95% CI: 0.65–0.73) when asked about past culture results while the control group had a kappa of 0.16 (95% CI: 0.12–0.20). Providers using the tool expressed improved understanding of previous cultures and found the tool easy to use (P < .001). Secondary outcomes showed no differences in prescribing practices.ConclusionA visual representation of culture sensitivities improves comprehension when compared to standard text-based representations.     

The design and evaluation of a graphical display for laboratory data

Objective

Advances in healthcare information technology have provided opportunities to present data in new, more effective ways. In this study, we designed a laboratory display that features small, data-dense graphics called sparklines, which have recently been promoted as effective representations of medical data but have not been well studied. The effect of this novel display on physicians'' interpretation of data was investigated.

Design

Twelve physicians talked aloud as they assessed laboratory data from four patients in a pediatric intensive care unit with the graph display and with a conventional table display.

Measurements

Verbalizations were coded based on the abnormal values and trends identified for each lab variable. The correspondence of interpretations of variables in each display was evaluated, and patterns were investigated across participants. Assessment time was also analyzed.

Results

Physicians completed assessments significantly faster with the graphical display (3.6 min vs 4.4 min, p=0.042). When compared across displays, 37% of interpretations did not match. Graphs were more useful when the visual cues in tables did not provide trend information, while slightly abnormal values were easier to identify with tables.

Conclusions

Data presentation format can affect how physicians interpret laboratory data. Graphic displays have several advantages over numeric displays but are not always optimal. User, task and data characteristics should be considered when designing information displays.     

构建临床决策支持系统的良性生态

在电子病历得到应用后,临床决策支持系统(CDSS)是进一步提升医疗安全与质量的重要手段。但是当前CDSS的应用却遇到许多困难。文章根据我国当前电子病历应用水平评估的情况归纳了临床决策支持应用的6种类型和5种常见的应用场景,分析了技术上促进CDSS应用需要解决的4个主要问题,提出了以优先解决电子病历系统与CDSS接口的标准化作为入手推动临床决策支持应用的方案。通过扩大CDSS的应用带动知识描述标准的建立,进而引导市场化方式发展通用的知识库,由此形成临床决策支持系统良性发展的生态。     

基于统一信息模型的CDSS与EMR接口技术研究与实现

临床决策支持系统(CDSS)的实施和应用具有重要的临床意义,也是医院电子病历系统(EMR)应用水平评价的重要内容.由于CDSS和EMR接口开发的困难,CDSS的部署成本高、效率低,阻碍了CDSS的广泛应用.本文在分析接口开发挑战的基础上,提出了一种基于统一信息模型的接口实现技术,并在临床案例中验证了方法的可行性.     

Evaluating the decision accuracy and speed of clinical data visualizations

Clinicians face an increasing volume of biomedical data. Assessing the efficacy of systems that enable accurate and timely clinical decision making merits corresponding attention. This paper discusses the multiple-reader multiple-case (MRMC) experimental design and linear mixed models as means of assessing and comparing decision accuracy and latency (time) for decision tasks in which clinician readers must interpret visual displays of data. These tools can assess and compare decision accuracy and latency (time). These experimental and statistical techniques, used extensively in radiology imaging studies, offer a number of practical and analytic advantages over more traditional quantitative methods such as percent-correct measurements and ANOVAs, and are recommended for their statistical efficiency and generalizability. An example analysis using readily available, free, and commercial statistical software is provided as an appendix. While these techniques are not appropriate for all evaluation questions, they can provide a valuable addition to the evaluative toolkit of medical informatics research.     

Assessing the quality of clinical and administrative data extracted from hospitals: the General Medicine Inpatient Initiative (GEMINI) experience

ObjectiveLarge clinical databases are increasingly used for research and quality improvement. We describe an approach to data quality assessment from the General Medicine Inpatient Initiative (GEMINI), which collects and standardizes administrative and clinical data from hospitals.MethodsThe GEMINI database contained 245 559 patient admissions at 7 hospitals in Ontario, Canada from 2010 to 2017. We performed 7 computational data quality checks and iteratively re-extracted data from hospitals to correct problems. Thereafter, GEMINI data were compared to data that were manually abstracted from the hospital’s electronic medical record for 23 419 selected data points on a sample of 7488 patients.ResultsComputational checks flagged 103 potential data quality issues, which were either corrected or documented to inform future analysis. For example, we identified the inclusion of canceled radiology tests, a time shift of transfusion data, and mistakenly processing the chemical symbol for sodium (“Na”) as a missing value. Manual validation identified 1 important data quality issue that was not detected by computational checks: transfusion dates and times at 1 site were unreliable. Apart from that single issue, across all data tables, GEMINI data had high overall accuracy (ranging from 98%–100%), sensitivity (95%–100%), specificity (99%–100%), positive predictive value (93%–100%), and negative predictive value (99%–100%) compared to the gold standard.Discussion and ConclusionComputational data quality checks with iterative re-extraction facilitated reliable data collection from hospitals but missed 1 critical quality issue. Combining computational and manual approaches may be optimal for assessing the quality of large multisite clinical databases.     

Can decision support combat incompleteness and bias in routine primary care data?

ObjectiveRoutine primary care data may be used for the derivation of clinical prediction rules and risk scores. We sought to measure the impact of a decision support system (DSS) on data completeness and freedom from bias.Materials and MethodsWe used the clinical documentation of 34 UK general practitioners who took part in a previous study evaluating the DSS. They consulted with 12 standardized patients. In addition to suggesting diagnoses, the DSS facilitates data coding. We compared the documentation from consultations with the electronic health record (EHR) (baseline consultations) vs consultations with the EHR-integrated DSS (supported consultations). We measured the proportion of EHR data items related to the physician’s final diagnosis. We expected that in baseline consultations, physicians would document only or predominantly observations related to their diagnosis, while in supported consultations, they would also document other observations as a result of exploring more diagnoses and/or ease of coding.ResultsSupported documentation contained significantly more codes (incidence rate ratio [IRR] = 5.76 [4.31, 7.70] P < .001) and less free text (IRR = 0.32 [0.27, 0.40] P < .001) than baseline documentation. As expected, the proportion of diagnosis-related data was significantly lower (b = −0.08 [−0.11, −0.05] P < .001) in the supported consultations, and this was the case for both codes and free text.ConclusionsWe provide evidence that data entry in the EHR is incomplete and reflects physicians’ cognitive biases. This has serious implications for epidemiological research that uses routine data. A DSS that facilitates and motivates data entry during the consultation can improve routine documentation.     

Optimizing gender-affirming medical care through anatomical inventories,clinical decision support,and population health management in electronic health record systems

Recent advances in electronic health records and health information technology are providing new opportunities to improve the quality of care for transgender and gender diverse people, a population that experiences significant health disparities. This article recommends changes to electronic health record systems that have the potential to optimize gender-affirming care. Specifically, we discuss the importance of creating an anatomical inventory form that captures organ diversity, and of developing clinical decision support tools and population health management systems that consider each patient’s gender identity, sex assigned at birth, and anatomy.     

OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases

ObjectiveTo assess usability and usefulness of a machine learning-based order recommender system applied to simulated clinical cases.Materials and Methods43 physicians entered orders for 5 simulated clinical cases using a clinical order entry interface with or without access to a previously developed automated order recommender system. Cases were randomly allocated to the recommender system in a 3:2 ratio. A panel of clinicians scored whether the orders placed were clinically appropriate. Our primary outcome included the difference in clinical appropriateness scores. Secondary outcomes included total number of orders, case time, and survey responses.ResultsClinical appropriateness scores per order were comparable for cases randomized to the order recommender system (mean difference -0.11 order per score, 95% CI: [-0.41, 0.20]). Physicians using the recommender placed more orders (median 16 vs 15 orders, incidence rate ratio 1.09, 95%CI: [1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Physicians used recommender suggestions in 98% of available cases. Approximately 95% of participants agreed the system would be useful for their workflows.DiscussionUser testing with a simulated electronic medical record interface can assess the value of machine learning and clinical decision support tools for clinician usability and acceptance before live deployments.ConclusionsClinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface in a simulated setting. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.     

Standards for reporting randomized controlled trials in medical informatics: a systematic review of CONSORT adherence in RCTs on clinical decision support

Introduction

The Consolidated Standards for Reporting Trials (CONSORT) were published to standardize reporting and improve the quality of clinical trials. The objective of this study is to assess CONSORT adherence in randomized clinical trials (RCT) of disease specific clinical decision support (CDS).

Methods

A systematic search was conducted of the Medline, EMBASE, and Cochrane databases. RCTs on CDS were assessed against CONSORT guidelines and the Jadad score.

Result

32 of 3784 papers identified in the primary search were included in the final review. 181 702 patients and 7315 physicians participated in the selected trials. Most trials were performed in primary care (22), including 897 general practitioner offices. RCTs assessing CDS for asthma (4), diabetes (4), and hyperlipidemia (3) were the most common. Thirteen CDS systems (40%) were implemented in electronic medical records, and 14 (43%) provided automatic alerts. CONSORT and Jadad scores were generally low; the mean CONSORT score was 30.75 (95% CI 27.0 to 34.5), median score 32, range 21–38. Fourteen trials (43%) did not clearly define the study objective, and 11 studies (34%) did not include a sample size calculation. Outcome measures were adequately identified and defined in 23 (71%) trials; adverse events or side effects were not reported in 20 trials (62%). Thirteen trials (40%) were of superior quality according to the Jadad score (≥3 points). Six trials (18%) reported on long-term implementation of CDS.

Conclusion

The overall quality of reporting RCTs was low. There is a need to develop standards for reporting RCTs in medical informatics.     

Data from clinical notes: a perspective on the tension between structure and flexible documentation

Clinical documentation is central to patient care. The success of electronic health record system adoption may depend on how well such systems support clinical documentation. A major goal of integrating clinical documentation into electronic heath record systems is to generate reusable data. As a result, there has been an emphasis on deploying computer-based documentation systems that prioritize direct structured documentation. Research has demonstrated that healthcare providers value different factors when writing clinical notes, such as narrative expressivity, amenability to the existing workflow, and usability. The authors explore the tension between expressivity and structured clinical documentation, review methods for obtaining reusable data from clinical notes, and recommend that healthcare providers be able to choose how to document patient care based on workflow and note content needs. When reusable data are needed from notes, providers can use structured documentation or rely on post-hoc text processing to produce structured data, as appropriate.     

Clinical decision support malfunctions related to medication routes: a case series

ObjectiveTo identify common medication route-related causes of clinical decision support (CDS) malfunctions and best practices for avoiding them.Materials and MethodsCase series of medication route-related CDS malfunctions from diverse healthcare provider organizations.ResultsNine cases were identified and described, including both false-positive and false-negative alert scenarios. A common cause was the inclusion of nonsystemically available medication routes in value sets (eg, eye drops, ear drops, or topical preparations) when only systemically available routes were appropriate.DiscussionThese value set errors are common, occur across healthcare provider organizations and electronic health record (EHR) systems, affect many different types of medications, and can impact the accuracy of CDS interventions. New knowledge management tools and processes for auditing existing value sets and supporting the creation of new value sets can mitigate many of these issues. Furthermore, value set issues can adversely affect other aspects of the EHR, such as quality reporting and population health management.ConclusionValue set issues related to medication routes are widespread and can lead to CDS malfunctions. Organizations should make appropriate investments in knowledge management tools and strategies, such as those outlined in our recommendations.     

基于结构化电子病历的中医临床路径设计与探索

临床路径的成功实施在保证医疗质量、控制医疗费用等方面发挥了重要作用。如何保证临床路径管理的顺利实施,以结构化电子病历为核心的医疗信息系统建设发挥了重要作用,为临床路径的制定、执行和评估与优化的全过程管理提供了数据保障。从数据特征角度阐述了基于结构化电子病历的临床路径电子化管理的可行性,为临床路径与医院信息系统的整合提供了建议与方法。     

Developing machine learning models to personalize care levels among emergency room patients for hospital admission

ObjectiveTo develop prediction models for intensive care unit (ICU) vs non-ICU level-of-care need within 24 hours of inpatient admission for emergency department (ED) patients using electronic health record data.Materials and MethodsUsing records of 41 654 ED visits to a tertiary academic center from 2015 to 2019, we tested 4 algorithms—feed-forward neural networks, regularized regression, random forests, and gradient-boosted trees—to predict ICU vs non-ICU level-of-care within 24 hours and at the 24th hour following admission. Simple-feature models included patient demographics, Emergency Severity Index (ESI), and vital sign summary. Complex-feature models added all vital signs, lab results, and counts of diagnosis, imaging, procedures, medications, and lab orders.ResultsThe best-performing model, a gradient-boosted tree using a full feature set, achieved an AUROC of 0.88 (95%CI: 0.87–0.89) and AUPRC of 0.65 (95%CI: 0.63–0.68) for predicting ICU care need within 24 hours of admission. The logistic regression model using ESI achieved an AUROC of 0.67 (95%CI: 0.65–0.70) and AUPRC of 0.37 (95%CI: 0.35–0.40). Using a discrimination threshold, such as 0.6, the positive predictive value, negative predictive value, sensitivity, and specificity were 85%, 89%, 30%, and 99%, respectively. Vital signs were the most important predictors.Discussion and ConclusionsUndertriaging admitted ED patients who subsequently require ICU care is common and associated with poorer outcomes. Machine learning models using readily available electronic health record data predict subsequent need for ICU admission with good discrimination, substantially better than the benchmarking ESI system. The results could be used in a multitiered clinical decision-support system to improve ED triage.     

MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care

The rise of digital data and computing power have contributed to significant advancements in artificial intelligence (AI), leading to the use of classification and prediction models in health care to enhance clinical decision-making for diagnosis, treatment and prognosis. However, such advances are limited by the lack of reporting standards for the data used to develop those models, the model architecture, and the model evaluation and validation processes. Here, we present MINIMAR (MINimum Information for Medical AI Reporting), a proposal describing the minimum information necessary to understand intended predictions, target populations, and hidden biases, and the ability to generalize these emerging technologies. We call for a standard to accurately and responsibly report on AI in health care. This will facilitate the design and implementation of these models and promote the development and use of associated clinical decision support tools, as well as manage concerns regarding accuracy and bias.     

CSER and eMERGE: current and potential state of the display of genetic information in the electronic health record

Objective Clinicians’ ability to use and interpret genetic information depends upon how those data are displayed in electronic health records (EHRs). There is a critical need to develop systems to effectively display genetic information in EHRs and augment clinical decision support (CDS).Materials and Methods The National Institutes of Health (NIH)-sponsored Clinical Sequencing Exploratory Research and Electronic Medical Records & Genomics EHR Working Groups conducted a multiphase, iterative process involving working group discussions and 2 surveys in order to determine how genetic and genomic information are currently displayed in EHRs, envision optimal uses for different types of genetic or genomic information, and prioritize areas for EHR improvement.Results There is substantial heterogeneity in how genetic information enters and is documented in EHR systems. Most institutions indicated that genetic information was displayed in multiple locations in their EHRs. Among surveyed institutions, genetic information enters the EHR through multiple laboratory sources and through clinician notes. For laboratory-based data, the source laboratory was the main determinant of the location of genetic information in the EHR. The highest priority recommendation was to address the need to implement CDS mechanisms and content for decision support for medically actionable genetic information.Conclusion Heterogeneity of genetic information flow and importance of source laboratory, rather than clinical content, as a determinant of information representation are major barriers to using genetic information optimally in patient care. Greater effort to develop interoperable systems to receive and consistently display genetic and/or genomic information and alert clinicians to genomic-dependent improvements to clinical care is recommended.     

Enhancing the use of EHR systems for pragmatic embedded research: lessons from the NIH Health Care Systems Research Collaboratory

ObjectiveWe identified challenges and solutions to using electronic health record (EHR) systems for the design and conduct of pragmatic research.Materials and MethodsSince 2012, the Health Care Systems Research Collaboratory has served as the resource coordinating center for 21 pragmatic clinical trial demonstration projects. The EHR Core working group invited these demonstration projects to complete a written semistructured survey and used an inductive approach to review responses and identify EHR-related challenges and suggested EHR enhancements.ResultsWe received survey responses from 20 projects and identified 21 challenges that fell into 6 broad themes: (1) inadequate collection of patient-reported outcome data, (2) lack of structured data collection, (3) data standardization, (4) resources to support customization of EHRs, (5) difficulties aggregating data across sites, and (6) accessing EHR data.DiscussionBased on these findings, we formulated 6 prerequisites for PCTs that would enable the conduct of pragmatic research: (1) integrate the collection of patient-centered data into EHR systems, (2) facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows, (3) support the creation of high-quality research data by using standards, (4) ensure adequate IT staff to support embedded research, (5) create aggregate, multidata type resources for multisite trials, and (6) create re-usable and automated queries.ConclusionWe are hopeful our collection of specific EHR challenges and research needs will drive health system leaders, policymakers, and EHR designers to support these suggestions to improve our national capacity for generating real-world evidence.