Moving Faster than the COVID-19 Pandemic: The Rapid,Digital Transformation of a Public Health System

Background  Queensland, Australia has been successful in containing the COVID-19 pandemic. Underpinning that response has been a highly effective virus containment strategy which relies on identification, isolation, and contact tracing of cases. The dramatic emergence of the COVID-19 pandemic rendered traditional paper-based systems for managing contact tracing no longer fit for purpose. A rapid digital transformation of the public health contact tracing system occurred to support this effort. Objectives  The objectives of the digital transformation were to shift legacy systems (paper or standalone electronic systems) to a digitally enabled public health system, where data are centered around the consumer rather than isolated databases. The objective of this paper is to outline this case study and detail the lessons learnt to inform and give confidence to others contemplating digitization of public health systems in response to the COVID-19 pandemic. Methods  This case study is set in Queensland, Australia. Universal health care is available. A multidisciplinary team was established consisting of clinical informaticians, developers, data strategists, and health information managers. An agile “pair-programming” approach was undertaken to application development and extensive change efforts were made to maximize adoption of the new digital workflows. Data governance and flows were changed to support rapid management of the pandemic. Results  The digital coronavirus application (DCOVA) is a web-based application that securely captures information about people required to quarantine and creates a multiagency secure database to support a successful containment strategy. Conclusion  Most of the literature surrounding digital transformation allows time for significant consultation, which was simply not possible under crisis conditions. Our observation is that staff was willing to adopt new digital systems because the reason for change (the COVID-19 pandemic) was clearly pressing. This case study highlights just how critical a unified purpose, is to successful, rapid digital transformation.     

Clinician and Health Care Leaders' Experiences with—and Perceptions of—COVID-19 Documentation Reduction Policies and Practices

Background  Substantial strategies to reduce clinical documentation were implemented by health care systems throughout the coronavirus disease-2019 (COVID-19) pandemic at national and local levels. This natural experiment provides an opportunity to study the impact of documentation reduction strategies on documentation burden among clinicians and other health professionals in the United States. Objectives  The aim of this study was to assess clinicians'' and other health care leaders'' experiences with and perceptions of COVID-19 documentation reduction strategies and identify which implemented strategies should be prioritized and remain permanent post-pandemic. Methods  We conducted a national survey of clinicians and health care leaders to understand COVID-19 documentation reduction strategies implemented during the pandemic using snowball sampling through professional networks, listservs, and social media. We developed and validated a 19-item survey leveraging existing post-COVID-19 policy and practice recommendations proposed by Sinsky and Linzer. Participants rated reduction strategies for impact on documentation burden on a scale of 0 to 100. Free-text responses were thematically analyzed. Results  Of the 351 surveys initiated, 193 (55%) were complete. Most participants were informaticians and/or clinicians and worked for a health system or in academia. A majority experienced telehealth expansion (81.9%) during the pandemic, which participants also rated as highly impactful (60.1–61.5) and preferred that it remain (90.5%). Implemented at lower proportions, documenting only pertinent positives to reduce note bloat (66.1 ± 28.3), c hanging compliance rules and performance metrics to eliminate those without evidence of net benefit (65.7 ± 26.3), and electronic health record (EHR) optimization sprints (64.3 ± 26.9) received the highest impact scores compared with other strategies presented; support for these strategies widely ranged (49.7–63.7%). Conclusion  The results of this survey suggest there are many perceived sources of and solutions for documentation burden. Within strategies, we found considerable support for telehealth, documenting pertinent positives, and changing compliance rules. We also found substantial variation in the experience of documentation burden among participants.     

Building a Learning Health System: Creating an Analytical Workflow for Evidence Generation to Inform Institutional Clinical Care Guidelines

Background  One key aspect of a learning health system (LHS) is utilizing data generated during care delivery to inform clinical care. However, institutional guidelines that utilize observational data are rare and require months to create, making current processes impractical for more urgent scenarios such as those posed by the COVID-19 pandemic. There exists a need to rapidly analyze institutional data to drive guideline creation where evidence from randomized control trials are unavailable. Objectives  This article provides a background on the current state of observational data generation in institutional guideline creation and details our institution''s experience in creating a novel workflow to (1) demonstrate the value of such a workflow, (2) demonstrate a real-world example, and (3) discuss difficulties encountered and future directions. Methods  Utilizing a multidisciplinary team of database specialists, clinicians, and informaticists, we created a workflow for identifying and translating a clinical need into a queryable format in our clinical data warehouse, creating data summaries and feeding this information back into clinical guideline creation. Results  Clinical questions posed by the hospital medicine division were answered in a rapid time frame and informed creation of institutional guidelines for the care of patients with COVID-19. The cost of setting up a workflow, answering the questions, and producing data summaries required around 300 hours of effort and $300,000 USD. Conclusion  A key component of an LHS is the ability to learn from data generated during care delivery. There are rare examples in the literature and we demonstrate one such example along with proposed thoughts of ideal multidisciplinary team formation and deployment.     

Rapid Implementation of a Complex,Multimodal Technology Response to COVID-19 at an Integrated Community-Based Health Care System

Background  The rapid spread of severe acute respiratory syndrome coronavirus-2 or SARS-CoV-2 necessitated a scaled treatment response to the novel coronavirus disease 2019 (COVID-19). Objective  This study aimed to characterize the design and rapid implementation of a complex, multimodal, technology response to COVID-19 led by the Intermountain Healthcare''s (Intermountain''s) Care Transformation Information Systems (CTIS) organization to build pandemic surge capacity. Methods  Intermountain has active community-spread cases of COVID-19 that are increasing. We used the Centers for Disease Control and Prevention Pandemic Intervals Framework (the Framework) to characterize CTIS leadership''s multimodal technology response to COVID-19 at Intermountain. We provide results on implementation feasibility and sustainability of health information technology (HIT) interventions as of June 30, 2020, characterize lessons learned and identify persistent barriers to sustained deployment. Results  We characterize the CTIS organization''s multimodal technology response to COVID-19 in five relevant areas of the Framework enabling (1) incident management, (2) surveillance, (3) laboratory testing, (4) community mitigation, and (5) medical care and countermeasures. We are seeing increased use of traditionally slow-to-adopt technologies that create additional surge capacity while sustaining patient safety and care quality. CTIS leadership recognized early that a multimodal technology intervention could enable additional surge capacity for health care delivery systems with a broad geographic and service scope. A statewide central tracking system to coordinate capacity planning and management response is needed. Order interoperability between health care systems remains a barrier to an integrated response. Conclusion  The rate of future pandemics is estimated to increase. The pandemic response of health care systems, like Intermountain, offers a blueprint for the leadership role that HIT organizations can play in mainstream care delivery, enabling a nimbler, virtual health care delivery system that is more responsive to current and future needs.     

Developing an mHealth Intervention to Reduce COVID-19–Associated Psychological Distress Among Health Care Workers in Nigeria: Protocol for a Design and Feasibility Study

BackgroundGlobally, COVID-19–related psychological distress is seriously eroding health care workers’ mental health and well-being, especially in low-income countries like Nigeria. The use of mobile health (mHealth) interventions is now increasingly recognized as an innovative approach that may improve mental health and well-being. This project aims to develop an mHealth psychological intervention (mPsyI) to reduce COVID-19–related psychological distress among health care workers in Nigeria.ObjectiveOur objective is to present a study protocol to determine the level of COVID-19–related psychological distress among health care workers in Nigeria; explore health care workers’ experience of COVID-19–related psychological distress; develop and pilot test mPsyI to reduce this distress; and assess the feasibility of this intervention (such as usability, engagement, and satisfaction).MethodsA mixed (quantitative and qualitative) methods approach is used in which health care workers will be recruited from 2 tertiary health care facilities in southwest Nigeria. The study is divided into 4 phases based on the study objectives. Phase 1 involves a quantitative survey to assess the type and levels of psychosocial distress. Phase 2 collects qualitative data on psychosocial distress among health care workers. Phase 3 involves development of the mHealth-based psychological intervention, and phase 4 is a mixed methods study to assess the feasibility and acceptability of the intervention.ResultsThis study was funded in November 2020 by the Global Effort on COVID-19 Health Research, and collection of preliminary baseline data started in July 2021.ConclusionsThis is the first study to report the development of an mHealth-based intervention to reduce COVID-19–related psychological distress among health care workers in Nigeria. Using a mixed methods design in this study can potentially facilitate the adaptation of an evidence-based treatment method that is culturally sensitive and cost-effective for the management of COVID-19–related psychological distress among health care workers in Nigeria.International Registered Report Identifier (IRRID)DERR1-10.2196/36174     

Tracking Hospital Visitors/Chaperones during the COVID-19 Pandemic

Objective  The coronavirus disease (COVID-19) is an emerging infectious disease with strong infectious power and fatality rate. To protect national health, government agencies have regulations on hospital chaperoning and visiting. This article presents the development and implementation of a monitoring system for hospital visiting and chaperoning during the COVID-19 pandemic. The study aimed to create a hospital visiting and chaperoning monitor system that uses nation-wide data sources to more accurately screen hospital visitors and chaperones, assist contract tracing, and prevent transmission of severe acute respiratory syndrome coronavirus 2. Methods  This project was implemented in 57 ward units of an academic medical center. The system was connected to the National Health Insurance (NHI) system and Hospital Information System (HIS), and built on the data of everyone who accessed either the hospital or ward using an NHI smart card or national identification card. To shorten the time for manual identification, we also developed a new system of “app for appointment visits and chaperones” to make appointments online. Results  After the implementation of the system, data from visitors and chaperones in the nursing information system could be accessed. Given that all data were registered in the HIS visiting/chaperoning monitor system, an epidemic investigation could be performed whenever there was a confirmed case. Conclusion  Through the establishment of this system, people entering the ward can be accurately controlled, and all the contacts of potential cases can be traced.     

Ambulatory Clinician's Guide to Inpatient Service: An Innovative Rapid Onboarding Strategy for the COVID-19 Pandemic

Background and Significance  When hospitals are subject to prolonged surges in patients, such as during the coronavirus disease 2019 (COVID-19) pandemic, additional clinicians may be needed to care for the rapid increase of acutely ill patients. How might we quickly prepare a large number of ambulatory-based clinicians to care for hospitalized patients using the inpatient workflow of the electronic health record (EHR)? Objectives  The aim of the study is to create a successful training intervention which prepares ambulatory-based clinicians as they transition to inpatient services. Methods  We created a training guide with embedded videos that describes the workflow of an inpatient clinician. We delivered this intervention via an e-mail hyperlink, a static hyperlink inside of the EHR, and an on-demand hyperlink within the EHR. Results  In anticipation of the first peak of inpatients with COVID-19 in April 2020, the training manual was accessed 261 times by 167 unique users as clinicians anticipated being called into service. As our institution has not yet needed to deploy ambulatory-based clinicians for inpatient service, usage data of the training document is still pending. Conclusion  We intend that our novel implementation of a multimedia, highly accessible onboarding document with access from points inside and outside of the EHR will improve clinician performance and serve as a helpful example to other organizations during the COVID-19 pandemic and beyond.     

A Framework for Systematic Assessment of Clinical Trial Population Representativeness Using Electronic Health Records Data

Background  Clinical trials are the gold standard for generating robust medical evidence, but clinical trial results often raise generalizability concerns, which can be attributed to the lack of population representativeness. The electronic health records (EHRs) data are useful for estimating the population representativeness of clinical trial study population. Objectives  This research aims to estimate the population representativeness of clinical trials systematically using EHR data during the early design stage. Methods  We present an end-to-end analytical framework for transforming free-text clinical trial eligibility criteria into executable database queries conformant with the Observational Medical Outcomes Partnership Common Data Model and for systematically quantifying the population representativeness for each clinical trial. Results  We calculated the population representativeness of 782 novel coronavirus disease 2019 (COVID-19) trials and 3,827 type 2 diabetes mellitus (T2DM) trials in the United States respectively using this framework. With the use of overly restrictive eligibility criteria, 85.7% of the COVID-19 trials and 30.1% of T2DM trials had poor population representativeness. Conclusion  This research demonstrates the potential of using the EHR data to assess the clinical trials population representativeness, providing data-driven metrics to inform the selection and optimization of eligibility criteria.     

Disparities in Telemedicine Access: A Cross-Sectional Study of a Newly Established Infrastructure during the COVID-19 Pandemic

Background  The COVID-19 pandemic led to dramatic increases in telemedicine use to provide outpatient care without in-person contact risks. Telemedicine increases options for health care access, but a “digital divide” of disparate access may prevent certain populations from realizing the benefits of telemedicine. Objectives  The study aimed to understand telemedicine utilization patterns after a widespread deployment to identify potential disparities exacerbated by expanded telemedicine usage. Methods  We performed a cross-sectional retrospective analysis of adults who scheduled outpatient visits between June 1, 2020 and August 31, 2020 at a single-integrated academic health system encompassing a broad range of subspecialties and a large geographic region in the Upper Midwest, during a period of time after the initial surge of COVID-19 when most standard clinical services had resumed. At the beginning of this study period, approximately 72% of provider visits were telemedicine visits. The primary study outcome was whether a patient had one or more video-based visits, compared with audio-only (telephone) visits or in-person visits only. The secondary outcome was whether a patient had any telemedicine visits (video-based or audio-only), compared with in-person visits only. Results  A total of 197,076 individuals were eligible (average age = 46 years, 56% females). Increasing age, rural status, Asian or Black/African American race, Hispanic ethnicity, and self-pay/uninsured status were significantly negatively associated with having a video visit. Digital literacy, measured by patient portal activation status, was significantly positively associated with having a video visit, as were Medicaid or Medicare as payer and American Indian/Alaskan Native race. Conclusion  Our findings reinforce previous evidence that older age, rural status, lower socioeconomic status, Asian race, Black/African American race, and Hispanic/Latino ethnicity are associated with lower rates of video-based telemedicine use. Health systems and policies should seek to mitigate such barriers to telemedicine when possible, with efforts such as digital literacy outreach and equitable distribution of telemedicine infrastructure.     

Extracting Medical Information from Paper COVID-19 Assessment Forms

Objective  This study examines the validity of optical mark recognition, a novel user interface, and crowdsourced data validation to rapidly digitize and extract data from paper COVID-19 assessment forms at a large medical center. Methods  An optical mark recognition/optical character recognition (OMR/OCR) system was developed to identify fields that were selected on 2,814 paper assessment forms, each with 141 fields which were used to assess potential COVID-19 infections. A novel user interface (UI) displayed mirrored forms showing the scanned assessment forms with OMR results superimposed on the left and an editable web form on the right to improve ease of data validation. Crowdsourced participants validated the results of the OMR system. Overall error rate and time taken to validate were calculated. A subset of forms was validated by multiple participants to calculate agreement between participants. Results  The OMR/OCR tools correctly extracted data from scanned forms fields with an average accuracy of 70% and median accuracy of 78% when the OMR/OCR results were compared with the results from crowd validation. Scanned forms were crowd-validated at a mean rate of 157 seconds per document and a volume of approximately 108 documents per day. A randomly selected subset of documents was reviewed by multiple participants, producing an interobserver agreement of 97% for documents when narrative-text fields were included and 98% when only Boolean and multiple-choice fields were considered. Conclusion  Due to the COVID-19 pandemic, it may be challenging for health care workers wearing personal protective equipment to interact with electronic health records. The combination of OMR/OCR technology, a novel UI, and crowdsourcing data-validation processes allowed for the efficient extraction of a large volume of paper medical documents produced during the COVID-19 pandemic.     

Do You Know What Your Scribe Did Last Spring? The Impact of COVID-19 on Medical Scribe Workflow

Objective  To understand the impact of the shift to virtual medicine induced by coronavirus disease 2019 (COVID-19) has had on the workflow of medical scribes. Design  This is a prospective observational survey-based study. Setting  This study was conducted at academic medical center in the United States. Participants  Seventy-four scribes working in ambulatory practices within an academic medical center. Interventions  All medical scribes received a survey assessing their workflow since beginning of COVID-19 restrictions. Primary and Secondary Outcomes  To assess the current workflow of medical scribes since transition to virtual care. Secondary outcomes are to assess the equipment used and location of their new workflow. Results  Fifty-seven scribes completed the survey. Overall 42% of scribes have transitioned to remote scribing with 97% serving as remote scribes for remote visits. This workflow is conducted at home and with personal equipment. Of those not working as scribes, 46% serve in preclinic support, with a wide range of EHR-related activities being reported. The remaining scribes have been either redeployed or furloughed. Conclusion  The rapid transition to virtual care brought about by COVID-19 has resulted in a dramatic shift in scribe workflow with the adoption of a previously unreported workflow of remote scribing for virtual care. Additional work is now needed to ensure these new workflows are safe and effective and that scribes are trained to work in this new paradigm.     

Fast-Tracking Health Data Standards Development and Adoption in Real-World Settings: A Pilot Approach

Background  Pilot-testing is important in standards development because it facilitates agile navigation of the gap between needs for and use of standards in real-world settings and can reveal the practicalities of implementation. As the implementation and use of health data standards are usually more complicated than anticipated, the Office of the National Coordinator for Health Information Technology (ONC) routinely oversees and organizes relevant pilot projects. Objectives  This article provides an in-depth look into a sample of ONC''s standards-focused pilot projects to (1) inform readers of the complexities of developing, implementing, and advancing standards and (2) guide those seeking to evaluate new standards through pilot projects. Methods  The ONC''s approach to conducting pilot projects begins with identifying a clinical care need, research requirement, or policy outcome that is not well supported by existing standards through a landscape review. ONC then selects a testing approach based on the identified need and maturity of relevant standards. Next, ONC identifies use cases and sites to pilot-test the relevant standard. Once complete, ONC publishes a report that informs subsequent projects and standards development. Results  Pilot projects presented here are organized into three categories related to their demonstrated focus and related approach: (1) improving standards for presenting and sharing clinical genetic data, (2) accelerating the development and implementation of new standards, and (3) facilitating clinical data reuse. Each project illustrates the pilot approach from inception to next steps, capturing the role of collaboration among standards development organizations, stakeholders, and end-users to ensure standards are practical and fit for purpose. Conclusion  The ONC approach identifies implementation difficulties prior to broader adoption and use of standards, and provides insight into the steps needed to scale use of standards. The ONC''s organization of pilot projects serves as a natural accelerator for building communities of practice, often providing a well-connected beneficiary of lessons learned.     

Toward a Learning Health Care System: A Systematic Review and Evidence-Based Conceptual Framework for Implementation of Clinical Analytics in a Digital Hospital

Objective  A learning health care system (LHS) uses routinely collected data to continuously monitor and improve health care outcomes. Little is reported on the challenges and methods used to implement the analytics underpinning an LHS. Our aim was to systematically review the literature for reports of real-time clinical analytics implementation in digital hospitals and to use these findings to synthesize a conceptual framework for LHS implementation. Methods  Embase, PubMed, and Web of Science databases were searched for clinical analytics derived from electronic health records in adult inpatient and emergency department settings between 2015 and 2021. Evidence was coded from the final study selection that related to (1) dashboard implementation challenges, (2) methods to overcome implementation challenges, and (3) dashboard assessment and impact. The evidences obtained, together with evidence extracted from relevant prior reviews, were mapped to an existing digital health transformation model to derive a conceptual framework for LHS analytics implementation. Results  A total of 238 candidate articles were reviewed and 14 met inclusion criteria. From the selected studies, we extracted 37 implementation challenges and 64 methods employed to overcome such challenges. We identified common approaches for evaluating the implementation of clinical dashboards. Six studies assessed clinical process outcomes and only four studies evaluated patient health outcomes. A conceptual framework for implementing the analytics of an LHS was developed. Conclusion  Health care organizations face diverse challenges when trying to implement real-time data analytics. These challenges have shifted over the past decade. While prior reviews identified fundamental information problems, such as data size and complexity, our review uncovered more postpilot challenges, such as supporting diverse users, workflows, and user-interface screens. Our review identified practical methods to overcome these challenges which have been incorporated into a conceptual framework. It is hoped this framework will support health care organizations deploying near-real-time clinical dashboards and progress toward an LHS.     

Impact of Changes in EHR Use during COVID-19 on Physician Trainee Mental Health

Objectives  This article investigates the association between changes in electronic health record (EHR) use during the coronavirus disease 2019 (COVID-19) pandemic on the rate of burnout, stress, posttraumatic stress disorder (PTSD), depression, and anxiety among physician trainees (residents and fellows). Methods  A total of 222 (of 1,375, 16.2%) physician trainees from an academic medical center responded to a Web-based survey. We compared the physician trainees who reported that their EHR use increased versus those whose EHR use stayed the same or decreased on outcomes related to depression, anxiety, stress, PTSD, and burnout using univariable and multivariable models. We examined whether self-reported exposure to COVID-19 patients moderated these relationships. Results  Physician trainees who reported increased use of EHR had higher burnout (adjusted mean, 1.48 [95% confidence interval [CI] 1.24, 1.71] vs. 1.05 [95% CI 0.93, 1.17]; p  = 0.001) and were more likely to exhibit symptoms of PTSD (adjusted mean = 15.09 [95% CI 9.12, 21.05] vs. 9.36 [95% CI 7.38, 11.28]; p  = 0.035). Physician trainees reporting increased EHR use outside of work were more likely to experience depression (adjusted mean, 8.37 [95% CI 5.68, 11.05] vs. 5.50 [95% CI 4.28, 6.72]; p  = 0.035). Among physician trainees with increased EHR use, those exposed to COVID-19 patients had significantly higher burnout (2.04, p  < 0.001) and depression scores (14.13, p  = 0.003). Conclusion  Increased EHR use was associated with higher burnout, depression, and PTSD outcomes among physician trainees. Although preliminary, these findings have implications for creating systemic changes to manage the wellness and well-being of trainees.     

Leveraging Real-World Data for the Selection of Relevant Eligibility Criteria for the Implementation of Electronic Recruitment Support in Clinical Trials

Background  Even though clinical trials are indispensable for medical research, they are frequently impaired by delayed or incomplete patient recruitment, resulting in cost overruns or aborted studies. Study protocols based on real-world data with precisely expressed eligibility criteria and realistic cohort estimations are crucial for successful study execution. The increasing availability of routine clinical data in electronic health records (EHRs) provides the opportunity to also support patient recruitment during the prescreening phase. While solutions for electronic recruitment support have been published, to our knowledge, no method for the prioritization of eligibility criteria in this context has been explored. Methods  In the context of the Electronic Health Records for Clinical Research (EHR4CR) project, we examined the eligibility criteria of the KATHERINE trial. Criteria were extracted from the study protocol, deduplicated, and decomposed. A paper chart review and data warehouse query were executed to retrieve clinical data for the resulting set of simplified criteria separately from both sources. Criteria were scored according to disease specificity, data availability, and discriminatory power based on their content and the clinical dataset. Results  The study protocol contained 35 eligibility criteria, which after simplification yielded 70 atomic criteria. For a cohort of 106 patients with breast cancer and neoadjuvant treatment, 47.9% of data elements were captured through paper chart review, with the data warehouse query yielding 26.9% of data elements. Score application resulted in a prioritized subset of 17 criteria, which yielded a sensitivity of 1.00 and specificity 0.57 on EHR data (paper charts, 1.00 and 0.80) compared with actual recruitment in the trial. Conclusion  It is possible to prioritize clinical trial eligibility criteria based on real-world data to optimize prescreening of patients on a selected subset of relevant and available criteria and reduce implementation efforts for recruitment support. The performance could be further improved by increasing EHR data coverage.     

COVID-19 Mass Vaccination Resource Calculator

Background  Despite the recent emergency use authorization of two vaccines for the prevention of the 2019 novel coronavirus (COVID-19) disease, vaccination rates are lower than expected. Vaccination efforts may be hampered by supply, delivery, storage, patient prioritization, administration infrastructure or logistics problems. To address the last issue, our institution is sharing publically a calculator to optimize the management of staffing and facility resources in an outpatient mass vaccination effort. Objective  By sharing our calculator locally and through this paper, we aim to help health organizations administering vaccines optimize resource allocation while maximizing efficiency. Methods  Our calculator determines the maximum number of vaccinations that can be administered per hour, the number of check-in staff (clerks) needed, the number of vaccination staff (nurses) needed, and the required room capacity needed for the vaccination and the mandatory 15-minute observation period after inoculation. Results  We provide a functional version of the calculator, allowing users to replicate the calculation for their own vaccine events. Conclusion  An efficient and organized vaccination program is critical to halting the spread of COVID-19. By sharing this calculator, it is our hope that other organizations may use it to facilitate rapid and efficient vaccination.     

Evaluation of Electronic Health Record-Based Suicide Risk Prediction Models on Contemporary Data

Background  Suicide risk prediction models have been developed by using information from patients'' electronic health records (EHR), but the time elapsed between model development and health system implementation is often substantial. Temporal changes in health systems and EHR coding practices necessitate the evaluation of such models in more contemporary data. Objectives  A set of published suicide risk prediction models developed by using EHR data from 2009 to 2015 across seven health systems reported c-statistics of 0.85 for suicide attempt and 0.83 to 0.86 for suicide death. Our objective was to evaluate these models'' performance with contemporary data (2014–2017) from these systems. Methods  We evaluated performance using mental health visits (6,832,439 to mental health specialty providers and 3,987,078 to general medical providers) from 2014 to 2017 made by 1,799,765 patients aged 13+ across the health systems. No visits in our evaluation were used in the previous model development. Outcomes were suicide attempt (health system records) and suicide death (state death certificates) within 90 days following a visit. We assessed calibration and computed c-statistics with 95% confidence intervals (CI) and cut-point specific estimates of sensitivity, specificity, and positive/negative predictive value. Results  Models were well calibrated; 46% of suicide attempts and 35% of suicide deaths in the mental health specialty sample were preceded by a visit (within 90 days) with a risk score in the top 5%. In the general medical sample, 53% of attempts and 35% of deaths were preceded by such a visit. Among these two samples, respectively, c-statistics were 0.862 (95% CI: 0.860–0.864) and 0.864 (95% CI: 0.860–0.869) for suicide attempt, and 0.806 (95% CI: 0.790–0.822) and 0.804 (95% CI: 0.782–0.829) for suicide death. Conclusion  Performance of the risk prediction models in this contemporary sample was similar to historical estimates for suicide attempt but modestly lower for suicide death. These published models can inform clinical practice and patient care today.