Leveraging Real-World Data for the Selection of Relevant Eligibility Criteria for the Implementation of Electronic Recruitment Support in Clinical Trials

Background  Even though clinical trials are indispensable for medical research, they are frequently impaired by delayed or incomplete patient recruitment, resulting in cost overruns or aborted studies. Study protocols based on real-world data with precisely expressed eligibility criteria and realistic cohort estimations are crucial for successful study execution. The increasing availability of routine clinical data in electronic health records (EHRs) provides the opportunity to also support patient recruitment during the prescreening phase. While solutions for electronic recruitment support have been published, to our knowledge, no method for the prioritization of eligibility criteria in this context has been explored. Methods  In the context of the Electronic Health Records for Clinical Research (EHR4CR) project, we examined the eligibility criteria of the KATHERINE trial. Criteria were extracted from the study protocol, deduplicated, and decomposed. A paper chart review and data warehouse query were executed to retrieve clinical data for the resulting set of simplified criteria separately from both sources. Criteria were scored according to disease specificity, data availability, and discriminatory power based on their content and the clinical dataset. Results  The study protocol contained 35 eligibility criteria, which after simplification yielded 70 atomic criteria. For a cohort of 106 patients with breast cancer and neoadjuvant treatment, 47.9% of data elements were captured through paper chart review, with the data warehouse query yielding 26.9% of data elements. Score application resulted in a prioritized subset of 17 criteria, which yielded a sensitivity of 1.00 and specificity 0.57 on EHR data (paper charts, 1.00 and 0.80) compared with actual recruitment in the trial. Conclusion  It is possible to prioritize clinical trial eligibility criteria based on real-world data to optimize prescreening of patients on a selected subset of relevant and available criteria and reduce implementation efforts for recruitment support. The performance could be further improved by increasing EHR data coverage.     

Ambulatory Clinician's Guide to Inpatient Service: An Innovative Rapid Onboarding Strategy for the COVID-19 Pandemic

Background and Significance  When hospitals are subject to prolonged surges in patients, such as during the coronavirus disease 2019 (COVID-19) pandemic, additional clinicians may be needed to care for the rapid increase of acutely ill patients. How might we quickly prepare a large number of ambulatory-based clinicians to care for hospitalized patients using the inpatient workflow of the electronic health record (EHR)? Objectives  The aim of the study is to create a successful training intervention which prepares ambulatory-based clinicians as they transition to inpatient services. Methods  We created a training guide with embedded videos that describes the workflow of an inpatient clinician. We delivered this intervention via an e-mail hyperlink, a static hyperlink inside of the EHR, and an on-demand hyperlink within the EHR. Results  In anticipation of the first peak of inpatients with COVID-19 in April 2020, the training manual was accessed 261 times by 167 unique users as clinicians anticipated being called into service. As our institution has not yet needed to deploy ambulatory-based clinicians for inpatient service, usage data of the training document is still pending. Conclusion  We intend that our novel implementation of a multimedia, highly accessible onboarding document with access from points inside and outside of the EHR will improve clinician performance and serve as a helpful example to other organizations during the COVID-19 pandemic and beyond.     

Transformation of Electronic Health Records and Questionnaire Data to OMOP CDM: A Feasibility Study Using SG_T2DM Dataset

Background  Diabetes mellitus (DM) is an important public health concern in Singapore and places a massive burden on health care spending. Tackling chronic diseases such as DM requires innovative strategies to integrate patients'' data from diverse sources and use scientific discovery to inform clinical practice that can help better manage the disease. The Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) was chosen as the framework for integrating data with disparate formats. Objective  The study aimed to evaluate the feasibility of converting Singapore based data source, comprising of electronic health records (EHR), cognitive and depression assessment questionnaire data to OMOP CDM standard. Additionally, we also validate whether our OMOP CDM instance is fit for the purpose of research by executing a simple treatment pathways study using Atlas, a graphical user interface tool to conduct analysis on OMOP CDM data as a proof of concept. Methods  We used de-identified EHR, cognitive, and depression assessment questionnaires data from a tertiary care hospital in Singapore to convert it to version 5.3.1 of OMOP CDM standard. We evaluate the OMOP CDM conversion by (1) assessing the mapping coverage (that is the percentage of source terms mapped to OMOP CDM standard); (2) local raw dataset versus CDM dataset analysis; and (3) Implementing Harmonized Intrinsic Data Quality Framework using an open-source R package called Data Quality Dashboard. Results  The content coverage of OMOP CDM vocabularies is more than 90% for clinical data, but only around 11% for questionnaire data. The comparison of characteristics between source and target data returned consistent results and our transformed data did not pass 38 (1.4%) out of 2,622 quality checks. Conclusion  Adoption of OMOP CDM at our site demonstrated that EHR data are feasible for standardization with minimal information loss, whereas challenges remain for standardizing cognitive and depression assessment questionnaire data that requires further work.     

Impact of Changes in EHR Use during COVID-19 on Physician Trainee Mental Health

Objectives  This article investigates the association between changes in electronic health record (EHR) use during the coronavirus disease 2019 (COVID-19) pandemic on the rate of burnout, stress, posttraumatic stress disorder (PTSD), depression, and anxiety among physician trainees (residents and fellows). Methods  A total of 222 (of 1,375, 16.2%) physician trainees from an academic medical center responded to a Web-based survey. We compared the physician trainees who reported that their EHR use increased versus those whose EHR use stayed the same or decreased on outcomes related to depression, anxiety, stress, PTSD, and burnout using univariable and multivariable models. We examined whether self-reported exposure to COVID-19 patients moderated these relationships. Results  Physician trainees who reported increased use of EHR had higher burnout (adjusted mean, 1.48 [95% confidence interval [CI] 1.24, 1.71] vs. 1.05 [95% CI 0.93, 1.17]; p  = 0.001) and were more likely to exhibit symptoms of PTSD (adjusted mean = 15.09 [95% CI 9.12, 21.05] vs. 9.36 [95% CI 7.38, 11.28]; p  = 0.035). Physician trainees reporting increased EHR use outside of work were more likely to experience depression (adjusted mean, 8.37 [95% CI 5.68, 11.05] vs. 5.50 [95% CI 4.28, 6.72]; p  = 0.035). Among physician trainees with increased EHR use, those exposed to COVID-19 patients had significantly higher burnout (2.04, p  < 0.001) and depression scores (14.13, p  = 0.003). Conclusion  Increased EHR use was associated with higher burnout, depression, and PTSD outcomes among physician trainees. Although preliminary, these findings have implications for creating systemic changes to manage the wellness and well-being of trainees.     

An Analysis of Electronic Health Record Work to Manage Asynchronous Clinical Messages among Breast Cancer Care Teams

Objective  Asynchronous messaging is an integral aspect of communication in clinical settings, but imposes additional work and potentially leads to inefficiency. The goal of this study was to describe the time spent using the electronic health record (EHR) to manage asynchronous communication to support breast cancer care coordination. Methods  We analyzed 3 years of audit logs and secure messaging logs from the EHR for care team members involved in breast cancer care at Vanderbilt University Medical Center. To evaluate trends in EHR use, we combined log data into sequences of events that occurred within 15 minutes of any other event by the same employee about the same patient. Results  Our cohort of 9,761 patients were the subject of 430,857 message threads by 7,194 employees over a 3-year period. Breast cancer care team members performed messaging actions in 37.5% of all EHR sessions, averaging 29.8 (standard deviation [SD] = 23.5) messaging sessions per day. Messaging sessions lasted an average of 1.1 (95% confidence interval: 0.99–1.24) minutes longer than nonmessaging sessions. On days when the cancer providers did not otherwise have clinical responsibilities, they still performed messaging actions in an average of 15 (SD = 11.9) sessions per day. Conclusion  At our institution, clinical messaging occurred in 35% of all EHR sessions. Clinical messaging, sometimes viewed as a supporting task of clinical work, is important to delivering and coordinating care across roles. Measuring the electronic work of asynchronous communication among care team members affords the opportunity to systematically identify opportunities to improve employee workload.     

Building a Learning Health System: Creating an Analytical Workflow for Evidence Generation to Inform Institutional Clinical Care Guidelines

Background  One key aspect of a learning health system (LHS) is utilizing data generated during care delivery to inform clinical care. However, institutional guidelines that utilize observational data are rare and require months to create, making current processes impractical for more urgent scenarios such as those posed by the COVID-19 pandemic. There exists a need to rapidly analyze institutional data to drive guideline creation where evidence from randomized control trials are unavailable. Objectives  This article provides a background on the current state of observational data generation in institutional guideline creation and details our institution''s experience in creating a novel workflow to (1) demonstrate the value of such a workflow, (2) demonstrate a real-world example, and (3) discuss difficulties encountered and future directions. Methods  Utilizing a multidisciplinary team of database specialists, clinicians, and informaticists, we created a workflow for identifying and translating a clinical need into a queryable format in our clinical data warehouse, creating data summaries and feeding this information back into clinical guideline creation. Results  Clinical questions posed by the hospital medicine division were answered in a rapid time frame and informed creation of institutional guidelines for the care of patients with COVID-19. The cost of setting up a workflow, answering the questions, and producing data summaries required around 300 hours of effort and $300,000 USD. Conclusion  A key component of an LHS is the ability to learn from data generated during care delivery. There are rare examples in the literature and we demonstrate one such example along with proposed thoughts of ideal multidisciplinary team formation and deployment.     

Measuring Electronic Health Record Use in the Pediatric ICU Using Audit-Logs and Screen Recordings

Background  Time spent in the electronic health record (EHR) has been identified as an important unit of measure for health care provider clinical activity. The lack of validation of audit-log based inpatient EHR time may have resulted in underuse of this data in studies focusing on inpatient patient outcomes, provider efficiency, provider satisfaction, etc. This has also led to a dearth of clinically relevant EHR usage metrics consistent with inpatient provider clinical activity. Objective  The aim of our study was to validate audit-log based EHR times using observed EHR-times extracted from screen recordings of EHR usage in the inpatient setting. Methods  This study was conducted in a 36-bed pediatric intensive care unit (PICU) at Lucile Packard Children''s Hospital Stanford between June 11 and July 14, 2020. Attending physicians, fellow physicians, hospitalists, and advanced practice providers with ≥0.5 full-time equivalent (FTE) for the prior four consecutive weeks and at least one EHR session recording were included in the study. Citrix session recording player was used to retrospectively review EHR session recordings that were captured as the provider interacted with the EHR. Results  EHR use patterns varied by provider type. Audit-log based total EHR time correlated strongly with both observed total EHR time ( r  = 0.98, p  < 0.001) and observed active EHR time ( r  = 0.95, p  < 0.001). Each minute of audit-log based total EHR time corresponded to 0.95 (0.87–1.02) minutes of observed total EHR time and 0.75 (0.67–0.83) minutes of observed active EHR time. Results were similar when stratified by provider role. Conclusion  Our study found inpatient audit-log based EHR time to correlate strongly with observed EHR time among pediatric critical care providers. These findings support the use of audit-log based EHR-time as a surrogate measure for inpatient provider EHR use, providing an opportunity for researchers and other stakeholders to leverage EHR audit-log data in measuring clinical activity and tracking outcomes of workflow improvement efforts longitudinally and across provider groups.     

Opportunities and Challenges of Telehealth in Disease Management during COVID-19 Pandemic: A Scoping Review

Objective  The novel coronavirus disease 2019 (COVID-19) pandemic is an unexpected universal problem that has changed health care access across the world. Telehealth is an effective solution for health care delivery during disasters and public health emergencies. This study was conducted to summarize the opportunities and challenges of using telehealth in health care delivery during the COVID-19 pandemic. Methods  A structured search was performed in the Web of Science, PubMed, Science Direct, and Scopus databases, as well as the Google Scholar search engine, for studies published until November 4, 2020. The reviewers analyzed 112 studies and identified opportunities and challenges. This review followed the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) protocols. Quality appraisal was done according to the Mixed Methods Appraisal Tool (MMAT) version 2018. Thematic analysis was applied for data analysis. Results  A total of 112 unique opportunities of telehealth application during the pandemic were categorized into 4 key themes, such as (1) clinical, (2) organizational, (3) technical, and (4) social, which were further divided into 11 initial themes and 26 unique concepts. Furthermore, 106 unique challenges were categorized into 6 key themes, such as (1) legal, (2) clinical, (3) organizational, (40 technical, (5) socioeconomic, and (6) data quality, which were divided into 16 initial themes and 37 unique concepts altogether. The clinical opportunities and legal challenges were the most frequent opportunities and challenges, respectively. Conclusion  The COVID-19 pandemic significantly accelerated the use of telehealth. This study could offer useful information to policymakers about the opportunities and challenges of implementing telehealth for providing accessible, safe, and efficient health care delivery to the patient population during and after COVID-19. Furthermore, it can assist policymakers to make informed decisions on implementing telehealth in response to the COVID-19 pandemic by addressing the obstacles ahead.     

Evaluating the Effect of a COVID-19 Predictive Model to Facilitate Discharge: A Randomized Controlled Trial

Background  We previously developed and validated a predictive model to help clinicians identify hospitalized adults with coronavirus disease 2019 (COVID-19) who may be ready for discharge given their low risk of adverse events. Whether this algorithm can prompt more timely discharge for stable patients in practice is unknown. Objectives  The aim of the study is to estimate the effect of displaying risk scores on length of stay (LOS). Methods  We integrated model output into the electronic health record (EHR) at four hospitals in one health system by displaying a green/orange/red score indicating low/moderate/high-risk in a patient list column and a larger COVID-19 summary report visible for each patient. Display of the score was pseudo-randomized 1:1 into intervention and control arms using a patient identifier passed to the model execution code. Intervention effect was assessed by comparing LOS between intervention and control groups. Adverse safety outcomes of death, hospice, and re-presentation were tested separately and as a composite indicator. We tracked adoption and sustained use through daily counts of score displays. Results  Enrolling 1,010 patients from May 15, 2020 to December 7, 2020, the trial found no detectable difference in LOS. The intervention had no impact on safety indicators of death, hospice or re-presentation after discharge. The scores were displayed consistently throughout the study period but the study lacks a causally linked process measure of provider actions based on the score. Secondary analysis revealed complex dynamics in LOS temporally, by primary symptom, and hospital location. Conclusion  An AI-based COVID-19 risk score displayed passively to clinicians during routine care of hospitalized adults with COVID-19 was safe but had no detectable impact on LOS. Health technology challenges such as insufficient adoption, nonuniform use, and provider trust compounded with temporal factors of the COVID-19 pandemic may have contributed to the null result. Trial registration  ClinicalTrials.gov identifier: {"type":"clinical-trial","attrs":{"text":"NCT04570488","term_id":"NCT04570488"}}NCT04570488.     

Early Detection of Prediabetes and T2DM Using Wearable Sensors and Internet-of-Things-Based Monitoring Applications

Background  Prediabetes and type 2 diabetes mellitus (T2DM) are one of the major long-term health conditions affecting global healthcare delivery. One of the few effective approaches is to actively manage diabetes via a healthy and active lifestyle. Objectives  This research is focused on early detection of prediabetes and T2DM using wearable technology and Internet-of-Things-based monitoring applications. Methods  We developed an artificial intelligence model based on adaptive neuro-fuzzy inference to detect prediabetes and T2DM via individualized monitoring. The key contributing factors to the proposed model include heart rate, heart rate variability, breathing rate, breathing volume, and activity data (steps, cadence, and calories). The data was collected using an advanced wearable body vest and combined with manual recordings of blood glucose, height, weight, age, and sex. The model analyzed the data alongside a clinical knowledgebase. Fuzzy rules were used to establish baseline values via existing interventions, clinical guidelines, and protocols. Results  The proposed model was tested and validated using Kappa analysis and achieved an overall agreement of 91%. Conclusion  We also present a 2-year follow-up observation from the prediction results of the original model. Moreover, the diabetic profile of a participant using M-health applications and a wearable vest (smart shirt) improved when compared to the traditional/routine practice.     

Defining Data Migration Across Multidisciplinary Ambulatory Clinics Using Participatory Design

Objective  This study aimed to develop an institutional approach for defining data migration based on participatory design principles. Methods  We outline a collaborative approach to define data migration as part of an electronic health record (EHR) transition at an urban hospital with 20 ambulatory clinics, based on participatory design. We developed an institution-specific list of data for migration based on physician end-user feedback. In this paper, we review the project planning phases, multidisciplinary governance, and methods used. Results  Detailed data migration feedback was obtained from 90% of participants. Depending on the specialty, requests for historical laboratory values ranged from 2 to as many as 145 unique laboratory types. Lookback periods requested by physicians varied and were ultimately assigned to provide the most clinical data. This clinical information was then combined to synthesize an overall proposed data migration request on behalf of the institution. Conclusion  Institutions undergoing an EHR transition should actively involve physician end-users and key stakeholders. Physician feedback is vital for developing a clinically relevant EHR environment but is often difficult to obtain. Challenges include physician time constraints and overall knowledge about health information technology. This study demonstrates how a participatory design can serve to improve the clinical end-user''s understanding of the technical aspects of an EHR implementation, as well as enhance the outcomes of such projects.     

Clinician and Health Care Leaders' Experiences with—and Perceptions of—COVID-19 Documentation Reduction Policies and Practices

Background  Substantial strategies to reduce clinical documentation were implemented by health care systems throughout the coronavirus disease-2019 (COVID-19) pandemic at national and local levels. This natural experiment provides an opportunity to study the impact of documentation reduction strategies on documentation burden among clinicians and other health professionals in the United States. Objectives  The aim of this study was to assess clinicians'' and other health care leaders'' experiences with and perceptions of COVID-19 documentation reduction strategies and identify which implemented strategies should be prioritized and remain permanent post-pandemic. Methods  We conducted a national survey of clinicians and health care leaders to understand COVID-19 documentation reduction strategies implemented during the pandemic using snowball sampling through professional networks, listservs, and social media. We developed and validated a 19-item survey leveraging existing post-COVID-19 policy and practice recommendations proposed by Sinsky and Linzer. Participants rated reduction strategies for impact on documentation burden on a scale of 0 to 100. Free-text responses were thematically analyzed. Results  Of the 351 surveys initiated, 193 (55%) were complete. Most participants were informaticians and/or clinicians and worked for a health system or in academia. A majority experienced telehealth expansion (81.9%) during the pandemic, which participants also rated as highly impactful (60.1–61.5) and preferred that it remain (90.5%). Implemented at lower proportions, documenting only pertinent positives to reduce note bloat (66.1 ± 28.3), c hanging compliance rules and performance metrics to eliminate those without evidence of net benefit (65.7 ± 26.3), and electronic health record (EHR) optimization sprints (64.3 ± 26.9) received the highest impact scores compared with other strategies presented; support for these strategies widely ranged (49.7–63.7%). Conclusion  The results of this survey suggest there are many perceived sources of and solutions for documentation burden. Within strategies, we found considerable support for telehealth, documenting pertinent positives, and changing compliance rules. We also found substantial variation in the experience of documentation burden among participants.     

Interaction Time with Electronic Health Records: A Systematic Review

Background  The amount of time that health care clinicians (physicians and nurses) spend interacting with the electronic health record is not well understood. Objective  This study aimed to evaluate the time that health care providers spend interacting with electronic health records (EHR). Methods  Data are retrieved from Ovid MEDLINE(R) and Epub Ahead of Print, In-Process and Other Non-Indexed Citations and Daily, (Ovid) Embase, CINAHL, and SCOPUS. Study Eligibility Criteria  Peer-reviewed studies that describe the use of EHR and include measurement of time either in hours, minutes, or in the percentage of a clinician''s workday. Papers were written in English and published between 1990 and 2021. Participants  All physicians and nurses involved in inpatient and outpatient settings. Study Appraisal and Synthesis Methods  A narrative synthesis of the results, providing summaries of interaction time with EHR. The studies were rated according to Quality Assessment Tool for Studies with Diverse Designs. Results  Out of 5,133 de-duplicated references identified through database searching, 18 met inclusion criteria. Most were time-motion studies (50%) that followed by logged-based analysis (44%). Most were conducted in the United States (94%) and examined a clinician workflow in the inpatient settings (83%). The average time was nearly 37% of time of their workday by physicians in both inpatient and outpatient settings and 22% of the workday by nurses in inpatient settings. The studies showed methodological heterogeneity. Conclusion  This systematic review evaluates the time that health care providers spend interacting with EHR. Interaction time with EHR varies depending on clinicians'' roles and clinical settings, computer systems, and users'' experience. The average time spent by physicians on EHR exceeded one-third of their workday. The finding is a possible indicator that the EHR has room for usability, functionality improvement, and workflow optimization.     

Searching the PDF Haystack: Automated Knowledge Discovery in Scanned EHR Documents

Background  Clinicians express concern that they may be unaware of important information contained in voluminous scanned and other outside documents contained in electronic health records (EHRs). An example is “unrecognized EHR risk factor information,” defined as risk factors for heritable cancer that exist within a patient''s EHR but are not known by current treating providers. In a related study using manual EHR chart review, we found that half of the women whose EHR contained risk factor information meet criteria for further genetic risk evaluation for heritable forms of breast and ovarian cancer. They were not referred for genetic counseling. Objectives  The purpose of this study was to compare the use of automated methods (optical character recognition with natural language processing) versus human review in their ability to identify risk factors for heritable breast and ovarian cancer within EHR scanned documents. Methods  We evaluated the accuracy of the chart review by comparing our criterion standard (physician chart review) versus an automated method involving Amazon''s Textract service (Amazon.com, Seattle, Washington, United States), a clinical language annotation modeling and processing toolkit (CLAMP) (Center for Computational Biomedicine at The University of Texas Health Science, Houston, Texas, United States), and a custom-written Java application. Results  We found that automated methods identified most cancer risk factor information that would otherwise require clinician manual review and therefore is at risk of being missed. Conclusion  The use of automated methods for identification of heritable risk factors within EHRs may provide an accurate yet rapid review of patients'' past medical histories. These methods could be further strengthened via improved analysis of handwritten notes, tables, and colloquial phrases.     

Patient Cohort Identification on Time Series Data Using the OMOP Common Data Model

Background  The identification of patient cohorts for recruiting patients into clinical trials requires an evaluation of study-specific inclusion and exclusion criteria. These criteria are specified depending on corresponding clinical facts. Some of these facts may not be present in the clinical source systems and need to be calculated either in advance or at cohort query runtime (so-called feasibility query). Objectives  We use the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) as the repository for our clinical data. However, Atlas, the graphical user interface of OMOP, does not offer the functionality to perform calculations on facts data. Therefore, we were in search for a different approach. The objective of this study is to investigate whether the Arden Syntax can be used for feasibility queries on the OMOP CDM to enable on-the-fly calculations at query runtime, to eliminate the need to precalculate data elements that are involved with researchers'' criteria specification. Methods  We implemented a service that reads the facts from the OMOP repository and provides it in a form which an Arden Syntax Medical Logic Module (MLM) can process. Then, we implemented an MLM that applies the eligibility criteria to every patient data set and outputs the list of eligible cases (i.e., performs the feasibility query). Results  The study resulted in an MLM-based feasibility query that identifies cases of overventilation as an example of how an on-the-fly calculation can be realized. The algorithm is split into two MLMs to provide the reusability of the approach. Conclusion  We found that MLMs are a suitable technology for feasibility queries on the OMOP CDM. Our method of performing on-the-fly calculations can be employed with any OMOP instance and without touching existing infrastructure like the Extract, Transform and Load pipeline. Therefore, we think that it is a well-suited method to perform on-the-fly calculations on OMOP.     

Shared-Task Worklists Improve Clinical Trial Recruitment Workflow in an Academic Emergency Department

Background  Clinical trials performed in our emergency department at Barnes-Jewish Hospital utilize a centralized infrastructure for alerting, screening, and enrollment with rule-based alerts sent to clinical research coordinators. Previously, all alerts were delivered as text messages via dedicated cellular phones. As the number of ongoing clinical trials increased, the volume of alerts grew to an unmanageable level. Therefore, we have changed our primary notification delivery method to study-specific, shared-task worklists integrated with our pre-existing web-based screening documentation system. Objective  To evaluate the effects on screening and recruitment workflow of replacing text-message delivery of clinical trial alerts with study-specific shared-task worklists in a high-volume academic emergency department supporting multiple concurrent clinical trials. Methods  We analyzed retrospective data on alerting, screening, and enrollment for 10 active clinical trials pre- and postimplementation of shared-task worklists. Results  Notifications signaling the presence of potentially eligible subjects for clinical trials were more likely to result in a screen ( p  < 0.001) with the implementation of shared-task worklists compared with notifications delivered as text messages for 8/10 clinical trials. The change in workflow did not alter the likelihood of a notification resulting in an enrollment ( p  = 0.473). The Director of Research reported a substantial reduction in the amount of time spent redirecting clinical research coordinator screening activities. Conclusion  Shared-task worklists, with the functionalities we have described, offer a viable alternative to delivery of clinical trial alerts via text message directly to clinical research coordinators recruiting for multiple concurrent clinical trials in a high-volume academic emergency department.     

Improving COVID-19 Research of University Hospitals in Germany: Formative Usability Evaluation of the CODEX Feasibility Portal

Background  Within the German “Network University Medicine,” a portal is to be developed to enable researchers to query on novel coronavirus disease 2019 (COVID-19) data from university hospitals for assessing the feasibility of a clinical study. Objectives  The usability of a prototype for federated feasibility queries was evaluated to identify design strengths and weaknesses and derive improvement recommendations for further development. Methods  In the course of a remote usability test with the thinking-aloud method and posttask interviews, 15 clinical researchers evaluated the usability of a prototype of the Feasibility Portal. The identified usability problems were rated according to severity, and improvement recommendations were derived. Results  The design of the prototype was rated as simple, intuitive, and as usable with little effort. The usability test reported a total of 26 problems, 8 of these were rated as “critical.” Usability problems and revision recommendations focus primarily on improving the visual distinguishability of selected inclusion and exclusion criteria, enabling a flexible approach to criteria linking, and enhancing the free-text search. Conclusion  Improvement proposals were developed for these user problems which will guide further development and the adaptation of the portal to user needs. This is an important prerequisite for correct and efficient use in everyday clinical work in the future. Results can provide developers of similar systems with a good starting point for interface conceptualizations. The methodological approach/the developed test guideline can serve as a template for similar evaluations.     

Health Intelligence Atlas: A Core Tool for Public Health Intelligence

Background  The dramatic increase in complexity and volume of health data has challenged traditional health systems to deliver useful information to their users. The novel coronavirus disease 2019 (COVID-19) pandemic has further exacerbated this problem and demonstrated the critical need for the 21st century approach. This approach needs to ingest relevant, diverse data sources, analyze them, and generate appropriate health intelligence products that enable users to take more effective and efficient actions for their specific challenges. Objectives  This article characterizes the Health Intelligence Atlas (HI-Atlas) development and implementation to produce Public Health Intelligence (PHI) that supports identifying and prioritizing high-risk communities by public health authorities. The HI-Atlas moves from post hoc observations to a proactive model-based approach for preplanning COVID-19 vaccine preparedness, distribution, and assessing the effectiveness of those plans. Results  Details are presented on how the HI-Atlas merged traditional surveillance data with social intelligence multidimensional data streams to produce the next level of health intelligence. Two-model use cases in a large county demonstrate how the HI-Atlas produced relevant PHI to inform public health decision makers to (1) support identification and prioritization of vulnerable communities at risk for COVID-19 spread and vaccine hesitancy, and (2) support the implementation of a generic model for planning equitable COVID-19 vaccine preparedness and distribution. Conclusion  The scalable models of data sources, analyses, and smart hybrid data layer visualizations implemented in the HI-Atlas are the Health Intelligence tools designed to support real-time proactive planning and monitoring for COVID-19 vaccine preparedness and distribution in counties and states.