Lessons learned from the introduction of an electronic safety net to enhance test result management in an Australian mothers’ hospital

This study describes the implementation and impact of an electronic test result acknowledgement (RA) system in the Mater Mothers’ Hospital in Brisbane, Australia. The Verdi application electronically records clinicians’ acknowledgement of the review of results. Hospital data (August 2011–August 2012) were extracted to measure clinicians’ acknowledgement practices. There were 27 354 inpatient test results for 6855 patients. All test results were acknowledged. 60% (95% CI 59% to 61%) of laboratory and 44% (95% CI 40% to 48%) of imaging results were acknowledged within 24 h. The median time between report availability and acknowledgement was 18.1 h for laboratory and 1 day 18 h for imaging results. The median time from when a result was first viewed to its acknowledgement was 7 min for laboratory and 1 min for imaging results. The longest recorded time to acknowledgement was 38 days. Electronic RA provides a safety net to enhance test result management.     

Use of population health data to refine diagnostic decision-making for pertussis

Objective

To improve identification of pertussis cases by developing a decision model that incorporates recent, local, population-level disease incidence.

Design

Retrospective cohort analysis of 443 infants tested for pertussis (2003–7).

Measurements

Three models (based on clinical data only, local disease incidence only, and a combination of clinical data and local disease incidence) to predict pertussis positivity were created with demographic, historical, physical exam, and state-wide pertussis data. Models were compared using sensitivity, specificity, area under the receiver-operating characteristics (ROC) curve (AUC), and related metrics.

Results

The model using only clinical data included cyanosis, cough for 1 week, and absence of fever, and was 89% sensitive (95% CI 79 to 99), 27% specific (95% CI 22 to 32) with an area under the ROC curve of 0.80. The model using only local incidence data performed best when the proportion positive of pertussis cultures in the region exceeded 10% in the 8–14 days prior to the infant''s associated visit, achieving 13% sensitivity, 53% specificity, and AUC 0.65. The combined model, built with patient-derived variables and local incidence data, included cyanosis, cough for 1 week, and the variable indicating that the proportion positive of pertussis cultures in the region exceeded 10% 8–14 days prior to the infant''s associated visit. This model was 100% sensitive (p<0.04, 95% CI 92 to 100), 38% specific (p<0.001, 95% CI 33 to 43), with AUC 0.82.

Conclusions

Incorporating recent, local population-level disease incidence improved the ability of a decision model to correctly identify infants with pertussis. Our findings support fostering bidirectional exchange between public health and clinical practice, and validate a method for integrating large-scale public health datasets with rich clinical data to improve decision-making and public health.     

Assessing the quality of clinical and administrative data extracted from hospitals: the General Medicine Inpatient Initiative (GEMINI) experience

ObjectiveLarge clinical databases are increasingly used for research and quality improvement. We describe an approach to data quality assessment from the General Medicine Inpatient Initiative (GEMINI), which collects and standardizes administrative and clinical data from hospitals.MethodsThe GEMINI database contained 245 559 patient admissions at 7 hospitals in Ontario, Canada from 2010 to 2017. We performed 7 computational data quality checks and iteratively re-extracted data from hospitals to correct problems. Thereafter, GEMINI data were compared to data that were manually abstracted from the hospital’s electronic medical record for 23 419 selected data points on a sample of 7488 patients.ResultsComputational checks flagged 103 potential data quality issues, which were either corrected or documented to inform future analysis. For example, we identified the inclusion of canceled radiology tests, a time shift of transfusion data, and mistakenly processing the chemical symbol for sodium (“Na”) as a missing value. Manual validation identified 1 important data quality issue that was not detected by computational checks: transfusion dates and times at 1 site were unreliable. Apart from that single issue, across all data tables, GEMINI data had high overall accuracy (ranging from 98%–100%), sensitivity (95%–100%), specificity (99%–100%), positive predictive value (93%–100%), and negative predictive value (99%–100%) compared to the gold standard.Discussion and ConclusionComputational data quality checks with iterative re-extraction facilitated reliable data collection from hospitals but missed 1 critical quality issue. Combining computational and manual approaches may be optimal for assessing the quality of large multisite clinical databases.     

The application of naive Bayes model averaging to predict Alzheimer's disease from genome-wide data

Objective

Predicting patient outcomes from genome-wide measurements holds significant promise for improving clinical care. The large number of measurements (eg, single nucleotide polymorphisms (SNPs)), however, makes this task computationally challenging. This paper evaluates the performance of an algorithm that predicts patient outcomes from genome-wide data by efficiently model averaging over an exponential number of naive Bayes (NB) models.

Design

This model-averaged naive Bayes (MANB) method was applied to predict late onset Alzheimer''s disease in 1411 individuals who each had 312 318 SNP measurements available as genome-wide predictive features. Its performance was compared to that of a naive Bayes algorithm without feature selection (NB) and with feature selection (FSNB).

Measurement

Performance of each algorithm was measured in terms of area under the ROC curve (AUC), calibration, and run time.

Results

The training time of MANB (16.1 s) was fast like NB (15.6 s), while FSNB (1684.2 s) was considerably slower. Each of the three algorithms required less than 0.1 s to predict the outcome of a test case. MANB had an AUC of 0.72, which is significantly better than the AUC of 0.59 by NB (p<0.00001), but not significantly different from the AUC of 0.71 by FSNB. MANB was better calibrated than NB, and FSNB was even better in calibration. A limitation was that only one dataset and two comparison algorithms were included in this study.

Conclusion

MANB performed comparatively well in predicting a clinical outcome from a high-dimensional genome-wide dataset. These results provide support for including MANB in the methods used to predict outcomes from large, genome-wide datasets.     

Full impact of laboratory information system requires direct use by clinical staff: cluster randomized controlled trial

Objective

To evaluate the time to communicate laboratory results to health centers (HCs) between the e-Chasqui web-based information system and the pre-existing paper-based system.

Methods

Cluster randomized controlled trial in 78 HCs in Peru. In the intervention group, 12 HCs had web access to results via e-Chasqui (point-of-care HCs) and forwarded results to 17 peripheral HCs. In the control group, 22 point-of-care HCs received paper results directly and forwarded them to 27 peripheral HCs. Baseline data were collected for 15 months. Post-randomization data were collected for at least 2 years. Comparisons were made between intervention and control groups, stratified by point-of-care versus peripheral HCs.

Results

For point-of-care HCs, the intervention group took less time to receive drug susceptibility tests (DSTs) (median 9 vs 16 days, p<0.001) and culture results (4 vs 8 days, p<0.001) and had a lower proportion of ‘late’ DSTs taking >60 days to arrive (p<0.001) than the control. For peripheral HCs, the intervention group had similar communication times for DST (median 22 vs 19 days, p=0.30) and culture (10 vs 9 days, p=0.10) results, as well as proportion of ‘late’ DSTs (p=0.57) compared with the control.

Conclusions

Only point-of-care HCs with direct access to the e-Chasqui information system had reduced communication times and fewer results with delays of >2 months. Peripheral HCs had no benefits from the system. This suggests that health establishments should have point-of-care access to reap the benefits of electronic laboratory reporting.     

Research and applications: Impact of an electronic medication management system on hospital doctors’ and nurses’ work: a controlled pre–post,time and motion study

Objective

To quantify and compare the time doctors and nurses spent on direct patient care, medication-related tasks, and interactions before and after electronic medication management system (eMMS) introduction.

Methods

Controlled pre–post, time and motion study of 129 doctors and nurses for 633.2 h on four wards in a 400-bed hospital in Sydney, Australia. We measured changes in proportions of time on tasks and interactions by period, intervention/control group, and profession.

Results

eMMS was associated with no significant change in proportions of time spent on direct care or medication-related tasks relative to control wards. In the post-period control ward, doctors spent 19.7% (2 h/10 h shift) of their time on direct care and 7.4% (44.4 min/10 h shift) on medication tasks, compared to intervention ward doctors (25.7% (2.6 h/shift; p=0.08) and 8.5% (51 min/shift; p=0.40), respectively). Control ward nurses in the post-period spent 22.1% (1.9 h/8.5 h shift) of their time on direct care and 23.7% on medication tasks compared to intervention ward nurses (26.1% (2.2 h/shift; p=0.23) and 22.6% (1.9 h/shift; p=0.28), respectively). We found intervention ward doctors spent less time alone (p=0.0003) and more time with other doctors (p=0.003) and patients (p=0.009). Nurses on the intervention wards spent less time with doctors following eMMS introduction (p=0.0001).

Conclusions

eMMS introduction did not result in redistribution of time away from direct care or towards medication tasks. Work patterns observed on these intervention wards were associated with previously reported significant reductions in prescribing error rates relative to the control wards.     

ICU nurses' acceptance of electronic health records

Objective

To assess intensive care unit (ICU) nurses'' acceptance of electronic health records (EHR) technology and examine the relationship between EHR design, implementation factors, and nurse acceptance.

Design

The authors analyzed data from two cross-sectional survey questionnaires distributed to nurses working in four ICUs at a northeastern US regional medical center, 3 months and 12 months after EHR implementation.

Measurements

Survey items were drawn from established instruments used to measure EHR acceptance and usability, and the usefulness of three EHR functionalities, specifically computerized provider order entry (CPOE), the electronic medication administration record (eMAR), and a nursing documentation flowsheet.

Results

On average, ICU nurses were more accepting of the EHR at 12 months as compared to 3 months. They also perceived the EHR as being more usable and both CPOE and eMAR as being more useful. Multivariate hierarchical modeling indicated that EHR usability and CPOE usefulness predicted EHR acceptance at both 3 and 12 months. At 3 months postimplementation, eMAR usefulness predicted EHR acceptance, but its effect disappeared at 12 months. Nursing flowsheet usefulness predicted EHR acceptance but only at 12 months.

Conclusion

As the push toward implementation of EHR technology continues, more hospitals will face issues related to acceptance of EHR technology by staff caring for critically ill patients. This research suggests that factors related to technology design have strong effects on acceptance, even 1 year following the EHR implementation.     

Editor's Choice: Increasing the response rate of text messaging data collection: a delayed randomized controlled trial

Objective To test the effectiveness of multiple interventions on increasing the response rate of text messaging for longitudinal data collection.Methods Our cohort included 283 caregivers of children aged 6–12 months who were participating in an anemia program in rural China. Using text messages to collect data on anemia medication adherence, we conducted a delayed randomized controlled trial to test multiple interventions (an additional four reminders; a ¥5.0 (US$0.79) credit reward for replying; and a feedback text message). After a 6-week pilot study with week 7 as the baseline measurement, we randomly allocated all participants into two groups: group 1 (n = 142) and group 2 (n = 141). During weeks 8–11, we introduced the interventions to group 1, and in weeks 12–15 the intervention was introduced to both groups. We compared the response rates between groups and explored factors affecting the response rate.Results During weeks 8–11, the response rates in group 1 increased and were significantly higher than in group 2 (p<0.05). During weeks 12–15, the response rate increased significantly in group 2 (p>0.05) and slightly decreased in group 1. Younger participants or participants who had children with lower hemoglobin concentration were more likely to reply (p = 0.02). Sending four reminders on the second day contributed to only 286 (11.7%) extra text messages.Discussion Our study showed that multiple interventions were effective in increasing response rate of text messaging data collection in rural China.Conclusions Larger multi-site studies are needed to find the most effective way of using these interventions to allow usage of text messaging data collection for health research.     

Research and applications: Patient-generated secure messages and eVisits on a patient portal: are patients at risk?

Background

Patient portals are becoming increasingly common, but the safety of patient messages and eVisits has not been well studied. Unlike patient-to-nurse telephonic communication, patient messages and eVisits involve an asynchronous process that could be hazardous if patients were using it for time-sensitive symptoms such as chest pain or dyspnea.

Methods

We retrospectively analyzed 7322 messages (6430 secure messages and 892 eVisits). To assess the overall risk associated with the messages, we looked for deaths within 30 days of the message and hospitalizations and emergency department (ED) visits within 7 days following the message. We also examined message content for symptoms of chest pain, breathing concerns, and other symptoms associated with high risk.

Results

Two deaths occurred within 30 days of a patient-generated message, but were not related to the message. There were six hospitalizations related to a previous secure message (0.09% of secure messages), and two hospitalizations related to a previous eVisit (0.22% of eVisits). High-risk symptoms were present in 3.5% of messages but a subject line search to identify these high-risk messages had a sensitivity of only 15% and a positive predictive value of 29%.

Conclusions

Patients use portal messages 3.5% of the time for potentially high-risk symptoms of chest pain, breathing concerns, abdominal pain, palpitations, lightheadedness, and vomiting. Death, hospitalization, or an ED visit was an infrequent outcome following a secure message or eVisit. Screening the message subject line for high-risk symptoms was not successful in identifying high-risk message content.     

Bionimbus: a cloud for managing,analyzing and sharing large genomics datasets

Background

As large genomics and phenotypic datasets are becoming more common, it is increasingly difficult for most researchers to access, manage, and analyze them. One possible approach is to provide the research community with several petabyte-scale cloud-based computing platforms containing these data, along with tools and resources to analyze it.

Methods

Bionimbus is an open source cloud-computing platform that is based primarily upon OpenStack, which manages on-demand virtual machines that provide the required computational resources, and GlusterFS, which is a high-performance clustered file system. Bionimbus also includes Tukey, which is a portal, and associated middleware that provides a single entry point and a single sign on for the various Bionimbus resources; and Yates, which automates the installation, configuration, and maintenance of the software infrastructure required.

Results

Bionimbus is used by a variety of projects to process genomics and phenotypic data. For example, it is used by an acute myeloid leukemia resequencing project at the University of Chicago. The project requires several computational pipelines, including pipelines for quality control, alignment, variant calling, and annotation. For each sample, the alignment step requires eight CPUs for about 12 h. BAM file sizes ranged from 5 GB to 10 GB for each sample.

Conclusions

Most members of the research community have difficulty downloading large genomics datasets and obtaining sufficient storage and computer resources to manage and analyze the data. Cloud computing platforms, such as Bionimbus, with data commons that contain large genomics datasets, are one choice for broadening access to research data in genomics.     

Research and applications: Breast cancer survivability prediction using labeled,unlabeled, and pseudo-labeled patient data

Background

Prognostic studies of breast cancer survivability have been aided by machine learning algorithms, which can predict the survival of a particular patient based on historical patient data. However, it is not easy to collect labeled patient records. It takes at least 5 years to label a patient record as ‘survived’ or ‘not survived’. Unguided trials of numerous types of oncology therapies are also very expensive. Confidentiality agreements with doctors and patients are also required to obtain labeled patient records.

Proposed method

These difficulties in the collection of labeled patient data have led researchers to consider semi-supervised learning (SSL), a recent machine learning algorithm, because it is also capable of utilizing unlabeled patient data, which is relatively easier to collect. Therefore, it is regarded as an algorithm that could circumvent the known difficulties. However, the fact is yet valid even on SSL that more labeled data lead to better prediction. To compensate for the lack of labeled patient data, we may consider the concept of tagging virtual labels to unlabeled patient data, that is, ‘pseudo-labels,’ and treating them as if they were labeled.

Results

Our proposed algorithm, ‘SSL Co-training’, implements this concept based on SSL. SSL Co-training was tested using the surveillance, epidemiology, and end results database for breast cancer and it delivered a mean accuracy of 76% and a mean area under the curve of 0.81.     

Electronic health records improve clinical note quality

Background and objective The clinical note documents the clinician''s information collection, problem assessment, clinical management, and its used for administrative purposes. Electronic health records (EHRs) are being implemented in clinical practices throughout the USA yet it is not known whether they improve the quality of clinical notes. The goal in this study was to determine if EHRs improve the quality of outpatient clinical notes.Materials and methods A five and a half year longitudinal retrospective multicenter quantitative study comparing the quality of handwritten and electronic outpatient clinical visit notes for 100 patients with type 2 diabetes at three time points: 6 months prior to the introduction of the EHR (before-EHR), 6 months after the introduction of the EHR (after-EHR), and 5 years after the introduction of the EHR (5-year-EHR). QNOTE, a validated quantitative instrument, was used to assess the quality of outpatient clinical notes. Its scores can range from a low of 0 to a high of 100. Sixteen primary care physicians with active practices used QNOTE to determine the quality of the 300 patient notes.Results The before-EHR, after-EHR, and 5-year-EHR grand mean scores (SD) were 52.0 (18.4), 61.2 (16.3), and 80.4 (8.9), respectively, and the change in scores for before-EHR to after-EHR and before-EHR to 5-year-EHR were 18% (p<0.0001) and 55% (p<0.0001), respectively. All the element and grand mean quality scores significantly improved over the 5-year time interval.Conclusions The EHR significantly improved the overall quality of the outpatient clinical note and the quality of all its elements, including the core and non-core elements. To our knowledge, this is the first study to demonstrate that the EHR significantly improves the quality of clinical notes.     

Influenza surveillance using electronic health records in the American Indian and Alaska Native population

Objective

Increasing use of electronic health records (EHRs) provides new opportunities for public health surveillance. During the 2009 influenza A (H1N1) virus pandemic, we developed a new EHR-based influenza-like illness (ILI) surveillance system designed to be resource sparing, rapidly scalable, and flexible. 4 weeks after the first pandemic case, ILI data from Indian Health Service (IHS) facilities were being analyzed.

Materials and methods

The system defines ILI as a patient visit containing either an influenza-specific International Classification of Disease, V.9 (ICD-9) code or one or more of 24 ILI-related ICD-9 codes plus a documented temperature ≥100°F. EHR-based data are uploaded nightly. To validate results, ILI visits identified by the new system were compared to ILI visits found by medical record review, and the new system''s results were compared with those of the traditional US ILI Surveillance Network.

Results

The system monitored ILI activity at an average of 60% of the 269 IHS electronic health databases. EHR-based surveillance detected ILI visits with a sensitivity of 96.4% and a specificity of 97.8% based on chart review (N=2375) of visits at two facilities in September 2009. At the peak of the pandemic (week 41, October 17, 2009), the median time from an ILI visit to data transmission was 6 days, with a mode of 1 day.

Discussion

EHR-based ILI surveillance was accurate, timely, occurred at the majority of IHS facilities nationwide, and provided useful information for decision makers. EHRs thus offer the opportunity to transform public health surveillance.     

Building longitudinal medication dose data using medication information extracted from clinical notes in electronic health records

ObjectiveTo develop an algorithm for building longitudinal medication dose datasets using information extracted from clinical notes in electronic health records (EHRs).Materials and MethodsWe developed an algorithm that converts medication information extracted using natural language processing (NLP) into a usable format and builds longitudinal medication dose datasets. We evaluated the algorithm on 2 medications extracted from clinical notes of Vanderbilt’s EHR and externally validated the algorithm using clinical notes from the MIMIC-III clinical care database.ResultsFor the evaluation using Vanderbilt’s EHR data, the performance of our algorithm was excellent; F1-measures were ≥0.98 for both dose intake and daily dose. For the external validation using MIMIC-III, the algorithm achieved F1-measures ≥0.85 for dose intake and ≥0.82 for daily dose.DiscussionOur algorithm addresses the challenge of building longitudinal medication dose data using information extracted from clinical notes. Overall performance was excellent, but the algorithm can perform poorly when incorrect information is extracted by NLP systems. Although it performed reasonably well when applied to the external data source, its performance was worse due to differences in the way the drug information was written. The algorithm is implemented in the R package, “EHR,” and the extracted data from Vanderbilt’s EHRs along with the gold standards are provided so that users can reproduce the results and help improve the algorithm.ConclusionOur algorithm for building longitudinal dose data provides a straightforward way to use EHR data for medication-based studies. The external validation results suggest its potential for applicability to other systems.     

Evaluating the reliability, validity, acceptability, and practicality of SMS text messaging as a tool to collect research data: results from the Feeding Your Baby project

Objective

To test the reliability, validity, acceptability, and practicality of short message service (SMS) messaging for collection of research data.

Materials and methods

The studies were carried out in a cohort of recently delivered women in Tayside, Scotland, UK, who were asked about their current infant feeding method and future feeding plans. Reliability was assessed by comparison of their responses to two SMS messages sent 1 day apart. Validity was assessed by comparison of their responses to text questions and the same question administered by phone 1 day later, by comparison with the same data collected from other sources, and by correlation with other related measures. Acceptability was evaluated using quantitative and qualitative questions, and practicality by analysis of a researcher log.

Results

Reliability of the factual SMS message gave perfect agreement. Reliabilities for the numerical question were reasonable, with κ between 0.76 (95% CI 0.56 to 0.96) and 0.80 (95% CI 0.59 to 1.00). Validity for data compared with that collected by phone within 24 h (κ =0.92 (95% CI 0.84 to 1.00)) and with health visitor data (κ =0.85 (95% CI 0.73 to 0.97)) was excellent. Correlation validity between the text responses and other related demographic and clinical measures was as expected. Participants found the method a convenient and acceptable way of providing data. For researchers, SMS text messaging provided an easy and functional method of gathering a large volume of data.

Conclusion

In this sample and for these questions, SMS was a reliable and valid method for capturing research data.     

Patient characteristics associated with venous thromboembolic events: a cohort study using pooled electronic health record data

Objective

To demonstrate the potential of de-identified clinical data from multiple healthcare systems using different electronic health records (EHR) to be efficiently used for very large retrospective cohort studies.

Materials and methods

Data of 959 030 patients, pooled from multiple different healthcare systems with distinct EHR, were obtained. Data were standardized and normalized using common ontologies, searchable through a HIPAA-compliant, patient de-identified web application (Explore; Explorys Inc). Patients were 26 years or older seen in multiple healthcare systems from 1999 to 2011 with data from EHR.

Results

Comparing obese, tall subjects with normal body mass index, short subjects, the venous thromboembolic events (VTE) OR was 1.83 (95% CI 1.76 to 1.91) for women and 1.21 (1.10 to 1.32) for men. Weight had more effect then height on VTE. Compared with Caucasian, Hispanic/Latino subjects had a much lower risk of VTE (female OR 0.47, 0.41 to 0.55; male OR 0.24, 0.20 to 0.28) and African-Americans a substantially higher risk (female OR 1.83, 1.76 to 1.91; male OR 1.58, 1.50 to 1.66). This 13-year retrospective study of almost one million patients was performed over approximately 125 h in 11 weeks, part time by the five authors.

Discussion

As research informatics tools develop and more clinical data become available in EHR, it is important to study and understand unique opportunities for clinical research informatics to transform the scale and resources needed to perform certain types of clinical research.

Conclusions

With the right clinical research informatics tools and EHR data, some types of very large cohort studies can be completed with minimal resources.     

Brief communication: Launching PCORnet,a national patient-centered clinical research network

The Patient-Centered Outcomes Research Institute (PCORI) has launched PCORnet, a major initiative to support an effective, sustainable national research infrastructure that will advance the use of electronic health data in comparative effectiveness research (CER) and other types of research. In December 2013, PCORI''s board of governors funded 11 clinical data research networks (CDRNs) and 18 patient-powered research networks (PPRNs) for a period of 18 months. CDRNs are based on the electronic health records and other electronic sources of very large populations receiving healthcare within integrated or networked delivery systems. PPRNs are built primarily by communities of motivated patients, forming partnerships with researchers. These patients intend to participate in clinical research, by generating questions, sharing data, volunteering for interventional trials, and interpreting and disseminating results. Rapidly building a new national resource to facilitate a large-scale, patient-centered CER is associated with a number of technical, regulatory, and organizational challenges, which are described here.     

Toward personalizing treatment for depression: predicting diagnosis and severity

Objective

Depression is a prevalent disorder difficult to diagnose and treat. In particular, depressed patients exhibit largely unpredictable responses to treatment. Toward the goal of personalizing treatment for depression, we develop and evaluate computational models that use electronic health record (EHR) data for predicting the diagnosis and severity of depression, and response to treatment.

Materials and methods

We develop regression-based models for predicting depression, its severity, and response to treatment from EHR data, using structured diagnosis and medication codes as well as free-text clinical reports. We used two datasets: 35 000 patients (5000 depressed) from the Palo Alto Medical Foundation and 5651 patients treated for depression from the Group Health Research Institute.

Results

Our models are able to predict a future diagnosis of depression up to 12 months in advance (area under the receiver operating characteristic curve (AUC) 0.70–0.80). We can differentiate patients with severe baseline depression from those with minimal or mild baseline depression (AUC 0.72). Baseline depression severity was the strongest predictor of treatment response for medication and psychotherapy.

Conclusions

It is possible to use EHR data to predict a diagnosis of depression up to 12 months in advance and to differentiate between extreme baseline levels of depression. The models use commonly available data on diagnosis, medication, and clinical progress notes, making them easily portable. The ability to automatically determine severity can facilitate assembly of large patient cohorts with similar severity from multiple sites, which may enable elucidation of the moderators of treatment response in the future.