Development of a “meta-model” to address missing data,predict patient-specific cancer survival and provide a foundation for clinical decision support

ObjectiveLike most real-world data, electronic health record (EHR)–derived data from oncology patients typically exhibits wide interpatient variability in terms of available data elements. This interpatient variability leads to missing data and can present critical challenges in developing and implementing predictive models to underlie clinical decision support for patient-specific oncology care. Here, we sought to develop a novel ensemble approach to addressing missing data that we term the “meta-model” and apply the meta-model to patient-specific cancer prognosis.Materials and MethodsUsing real-world data, we developed a suite of individual random survival forest models to predict survival in patients with advanced lung cancer, colorectal cancer, and breast cancer. Individual models varied by the predictor data used. We combined models for each cancer type into a meta-model that predicted survival for each patient using a weighted mean of the individual models for which the patient had all requisite predictors.ResultsThe meta-model significantly outperformed many of the individual models and performed similarly to the best performing individual models. Comparisons of the meta-model to a more traditional imputation-based method of addressing missing data supported the meta-model’s utility.ConclusionsWe developed a novel machine learning–based strategy to underlie clinical decision support and predict survival in cancer patients, despite missing data. The meta-model may more generally provide a tool for addressing missing data across a variety of clinical prediction problems. Moreover, the meta-model may address other challenges in clinical predictive modeling including model extensibility and integration of predictive algorithms trained across different institutions and datasets.     

Psychosocial concerns predict longitudinal trajectories of distress in newly diagnosed cancer patients

INTRODUCTIONUnmet psychosocial concerns are associated with emotional distress among cancer patients. This longitudinal study aimed to identify specific psychosocial concern profiles and trajectories of emotional distress, and examine their association among newly diagnosed adult cancer patients across the first year of diagnosis.METHODSAdult patients aged 21–64 years were screened to determine their eligibility for this study. Psychosocial concerns and psychological distress were measured using the Problem List and the Distress Thermometer, respectively (n = 221). Latent transition mixture analysis was used to determine specific psychosocial concern profiles and trajectories of emotional distress, and examine associations with adjustments made for demographic and medical variables.RESULTSTwo classes of psychosocial concerns were identified: low (81%) and high (19%) concerns. Two trajectories of distress were identified: low stable (69%) and high stable (31%) trajectories. Patients in the high concerns class were significantly more likely to demonstrate the high stable trajectory of distress.CONCLUSIONOur findings highlight the importance of concurrent assessment of multiple psychosocial concerns and screening of emotional distress throughout a cancer patient’s treatment journey. Such assessments can effectively guide interventions to address individual concerns and alleviate emotional distress among newly diagnosed cancer patients.     

Pattern discovery,validation, and online experiments: a methodology for discovering television shows for public health announcements

ObjectivePublic Health Announcements (PHAs) on television are a means of raising awareness about risk behaviors and chronic conditions. PHAs’ scarce airtime puts stress on their target audience reach. We seek to help health campaigns select television shows for their PHAs about smoking, binge drinking, drug overdose, obesity, diabetes, STDs, and other conditions using available statistics.Materials and MethodsUsing Nielsen’s TV viewership database for the entire US panel, we presented a novel show discovery methodology for PHAs that combined (i) pattern discovery from high-dimensional data (ii) nonparametric tests for validation, and (iii) online experiments on Facebook.ResultsThe nonparametric tests verified the robustness of the discovered associations between the popularity of certain shows and health conditions. Findings from fifty (independent) online experiments (where our awareness messages were seen by nearly 1.5 million American adults) empirically demonstrated the value of the methodology.DiscussionFor 2016, the methodology identified several shows whose popularities were genuinely associated with certain health conditions, opening up the possibility of health agencies embracing both big data and large-scale experimentation to address an old problem in a new way.ConclusionPolicy makers can repeatedly apply the methodology as new data streams in, with perhaps different feature sets, pattern discovery techniques, and online experiments running over longer periods. The comparatively lower initial investment in the methodology can pay off by identifying several shows for a potentially national television campaign. As simply a by-product, the initial investment also results in awareness messages that might reach millions of individuals.     

Can we understand population healthcare needs using electronic medical records?

INTRODUCTIONThe identification of population-level healthcare needs using hospital electronic medical records (EMRs) is a promising approach for the evaluation and development of tailored healthcare services. Population segmentation based on healthcare needs may be possible using information on health and social service needs from EMRs. However, it is currently unknown if EMRs from restructured hospitals in Singapore provide information of sufficient quality for this purpose. We compared the inter-rater reliability between a population segment that was assigned prospectively and one that was assigned retrospectively based on EMR review.METHODS200 non-critical patients aged ≥ 55 years were prospectively evaluated by clinicians for their healthcare needs in the emergency department at Singapore General Hospital, Singapore. Trained clinician raters with no prior knowledge of these patients subsequently accessed the EMR up to the prospective rating date. A similar healthcare needs evaluation was conducted using the EMR. The inter-rater reliability between the two rating sets was evaluated using Cohen’s Kappa and the incidence of missing information was tabulated.RESULTSThe inter-rater reliability for the medical ‘global impression’ rating was 0.37 for doctors and 0.35 for nurses. The inter-rater reliability for the same variable, retrospectively rated by two doctors, was 0.75. Variables with a higher incidence of missing EMR information such as ‘social support in case of need’ and ‘patient activation’ had poorer inter-rater reliability.CONCLUSIONPre-existing EMR systems may not capture sufficient information for reliable determination of healthcare needs. Thus, we should consider integrating policy-relevant healthcare need variables into EMRs.     

STAN: spatio-temporal attention network for pandemic prediction using real-world evidence

ObjectiveWe aim to develop a hybrid model for earlier and more accurate predictions for the number of infected cases in pandemics by (1) using patients’ claims data from different counties and states that capture local disease status and medical resource utilization; (2) utilizing demographic similarity and geographical proximity between locations; and (3) integrating pandemic transmission dynamics into a deep learning model.Materials and MethodsWe proposed a spatio-temporal attention network (STAN) for pandemic prediction. It uses a graph attention network to capture spatio-temporal trends of disease dynamics and to predict the number of cases for a fixed number of days into the future. We also designed a dynamics-based loss term for enhancing long-term predictions. STAN was tested using both real-world patient claims data and COVID-19 statistics over time across US counties.ResultsSTAN outperforms traditional epidemiological models such as susceptible-infectious-recovered (SIR), susceptible-exposed-infectious-recovered (SEIR), and deep learning models on both long-term and short-term predictions, achieving up to 87% reduction in mean squared error compared to the best baseline prediction model.ConclusionsBy combining information from real-world claims data and disease case counts data, STAN can better predict disease status and medical resource utilization.     

Application of natural language processing techniques to identify off-label drug usage from various online health communities

ObjectiveOutcomes mentioned on online health communities (OHCs) by patients can serve as a source of evidence for off-label drug usage evaluation, but identifying these outcomes manually is tedious work. We have built a natural language processing model to identify off-label usage of drugs mentioned in these patient posts.Materials and MethodsSingle patient posts from 4 major OHCs were considered for this study. A text classification model was built to classify the posts as either relevant or not relevant based on patient experience. The relevant posts were passed through a spelling correction tool, CSpell, and then medications and indications from these posts were identified using cTAKES (clinical Text Analysis and Knowledge Extraction System), a named entity recognition tool. Drug and indication pairs were identified using a dependency parser. Finally, if the paired indication was not mentioned on the label of the drug approved by U.S. Food and Drug Administration, it was tagged as off-label use of that drug.ResultsUsing this algorithm, we identified 289 off-label indications, achieving a recall of 76%.ConclusionsThe method designed in this study identifies and extracts the semantic relationship between drugs and indications from demotic posts in OHCs. The results demonstrate the feasibility of using natural language processing techniques in identifying off-label drug usage across online health forums for a variety of drugs. Understanding patients’ off-label use of drugs may be able to help manufacturers innovate to better address patients’ needs and assist doctors’ prescribing decisions.     

Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees

ObjectiveAfter deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating/fitting, we study online methods with performance guarantees. Materials and MethodsWe introduce 2 procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting Chronic Obstructive Pulmonary Disease (COPD) risk. We derive “Type I and II” regret bounds, which guarantee the procedures are noninferior to a static model and competitive with an oracle logistic reviser in terms of the average loss.ResultsBoth procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95%CI, 0.818–0.938). Online recalibration using BLR and MarBLR improved the aECI towards the ideal value of zero, attaining 0.265 (95%CI, 0.230–0.300) and 0.241 (95%CI, 0.216–0.266), respectively. When performing more extensive logistic model revisions, BLR and MarBLR increased the average area under the receiver-operating characteristic curve (aAUC) from 0.767 (95%CI, 0.765–0.769) to 0.800 (95%CI, 0.798–0.802) and 0.799 (95%CI, 0.797–0.801), respectively, in stationary settings and protected against substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually refitted gradient boosted tree to achieve aAUCs of 0.924 (95%CI, 0.913–0.935) and 0.925 (95%CI, 0.914–0.935), compared to the static model’s aAUC of 0.904 (95%CI, 0.892–0.916).DiscussionDespite its simplicity, BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data.ConclusionsBLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time.     

Healthcare Process Modeling to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals): Development and evaluation of a conceptual framework

ObjectiveThere are signals of clinicians’ expert and knowledge-driven behaviors within clinical information systems (CIS) that can be exploited to support clinical prediction. Describe development of the Healthcare Process Modeling Framework to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals).Materials and MethodsWe employed an iterative framework development approach that combined data-driven modeling and simulation testing to define and refine a process for phenotyping clinician behaviors. Our framework was developed and evaluated based on the Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) predictive model to detect and leverage signals of clinician expertise for prediction of patient trajectories.ResultsSeven themes—identified during development and simulation testing of the CONCERN model—informed framework development. The HPM-ExpertSignals conceptual framework includes a 3-step modeling technique: (1) identify patterns of clinical behaviors from user interaction with CIS; (2) interpret patterns as proxies of an individual’s decisions, knowledge, and expertise; and (3) use patterns in predictive models for associations with outcomes. The CONCERN model differentiated at risk patients earlier than other early warning scores, lending confidence to the HPM-ExpertSignals framework.DiscussionThe HPM-ExpertSignals framework moves beyond transactional data analytics to model clinical knowledge, decision making, and CIS interactions, which can support predictive modeling with a focus on the rapid and frequent patient surveillance cycle.ConclusionsWe propose this framework as an approach to embed clinicians’ knowledge-driven behaviors in predictions and inferences to facilitate capture of healthcare processes that are activated independently, and sometimes well before, physiological changes are apparent.     

Examining association between cohesion and diversity in collaboration networks of pharmaceutical clinical trials with drug approvals

ObjectiveClinical trials ensure that pharmaceutical treatments are safe, efficacious, and effective for public consumption, but are extremely complex, taking up to 10 years and $2.6 billion to complete. One main source of complexity arises from the collaboration between actors, and network science methodologies can be leveraged to explore that complexity. We aim to characterize collaborations between actors in the clinical trials context and investigate trends of successful actors.Materials and MethodsWe constructed a temporal network of clinical trial collaborations between large and small-size pharmaceutical companies, academic institutions, nonprofit organizations, hospital systems, and government agencies from public and proprietary data and introduced metrics to quantify actors’ collaboration network structure, organizational behavior, and partnership characteristics. A multivariable regression analysis was conducted to determine the metrics’ relationship with success.ResultsWe found a positive correlation between the number of successful approved trials and interdisciplinary collaborations measured by a collaboration diversity metric (P < .01). Our results also showed a negative effect of the local clustering coefficient (P < .01) on the success of clinical trials. Large pharmaceutical companies have the lowest local clustering coefficient and more diversity in partnerships across biomedical specializations.ConclusionsLarge pharmaceutical companies are more likely to collaborate with a wider range of actors from other specialties, especially smaller industry actors who are newcomers in clinical research, resulting in exclusive access to smaller actors. Future investigations are needed to show how concentrations of influence and resources might result in diminished gains in treatment development.     

Inviting patients to identify diagnostic concerns through structured evaluation of their online visit notes

BackgroundThe 21st Century Cures Act mandates patients’ access to their electronic health record (EHR) notes. To our knowledge, no previous work has systematically invited patients to proactively report diagnostic concerns while documenting and tracking their diagnostic experiences through EHR-based clinician note review.ObjectiveTo test if patients can identify concerns about their diagnosis through structured evaluation of their online visit notes.MethodsIn a large integrated health system, patients aged 18–85 years actively using the patient portal and seen between October 2019 and February 2020 were invited to respond to an online questionnaire if an EHR algorithm detected any recent unexpected return visit following an initial primary care consultation (“at-risk” visit). We developed and tested an instrument (Safer Dx Patient Instrument) to help patients identify concerns related to several dimensions of the diagnostic process based on notes review and recall of recent “at-risk” visits. Additional questions assessed patients’ trust in their providers and their general feelings about the visit. The primary outcome was a self-reported diagnostic concern. Multivariate logistic regression tested whether the primary outcome was predicted by instrument variables.ResultsOf 293 566 visits, the algorithm identified 1282 eligible patients, of whom 486 responded. After applying exclusion criteria, 418 patients were included in the analysis. Fifty-one patients (12.2%) identified a diagnostic concern. Patients were more likely to report a concern if they disagreed with statements “the care plan the provider developed for me addressed all my medical concerns” [odds ratio (OR), 2.65; 95% confidence interval [CI], 1.45–4.87) and “I trust the provider that I saw during my visit” (OR, 2.10; 95% CI, 1.19–3.71) and agreed with the statement “I did not have a good feeling about my visit” (OR, 1.48; 95% CI, 1.09–2.01).ConclusionPatients can identify diagnostic concerns based on a proactive online structured evaluation of visit notes. This surveillance strategy could potentially improve transparency in the diagnostic process.     

Deep propensity network using a sparse autoencoder for estimation of treatment effects

ObjectiveDrawing causal estimates from observational data is problematic, because datasets often contain underlying bias (eg, discrimination in treatment assignment). To examine causal effects, it is important to evaluate what-if scenarios—the so-called “counterfactuals.” We propose a novel deep learning architecture for propensity score matching and counterfactual prediction—the deep propensity network using a sparse autoencoder (DPN-SA)—to tackle the problems of high dimensionality, nonlinear/nonparallel treatment assignment, and residual confounding when estimating treatment effects.Materials and MethodsWe used 2 randomized prospective datasets, a semisynthetic one with nonlinear/nonparallel treatment selection bias and simulated counterfactual outcomes from the Infant Health and Development Program and a real-world dataset from the LaLonde’s employment training program. We compared different configurations of the DPN-SA against logistic regression and LASSO as well as deep counterfactual networks with propensity dropout (DCN-PD). Models’ performances were assessed in terms of average treatment effects, mean squared error in precision on effect’s heterogeneity, and average treatment effect on the treated, over multiple training/test runs.ResultsThe DPN-SA outperformed logistic regression and LASSO by 36%–63%, and DCN-PD by 6%–10% across all datasets. All deep learning architectures yielded average treatment effects close to the true ones with low variance. Results were also robust to noise-injection and addition of correlated variables. Code is publicly available at https://github.com/Shantanu48114860/DPN-SAz.Discussion and ConclusionDeep sparse autoencoders are particularly suited for treatment effect estimation studies using electronic health records because they can handle high-dimensional covariate sets, large sample sizes, and complex heterogeneity in treatment assignments.     

Harmonizing units and values of quantitative data elements in a very large nationally pooled electronic health record (EHR) dataset

ObjectiveThe goals of this study were to harmonize data from electronic health records (EHRs) into common units, and impute units that were missing.Materials and MethodsThe National COVID Cohort Collaborative (N3C) table of laboratory measurement data—over 3.1 billion patient records and over 19 000 unique measurement concepts in the Observational Medical Outcomes Partnership (OMOP) common-data-model format from 55 data partners. We grouped ontologically similar OMOP concepts together for 52 variables relevant to COVID-19 research, and developed a unit-harmonization pipeline comprised of (1) selecting a canonical unit for each measurement variable, (2) arriving at a formula for conversion, (3) obtaining clinical review of each formula, (4) applying the formula to convert data values in each unit into the target canonical unit, and (5) removing any harmonized value that fell outside of accepted value ranges for the variable. For data with missing units for all the results within a lab test for a data partner, we compared values with pooled values of all data partners, using the Kolmogorov-Smirnov test.ResultsOf the concepts without missing values, we harmonized 88.1% of the values, and imputed units for 78.2% of records where units were absent (41% of contributors’ records lacked units).DiscussionThe harmonization and inference methods developed herein can serve as a resource for initiatives aiming to extract insight from heterogeneous EHR collections. Unique properties of centralized data are harnessed to enable unit inference.ConclusionThe pipeline we developed for the pooled N3C data enables use of measurements that would otherwise be unavailable for analysis.     

Design thinking in applied informatics: what can we learn from Project HealthDesign?

ObjectiveThe goals of this study are to describe the value and impact of Project HealthDesign (PHD), a program of the Robert Wood Johnson Foundation that applied design thinking to personal health records, and to explore the applicability of the PHD model to another challenging translational informatics problem: the integration of AI into the healthcare system.Materials and MethodsWe assessed PHD’s impact and value in 2 ways. First, we analyzed publication impact by calculating a PHD h-index and characterizing the professional domains of citing journals. Next, we surveyed and interviewed PHD grantees, expert consultants, and codirectors to assess the program’s components and the potential future application of design thinking to artificial intelligence (AI) integration into healthcare.ResultsThere was a total of 1171 unique citations to PHD-funded work (collective h-index of 25). Studies citing PHD span medical, legal, and computational journals. Participants stated that this project transformed their thinking, altered their career trajectory, and resulted in technology transfer into the commercial sector. Participants felt, in general, that the approach would be valuable in solving contemporary challenges integrating AI in healthcare including complex social questions, integrating knowledge from multiple domains, implementation, and governance.ConclusionDesign thinking is a systematic approach to problem-solving characterized by cooperation and collaboration. PHD generated significant impacts as measured by citations, reach, and overall effect on participants. PHD’s design thinking methods are potentially useful to other work on cyber–physical systems, such as the use of AI in healthcare, to propose structural or policy-related changes that may affect adoption, value, and improvement of the care delivery system.     

Smart hospital infrastructure: geomagnetic in-hospital medical worker tracking

PurposeLocation visualization is essential for locating people/objects, improving efficiency, and preventing accidents. In hospitals, Wi-Fi, Bluetooth low energy (BLE) Beacon, indoor messaging system, and similar methods have generally been used for tracking, with Wi-Fi and BLE being the most common. Recently, nurses are increasingly using mobile devices, such as smartphones and tablets, while shifting. The accuracy when using Wi-Fi or BLE may be affected by interference or multipath propagation. In this research, we evaluated the positioning accuracy of geomagnetic indoor positioning in hospitals.Materials and MethodsWe compared the position measurement accuracy of a geomagnetic method alone, Wi-Fi alone, BLE beacons alone, geomagnetic plus Wi-Fi, and geomagnetic plus BLE in a general inpatient ward, using a geomagnetic positioning algorithm by GiPStech. The existing Wi-Fi infrastructure was used, and 20 additional BLE beacons were installed. Our first experiment compared these methods’ accuracy for 8 test routes, while the second experiment verified a combined geomagnetic/BLE beacon method using 3 routes based on actual daily activities.ResultsThe experimental results demonstrated that the most accurate method was geomagnetic/BLE, followed by geomagnetic/Wi-Fi, and then geomagnetic alone.DiscussionThe geomagnetic method’s positioning accuracy varied widely, but combining it with BLE beacons reduced the average position error to approximately 1.2 m, and the positioning accuracy could be improved further. We believe this could effectively target humans (patients) where errors of up to 3 m can generally be tolerated.ConclusionIn conjunction with BLE beacons, geomagnetic positioning could be sufficiently effective for many in-hospital localization tasks.     

Doctors and social media: knowledge gaps and unsafe practices

INTRODUCTIONEasy access and availability of communication tools have facilitated doctors’ communication, adding new challenges. Through this study, we aimed to determine the profile of the knowledge and practices of doctors in our institution, and to identify knowledge gaps in the use of social media accounts.METHODSAn anonymous survey was sent by electronic mail in March–May 2018 to 931 doctors working in National University Hospital, Singapore. It included questions on demographics; use of social media; and case-based scenarios involving professionalism, patient-doctor relationship and personal practices of social media use.RESULTSThe response rate was 12.8%. The majority of the respondents owned a social media account (93.3%), had not received education on social media use in medical school (84.0%), did not own a separate work phone (80.7%) and claimed to have no medical education on this as a doctor (58.8%). Unawareness of the institution’s social media policy was reported by 14.3% of the respondents. Questions on knowledge of the privacy settings of their account were incorrectly answered. Only 75.6%–82.4% of the participants responded ‘no’ when asked if they would post pictures of patients or their results, even if there were no patient identifiers.CONCLUSIONThere is inadequate knowledge regarding institutional social media policy and privacy settings of social media accounts among doctors. Regarding practices in social media use, while most agree that caution should be exercised for online posts involving patients, ambiguity still exists. The emerging knowledge deficit and potentially unsafe practices that are identified can be addressed through continuing medical education and training on social media use.     

The emerging role of clinical informatics fellows in service learning during the COVID-19 pandemic

ObjectiveThe study sought to describe the contributions of clinical informatics (CI) fellows to their institutions’ coronavirus disease 2019 (COVID-19) response.Materials and MethodsWe designed a survey to capture key domains of health informatics and perceptions regarding fellows’ application of their CI skills. We also conducted detailed interviews with select fellows and described their specific projects in a brief case series.ResultsForty-one of the 99 CI fellows responded to our survey. Seventy-five percent agreed that they were “able to apply clinical informatics training and interest to the COVID-19 response.” The most common project types were telemedicine (63%), reporting and analytics (49%), and electronic health record builds and governance (32%). Telehealth projects included training providers on existing telehealth tools, building entirely new virtual clinics for video triage of COVID-19 patients, and pioneering workflows and implementation of brand-new emergency department and inpatient video visit types. Analytics projects included reports and dashboards for institutional leadership, as well as developing digital contact tracing tools. For electronic health record builds, fellows directly contributed to note templates with embedded screening and testing guidance, adding COVID-19 tests to order sets, and validating clinical triage workflows.DiscussionFellows were engaged in projects that span the breadth of the CI specialty and were able to make system-wide contributions in line with their educational milestones.ConclusionsCI fellows contributed meaningfully and rapidly to their institutions’ response to the COVID-19 pandemic.     

Public awareness,beliefs and attitude towards depressive disorders in Saudi Arabia

Objectives:To investigate the degree of public awareness, beliefs, and attitudes regarding major depression and available treatment options in the Saudi population.Methods:A community-based cross-sectional study of 1,188 participants was carried out from March to April 2021 in Ha’il, Saudi Arabia using an online self-administered questionnaire. Using a snowball sampling technique, the authors targeted the Saudi population living in Ha’il region.Results:Overall, 65.6% of the participants had good awareness regarding depression disorder in total. Of the participants, 72.9% had good awareness regarding general awareness, 85.4% regarding depression symptoms, 12.3% regarding risk factors, and 15.7% regarding treatments. Of the participants, 67.3% believed that depression was caused by lack of faith and 45.5% believed that depression was caused by “the evil eye” or black magic. Of the participants, 56% believed in faith healers as a legitimate treatment approach. Of the participants, 63.9% were willing to work with individuals with depression, 62.7% were willing to establish friendships with them, and 27.9% believed that individuals with depression had weak personalities.Conclusion:The general population exhibited good general awareness regarding depression and its symptoms, but knowledge of risk factors and treatments was poor. Our findings underscore the need for public educational programs to increase public awareness about the risk factors and treatment options for depression.