Stroke risk prediction using machine learning: a prospective cohort study of 0.5 million Chinese adults

ObjectiveTo compare Cox models, machine learning (ML), and ensemble models combining both approaches, for prediction of stroke risk in a prospective study of Chinese adults.Materials and MethodsWe evaluated models for stroke risk at varying intervals of follow-up (<9 years, 0–3 years, 3–6 years, 6–9 years) in 503 842 adults without prior history of stroke recruited from 10 areas in China in 2004–2008. Inputs included sociodemographic factors, diet, medical history, physical activity, and physical measurements. We compared discrimination and calibration of Cox regression, logistic regression, support vector machines, random survival forests, gradient boosted trees (GBT), and multilayer perceptrons, benchmarking performance against the 2017 Framingham Stroke Risk Profile. We then developed an ensemble approach to identify individuals at high risk of stroke (>10% predicted 9-yr stroke risk) by selectively applying either a GBT or Cox model based on individual-level characteristics.ResultsFor 9-yr stroke risk prediction, GBT provided the best discrimination (AUROC: 0.833 in men, 0.836 in women) and calibration, with consistent results in each interval of follow-up. The ensemble approach yielded incrementally higher accuracy (men: 76%, women: 80%), specificity (men: 76%, women: 81%), and positive predictive value (men: 26%, women: 24%) compared to any of the single-model approaches.Discussion and ConclusionAmong several approaches, an ensemble model combining both GBT and Cox models achieved the best performance for identifying individuals at high risk of stroke in a contemporary study of Chinese adults. The results highlight the potential value of expanding the use of ML in clinical practice.     

Development and validation of prediction models for mechanical ventilation,renal replacement therapy,and readmission in COVID-19 patients

ObjectiveCoronavirus disease 2019 (COVID-19) patients are at risk for resource-intensive outcomes including mechanical ventilation (MV), renal replacement therapy (RRT), and readmission. Accurate outcome prognostication could facilitate hospital resource allocation. We develop and validate predictive models for each outcome using retrospective electronic health record data for COVID-19 patients treated between March 2 and May 6, 2020.Materials and MethodsFor each outcome, we trained 3 classes of prediction models using clinical data for a cohort of SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2)–positive patients (n = 2256). Cross-validation was used to select the best-performing models per the areas under the receiver-operating characteristic and precision-recall curves. Models were validated using a held-out cohort (n = 855). We measured each model’s calibration and evaluated feature importances to interpret model output.ResultsThe predictive performance for our selected models on the held-out cohort was as follows: area under the receiver-operating characteristic curve—MV 0.743 (95% CI, 0.682-0.812), RRT 0.847 (95% CI, 0.772-0.936), readmission 0.871 (95% CI, 0.830-0.917); area under the precision-recall curve—MV 0.137 (95% CI, 0.047-0.175), RRT 0.325 (95% CI, 0.117-0.497), readmission 0.504 (95% CI, 0.388-0.604). Predictions were well calibrated, and the most important features within each model were consistent with clinical intuition.DiscussionOur models produce performant, well-calibrated, and interpretable predictions for COVID-19 patients at risk for the target outcomes. They demonstrate the potential to accurately estimate outcome prognosis in resource-constrained care sites managing COVID-19 patients.ConclusionsWe develop and validate prognostic models targeting MV, RRT, and readmission for hospitalized COVID-19 patients which produce accurate, interpretable predictions. Additional external validation studies are needed to further verify the generalizability of our results.     

Machine-learning model to predict the cause of death using a stacking ensemble method for observational data

ObjectiveCause of death is used as an important outcome of clinical research; however, access to cause-of-death data is limited. This study aimed to develop and validate a machine-learning model that predicts the cause of death from the patient’s last medical checkup.Materials and MethodsTo classify the mortality status and each individual cause of death, we used a stacking ensemble method. The prediction outcomes were all-cause mortality, 8 leading causes of death in South Korea, and other causes. The clinical data of study populations were extracted from the national claims (n = 174 747) and electronic health records (n = 729 065) and were used for model development and external validation. Moreover, we imputed the cause of death from the data of 3 US claims databases (n = 994 518, 995 372, and 407 604, respectively). All databases were formatted to the Observational Medical Outcomes Partnership Common Data Model.ResultsThe generalized area under the receiver operating characteristic curve (AUROC) of the model predicting the cause of death within 60 days was 0.9511. Moreover, the AUROC of the external validation was 0.8887. Among the causes of death imputed in the Medicare Supplemental database, 11.32% of deaths were due to malignant neoplastic disease.DiscussionThis study showed the potential of machine-learning models as a new alternative to address the lack of access to cause-of-death data. All processes were disclosed to maintain transparency, and the model was easily applicable to other institutions.ConclusionA machine-learning model with competent performance was developed to predict cause of death.     

Validation of the kidney failure risk equation in predicting the risk of progression to kidney failure in a multi-ethnic Singapore chronic kidney disease cohort

INTRODUCTIONThe Kidney Failure Risk Equation (KFRE) was developed to predict the risk of progression to end-stage kidney disease (ESKD). Although the KFRE has been validated in multinational cohorts, the Southeast Asian population was under-represented. This study aimed to validate the KFRE in a multi-ethnic Singapore chronic kidney disease (CKD) cohort.METHODSStage 3–5 CKD patients referred to the renal medicine department at Singapore General Hospital in 2009 were included. The primary outcome (time to ESKD) was traced until 30 June 2017. The eight- and four-variable KFRE (non-North America) models using age, gender, estimated glomerular filtration rate, urine albumin-creatinine ratio, serum albumin, phosphate, bicarbonate and calcium were validated in our cohort. Cox regression, likelihood ratio (Χ²), adequacy index, Harrell’s C-index and calibration curves were calculated to assess the predictive performance, discrimination and calibration of these models on the cohort.RESULTSA total of 1,128 patients were included. During the study period, 252 (22.3%) patients reached ESKD at a median time to ESKD of 84.8 (range 0.1–104.7) months. Both the eight- and four-variable KFRE models showed excellent predictive performance and discrimination (eight-variable: C-index 0.872, 95% confidence interval [CI] 0.850–0.894, adequacy index 97.3%; four-variable: C-index 0.874, 95% CI 0.852–0.896, adequacy index 97.9%). There was no incremental improvement in the prediction ability of the eight-variable model over the four-variable model in this cohort.CONCLUSIONThe KFRE was validated in a multi-ethnic Singapore CKD cohort. This risk score may help to identify patients requiring early renal care.     

An objective framework for evaluating unrecognized bias in medical AI models predicting COVID-19 outcomes

ObjectiveThe increasing translation of artificial intelligence (AI)/machine learning (ML) models into clinical practice brings an increased risk of direct harm from modeling bias; however, bias remains incompletely measured in many medical AI applications. This article aims to provide a framework for objective evaluation of medical AI from multiple aspects, focusing on binary classification models.Materials and MethodsUsing data from over 56 000 Mass General Brigham (MGB) patients with confirmed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), we evaluate unrecognized bias in 4 AI models developed during the early months of the pandemic in Boston, Massachusetts that predict risks of hospital admission, ICU admission, mechanical ventilation, and death after a SARS-CoV-2 infection purely based on their pre-infection longitudinal medical records. Models were evaluated both retrospectively and prospectively using model-level metrics of discrimination, accuracy, and reliability, and a novel individual-level metric for error.ResultsWe found inconsistent instances of model-level bias in the prediction models. From an individual-level aspect, however, we found most all models performing with slightly higher error rates for older patients.DiscussionWhile a model can be biased against certain protected groups (ie, perform worse) in certain tasks, it can be at the same time biased towards another protected group (ie, perform better). As such, current bias evaluation studies may lack a full depiction of the variable effects of a model on its subpopulations.ConclusionOnly a holistic evaluation, a diligent search for unrecognized bias, can provide enough information for an unbiased judgment of AI bias that can invigorate follow-up investigations on identifying the underlying roots of bias and ultimately make a change.     

The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression

ObjectiveMethods to correct class imbalance (imbalance between the frequency of outcome events and nonevents) are receiving increasing interest for developing prediction models. We examined the effect of imbalance correction on the performance of logistic regression models.Material and MethodsPrediction models were developed using standard and penalized (ridge) logistic regression under 4 methods to address class imbalance: no correction, random undersampling, random oversampling, and SMOTE. Model performance was evaluated in terms of discrimination, calibration, and classification. Using Monte Carlo simulations, we studied the impact of training set size, number of predictors, and the outcome event fraction. A case study on prediction modeling for ovarian cancer diagnosis is presented.ResultsThe use of random undersampling, random oversampling, or SMOTE yielded poorly calibrated models: the probability to belong to the minority class was strongly overestimated. These methods did not result in higher areas under the ROC curve when compared with models developed without correction for class imbalance. Although imbalance correction improved the balance between sensitivity and specificity, similar results were obtained by shifting the probability threshold instead.DiscussionImbalance correction led to models with strong miscalibration without better ability to distinguish between patients with and without the outcome event. The inaccurate probability estimates reduce the clinical utility of the model, because decisions about treatment are ill-informed.ConclusionOutcome imbalance is not a problem in itself, imbalance correction may even worsen model performance.     

Predicting pressure injury using nursing assessment phenotypes and machine learning methods

ObjectivePressure injuries are common and serious complications for hospitalized patients. The pressure injury rate is an important patient safety metric and an indicator of the quality of nursing care. Timely and accurate prediction of pressure injury risk can significantly facilitate early prevention and treatment and avoid adverse outcomes. While many pressure injury risk assessment tools exist, most were developed before there was access to large clinical datasets and advanced statistical methods, limiting their accuracy. In this paper, we describe the development of machine learning-based predictive models, using phenotypes derived from nurse-entered direct patient assessment data.MethodsWe utilized rich electronic health record data, including full assessment records entered by nurses, from 5 different hospitals affiliated with a large integrated healthcare organization to develop machine learning-based prediction models for pressure injury. Five-fold cross-validation was conducted to evaluate model performance.ResultsTwo pressure injury phenotypes were defined for model development: nonhospital acquired pressure injury (N = 4398) and hospital acquired pressure injury (N = 1767), representing 2 distinct clinical scenarios. A total of 28 clinical features were extracted and multiple machine learning predictive models were developed for both pressure injury phenotypes. The random forest model performed best and achieved an AUC of 0.92 and 0.94 in 2 test sets, respectively. The Glasgow coma scale, a nurse-entered level of consciousness measurement, was the most important feature for both groups.ConclusionsThis model accurately predicts pressure injury development and, if validated externally, may be helpful in widespread pressure injury prevention.     

Development and internal validation of China mortality prediction model in trauma based on ICD-10-CM lexicon: CMPMIT-ICD10

Background:Models to predict mortality in trauma play an important role in outcome prediction and severity adjustment, which informs trauma quality assessment and research. Hospitals in China typically use the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) to describe injury. However, there is no suitable prediction model for China. This study attempts to develop a new mortality prediction model based on the ICD-10-CM lexicon and a Chinese database.Methods:This retrospective study extracted the data of all trauma patients admitted to the Beijing Red Cross Emergency Center, from January 2012 to July 2018 (n = 40,205). We used relevant predictive variables to establish a prediction model following logistic regression analysis. The performance of the model was assessed based on discrimination and calibration. The bootstrapping method was used for internal validation and adjustment of model performance.Results:Sex, age, new region-severity codes, comorbidities, traumatic shock, and coma were finally included in the new model as key predictors of mortality. Among them, coma and traumatic shock had the highest scores in the model. The discrimination and calibration of this model were significant, and the internal validation performance was good. The values of the area under the curve and Brier score for the new model were 0.9640 and 0.0177, respectively; after adjustment of the bootstrapping method, they were 0.9630 and 0.0178, respectively.Conclusions:The new model (China Mortality Prediction Model in Trauma based on the ICD-10-CM lexicon) showed great discrimination and calibration, and performed well in internal validation; it should be further verified externally.     

Predicting brain function status changes in critically ill patients via Machine learning

ObjectiveIn intensive care units (ICUs), a patient’s brain function status can shift from a state of acute brain dysfunction (ABD) to one that is ABD-free and vice versa, which is challenging to forecast and, in turn, hampers the allocation of hospital resources. We aim to develop a machine learning model to predict next-day brain function status changes.Materials and MethodsUsing multicenter prospective adult cohorts involving medical and surgical ICU patients from 2 civilian and 3 Veteran Affairs hospitals, we trained and externally validated a light gradient boosting machine to predict brain function status changes. We compared the performances of the boosting model against state-of-the-art models—an ABD predictive model and its variants. We applied Shapley additive explanations to identify influential factors to develop a compact model.ResultsThere were 1026 critically ill patients without evidence of prior major dementia, or structural brain diseases, from whom 12 295 daily transitions (ABD: 5847 days; ABD-free: 6448 days) were observed. The boosting model achieved an area under the receiver-operating characteristic curve (AUROC) of 0.824 (95% confidence interval [CI], 0.821-0.827), compared with the state-of-the-art models of 0.697 (95% CI, 0.693-0.701) with P < .001. Using 13 identified top influential factors, the compact model achieved 99.4% of the boosting model on AUROC. The boosting and the compact models demonstrated high generalizability in external validation by achieving an AUROC of 0.812 (95% CI, 0.812-0.813).ConclusionThe inputs of the compact model are based on several simple questions that clinicians can quickly answer in practice, which demonstrates the model has direct prospective deployment potential into clinical practice, aiding in critical hospital resource allocation.     

Predicting hospitalization of COVID-19 positive patients using clinician-guided machine learning methods

ObjectivesThe coronavirus disease 2019 (COVID-19) is a resource-intensive global pandemic. It is important for healthcare systems to identify high-risk COVID-19-positive patients who need timely health care. This study was conducted to predict the hospitalization of older adults who have tested positive for COVID-19.MethodsWe screened all patients with COVID test records from 11 Mass General Brigham hospitals to identify the study population. A total of 1495 patients with age 65 and above from the outpatient setting were included in the final cohort, among which 459 patients were hospitalized. We conducted a clinician-guided, 3-stage feature selection, and phenotyping process using iterative combinations of literature review, clinician expert opinion, and electronic healthcare record data exploration. A list of 44 features, including temporal features, was generated from this process and used for model training. Four machine learning prediction models were developed, including regularized logistic regression, support vector machine, random forest, and neural network.ResultsAll 4 models achieved area under the receiver operating characteristic curve (AUC) greater than 0.80. Random forest achieved the best predictive performance (AUC = 0.83). Albumin, an index for nutritional status, was found to have the strongest association with hospitalization among COVID positive older adults.ConclusionsIn this study, we developed 4 machine learning models for predicting general hospitalization among COVID positive older adults. We identified important clinical factors associated with hospitalization and observed temporal patterns in our study cohort. Our modeling pipeline and algorithm could potentially be used to facilitate more accurate and efficient decision support for triaging COVID positive patients.     

Machine learning for initial insulin estimation in hospitalized patients

ObjectiveThe study sought to determine whether machine learning can predict initial inpatient total daily dose (TDD) of insulin from electronic health records more accurately than existing guideline-based dosing recommendations.Materials and MethodsUsing electronic health records from a tertiary academic center between 2008 and 2020 of 16,848 inpatients receiving subcutaneous insulin who achieved target blood glucose control of 100-180 mg/dL on a calendar day, we trained an ensemble machine learning algorithm consisting of regularized regression, random forest, and gradient boosted tree models for 2-stage TDD prediction. We evaluated the ability to predict patients requiring more than 6 units TDD and their point-value TDDs to achieve target glucose control.ResultsThe method achieves an area under the receiver-operating characteristic curve of 0.85 (95% confidence interval [CI], 0.84-0.87) and area under the precision-recall curve of 0.65 (95% CI, 0.64-0.67) for classifying patients who require more than 6 units TDD. For patients requiring more than 6 units TDD, the mean absolute percent error in dose prediction based on standard clinical calculators using patient weight is in the range of 136%-329%, while the regression model based on weight improves to 60% (95% CI, 57%-63%), and the full ensemble model further improves to 51% (95% CI, 48%-54%).DiscussionOwingto the narrow therapeutic window and wide individual variability, insulin dosing requires adaptive and predictive approaches that can be supported through data-driven analytic tools.ConclusionsMachine learning approaches based on readily available electronic medical records can discriminate which inpatients will require more than 6 units TDD and estimate individual doses more accurately than standard guidelines and practices.     

基于集成学习模型预测重症患者再入重症监护病房的风险

目的: 基于集成学习算法建立患者再入重症监护病房(intensive care unit, ICU)的风险预测模型,并比较各个模型的预测性能。方法: 使用美国重症医学数据库(medical information mart for intensive care,MIMIC)-Ⅲ,根据纳入、排除标准筛选患者,提取人口学特征、生命体征、实验室检查、合并症等可能对结局有预测作用的变量,基于集成学习方法随机森林、自适应提升算法(adaptive boosting, AdaBoost)和梯度提升决策树(gradient boosting decision tree, GBDT)建立再入ICU预测模型,并比较集成学习与Logistic回归的预测性能。使用五折交叉验证后的平均灵敏度、阳性预测值、阴性预测值、假阳性率、假阴性率、受试者工作特征曲线下面积(area under the receiver operating characteristic curve,AUROC)和Brier评分评价模型效果,基于最佳性能模型给出重要性排序前10位的预测变量。结果: 所有模型中,GBDT (AUROC=0.858)优于随机森林(AUROC=0.827),略好于AdaBoost (AUROC=0.851)。与Logistic回归(AUROC=0.810)相比,集成学习算法在区分度上均有较大的提升。GBDT算法给出的变量重要性排序中,平均动脉压、收缩压、舒张压、心率、尿量、血肌酐等变量排序靠前,相对而言,再入ICU患者的心血管功能和肾功能更差。结论: 基于集成学习算法的患者再入ICU预测模型表现出较好的性能,优于Logistic回归。使用集成学习算法建立的再入ICU风险预测模型可用于识别再入ICU风险高的患者,医务人员可针对高风险患者采取干预措施,改善患者的整体临床结局。     

Development and validation of a nomogram for predicting survival of breast cancer patients with ipsilateral supraclavicular lymph node metastasis

Background:Breast cancer patients with ipsilateral supraclavicular lymph node metastasis (ISLNM) but without distant metastasis are considered to have a poor prognosis. This study aimed to develop a nomogram to predict the overall survival (OS) of breast cancer patients with ISLNM but without distant metastasis.Methods:Medical records of breast cancer patients who received surgical treatment at the Affiliated Cancer Hospital of Zhengzhou University, Jiyuan People''s Hospital and Huaxian People''s Hospital between December 21, 2012 and June 30, 2020 were reviewed retrospectively. Overall, 345 patients with pathologically confirmed ISLNM and without evidence of distant metastasis were identified. They were further randomized 2:1 and divided into training (n = 231) and validation (n = 114) cohorts. A nomogram to predict the probability of OS was constructed based on clinicopathologic variables identified by the univariable and multivariable analyses. The predictive accuracy and discriminative ability were measured by calibration plots, concordance index (C-index), and risk group stratification.Results:Univariable analysis showed that estrogen receptor-positive (ER+), progesterone receptor-positive (PR+), human epidermal growth factor receptor 2-positive (HER2+) with Herceptin treatment, and a low axillary lymph node ratio (ALNR) were prognostic factors for better OS. PR+, HER2+ with Herceptin treatment, and a low ALNR remained independent prognostic factors for better OS on multivariable analysis. These variables were incorporated into a nomogram to predict the 1-, 3-, and 5-year OS of breast cancer patients with ISLNM. The C-indexes of the nomogram were 0.737 (95% confidence interval [CI]: 0.660–0.813) and 0.759 (95% CI: 0.636–0.881) for the training and the validation cohorts, respectively. The calibration plots presented excellent agreement between the nomogram prediction and actual observation for 3 and 5 years, but not 1 year, OS in both the cohorts. The nomogram was also able to stratify patients into different risk groups.Conclusions:In this study, we established and validated a novel nomogram for predicting survival of patients with ISLNM. This nomogram may, to some extent, allow clinicians to more accurately estimate prognosis and to make personalized therapeutic decisions for individual patients with ISLNM.     

Development of a “meta-model” to address missing data,predict patient-specific cancer survival and provide a foundation for clinical decision support

ObjectiveLike most real-world data, electronic health record (EHR)–derived data from oncology patients typically exhibits wide interpatient variability in terms of available data elements. This interpatient variability leads to missing data and can present critical challenges in developing and implementing predictive models to underlie clinical decision support for patient-specific oncology care. Here, we sought to develop a novel ensemble approach to addressing missing data that we term the “meta-model” and apply the meta-model to patient-specific cancer prognosis.Materials and MethodsUsing real-world data, we developed a suite of individual random survival forest models to predict survival in patients with advanced lung cancer, colorectal cancer, and breast cancer. Individual models varied by the predictor data used. We combined models for each cancer type into a meta-model that predicted survival for each patient using a weighted mean of the individual models for which the patient had all requisite predictors.ResultsThe meta-model significantly outperformed many of the individual models and performed similarly to the best performing individual models. Comparisons of the meta-model to a more traditional imputation-based method of addressing missing data supported the meta-model’s utility.ConclusionsWe developed a novel machine learning–based strategy to underlie clinical decision support and predict survival in cancer patients, despite missing data. The meta-model may more generally provide a tool for addressing missing data across a variety of clinical prediction problems. Moreover, the meta-model may address other challenges in clinical predictive modeling including model extensibility and integration of predictive algorithms trained across different institutions and datasets.     

Nomogram to predict pregnancy outcomes of emergency oocyte freeze-thaw cycles

Background:Existing clinical prediction models for in vitro fertilization are based on the fresh oocyte cycle, and there is no prediction model to evaluate the probability of successful thawing of cryopreserved mature oocytes. This research aims to identify and study the characteristics of pre-oocyte-retrieval patients that can affect the pregnancy outcomes of emergency oocyte freeze-thaw cycles.Methods:Data were collected from the Reproductive Center, Peking University Third Hospital of China. Multivariable logistic regression model was used to derive the nomogram. Nomogram model performance was assessed by examining the discrimination and calibration in the development and validation cohorts. Discriminatory ability was assessed using the area under the receiver operating characteristic curve (AUC), and calibration was assessed using the Hosmer–Lemeshow goodness-of-fit test and calibration plots.Results:The predictors in the model of “no transferable embryo cycles” are female age (odds ratio [OR] = 1.099, 95% confidence interval [CI] = 1.003–1.205, P = 0.0440), duration of infertility (OR = 1.140, 95% CI = 1.018–1.276, P = 0.0240), basal follicle-stimulating hormone (FSH) level (OR = 1.205, 95% CI = 1.051–1.382, P = 0.0084), basal estradiol (E2) level (OR = 1.006, 95% CI = 1.001–1.010, P = 0.0120), and sperm from microdissection testicular sperm extraction (MESA) (OR = 7.741, 95% CI = 2.905–20.632, P < 0.0010). Upon assessing predictive ability, the AUC for the “no transferable embryo cycles” model was 0.799 (95% CI: 0.722–0.875, P < 0.0010). The Hosmer–Lemeshow test (P = 0.7210) and calibration curve showed good calibration for the prediction of no transferable embryo cycles. The predictors in the cumulative live birth were the number of follicles on the day of human chorionic gonadotropin (hCG) administration (OR = 1.088, 95% CI = 1.030–1.149, P = 0.0020) and endometriosis (OR = 0.172, 95% CI = 0.035–0.853, P = 0.0310). The AUC for the “cumulative live birth” model was 0.724 (95% CI: 0.647–0.801, P < 0.0010). The Hosmer–Lemeshow test (P = 0.5620) and calibration curve showed good calibration for the prediction of cumulative live birth.Conclusions:The predictors in the final multivariate logistic regression models found to be significantly associated with poor pregnancy outcomes were increasing female age, duration of infertility, high basal FSH and E2 level, endometriosis, sperm from MESA, and low number of follicles with a diameter >10 mm on the day of hCG administration.     

Demonstrating an approach for evaluating synthetic geospatial and temporal epidemiologic data utility: results from analyzing >1.8 million SARS-CoV-2 tests in the United States National COVID Cohort Collaborative (N3C)

ObjectiveThis study sought to evaluate whether synthetic data derived from a national coronavirus disease 2019 (COVID-19) dataset could be used for geospatial and temporal epidemic analyses.Materials and MethodsUsing an original dataset (n = 1 854 968 severe acute respiratory syndrome coronavirus 2 tests) and its synthetic derivative, we compared key indicators of COVID-19 community spread through analysis of aggregate and zip code-level epidemic curves, patient characteristics and outcomes, distribution of tests by zip code, and indicator counts stratified by month and zip code. Similarity between the data was statistically and qualitatively evaluated.ResultsIn general, synthetic data closely matched original data for epidemic curves, patient characteristics, and outcomes. Synthetic data suppressed labels of zip codes with few total tests (mean = 2.9 ± 2.4; max = 16 tests; 66% reduction of unique zip codes). Epidemic curves and monthly indicator counts were similar between synthetic and original data in a random sample of the most tested (top 1%; n = 171) and for all unsuppressed zip codes (n = 5819), respectively. In small sample sizes, synthetic data utility was notably decreased.DiscussionAnalyses on the population-level and of densely tested zip codes (which contained most of the data) were similar between original and synthetically derived datasets. Analyses of sparsely tested populations were less similar and had more data suppression.ConclusionIn general, synthetic data were successfully used to analyze geospatial and temporal trends. Analyses using small sample sizes or populations were limited, in part due to purposeful data label suppression—an attribute disclosure countermeasure. Users should consider data fitness for use in these cases.     

影像组学联合预测模型在预测非小细胞肺癌淋巴结转移中的建立和应用价值

  目的  开发并验证影像组学模型,用于预测非小细胞肺癌术前淋巴结转移风险。  方法  2014年1月至2015年12月100例经临床病理确诊的非小细胞肺癌100例组成训练组,并用该数据建立影像组学预测模型。影像组学特征在平扫及增强CT上进行提取。Lasso-logistic模型用于数据降维、特征选择以及影像组学标记的建立。一致性系数（ICCs）用于评价观察者内部以及观察者之间的重复一致性。以一致性指数（C-index）评价影像组学标签对淋巴结转移的鉴别预测能力,并采用受试者工作特性（ROC）曲线下面积（AUC）展示。多因素logistic回归分析用于建立影像组学联合预测模型,该预测模型的参数包括影像组学标记和独立的临床危险因素。建立的影像组学模型由2016年1月至2017年12月连续纳入的100例非小细胞肺癌病例组成验证组进行验证。采用AUC评价该模型的鉴别预测效能,并用Delong检验进行模型间（联合预测模型与单纯使用22个影像组学标记的模型之间）的比较;用Hosmer-Lemeshow good of fit test（拟合优度检验）评价预测模型的校准度,其结果使用校正曲线表示,以比较模型预测的结果与实际淋巴结转移的一致性。  结果  提取特征时,观察者内部和观察者间的一致性好,ICC均大于0.75。从300个影像组学特征中提取出22个,其组成的影像组学标记,对于鉴别预测淋巴结转移状态的AUC,训练组为0.781,验证组为0.776。建立的影像组学预测模型包含了影像组学标记和血清癌胚抗原（CEA）、细胞角蛋白19片段抗原（CYFRA21-1）、癌抗原125（CA125）水平。用此联合预测模型预测淋巴结转移状态,训练组的AUC为0.836,验证组的AUC为0.821,均高于训练组和验证组单纯使用22个影像组学标记的模型,差异有统计学意义（P<0.05）。影像组学联合预测模型在训练组和验证组中均有较好的校准度,与实际淋巴结转移一致性高。  结论  本研究开发了一个包含了影像组学特征、临床危险因素的影像组学联合预测模型,该模型能够直观预测非小细胞肺癌患者术前的淋巴结转移风险。     

Characterizing phenotypic abnormalities associated with high-risk individuals developing lung cancer using electronic health records from the All of Us researcher workbench

ObjectiveThe study sought to test the feasibility of conducting a phenome-wide association study to characterize phenotypic abnormalities associated with individuals at high risk for lung cancer using electronic health records.Materials and MethodsWe used the beta release of the All of Us Researcher Workbench with clinical and survey data from a population of 225 000 subjects. We identified 3 cohorts of individuals at high risk to develop lung cancer based on (1) the 2013 U.S. Preventive Services Task Force criteria, (2) the long-term quitters of cigarette smoking criteria, and (3) the younger age of onset criteria. We applied the logistic regression analysis to identify the significant associations between individuals’ phenotypes and their risk categories. We validated our findings against a lung cancer cohort from the same population and conducted an expert review to understand whether these associations are known or potentially novel.ResultsWe found a total of 214 statistically significant associations (P < .05 with a Bonferroni correction and odds ratio > 1.5) enriched in the high-risk individuals from 3 cohorts, and 15 enriched in the low-risk individuals. Forty significant associations enriched in the high-risk individuals and 13 enriched in the low-risk individuals were validated in the cancer cohort. Expert review identified 15 potentially new associations enriched in the high-risk individuals.ConclusionsIt is feasible to conduct a phenome-wide association study to characterize phenotypic abnormalities associated in high-risk individuals developing lung cancer using electronic health records. The All of Us Research Workbench is a promising resource for the research studies to evaluate and optimize lung cancer screening criteria.     

A theory-based meta-regression of factors influencing clinical decision support adoption and implementation

ObjectiveThe purpose of the study was to explore the theoretical underpinnings of effective clinical decision support (CDS) factors using the comparative effectiveness results.Materials and MethodsWe leveraged search results from a previous systematic literature review and updated the search to screen articles published from January 2017 to January 2020. We included randomized controlled trials and cluster randomized controlled trials that compared a CDS intervention with and without specific factors. We used random effects meta-regression procedures to analyze clinician behavior for the aggregate effects. The theoretical model was the Unified Theory of Acceptance and Use of Technology (UTAUT) model with motivational control.ResultsThirty-four studies were included. The meta-regression models identified the importance of effort expectancy (estimated coefficient = −0.162; P = .0003); facilitating conditions (estimated coefficient = 0.094; P = .013); and performance expectancy with motivational control (estimated coefficient = 1.029; P = .022). Each of these factors created a significant impact on clinician behavior. The meta-regression model with the multivariate analysis explained a large amount of the heterogeneity across studies (R2 =88.32%).DiscussionThree positive factors were identified: low effort to use, low controllability, and providing more infrastructure and implementation strategies to support the CDS. The multivariate analysis suggests that passive CDS could be effective if users believe the CDS is useful and/or social expectations to use the CDS intervention exist.ConclusionsOverall, a modified UTAUT model that includes motivational control is an appropriate model to understand psychological factors associated with CDS effectiveness and to guide CDS design, implementation, and optimization.     

Developing machine learning models to personalize care levels among emergency room patients for hospital admission

ObjectiveTo develop prediction models for intensive care unit (ICU) vs non-ICU level-of-care need within 24 hours of inpatient admission for emergency department (ED) patients using electronic health record data.Materials and MethodsUsing records of 41 654 ED visits to a tertiary academic center from 2015 to 2019, we tested 4 algorithms—feed-forward neural networks, regularized regression, random forests, and gradient-boosted trees—to predict ICU vs non-ICU level-of-care within 24 hours and at the 24th hour following admission. Simple-feature models included patient demographics, Emergency Severity Index (ESI), and vital sign summary. Complex-feature models added all vital signs, lab results, and counts of diagnosis, imaging, procedures, medications, and lab orders.ResultsThe best-performing model, a gradient-boosted tree using a full feature set, achieved an AUROC of 0.88 (95%CI: 0.87–0.89) and AUPRC of 0.65 (95%CI: 0.63–0.68) for predicting ICU care need within 24 hours of admission. The logistic regression model using ESI achieved an AUROC of 0.67 (95%CI: 0.65–0.70) and AUPRC of 0.37 (95%CI: 0.35–0.40). Using a discrimination threshold, such as 0.6, the positive predictive value, negative predictive value, sensitivity, and specificity were 85%, 89%, 30%, and 99%, respectively. Vital signs were the most important predictors.Discussion and ConclusionsUndertriaging admitted ED patients who subsequently require ICU care is common and associated with poorer outcomes. Machine learning models using readily available electronic health record data predict subsequent need for ICU admission with good discrimination, substantially better than the benchmarking ESI system. The results could be used in a multitiered clinical decision-support system to improve ED triage.