Comparison of approaches for incorporating new information into existing risk prediction models

We compare the calibration and variability of risk prediction models that were estimated using various approaches for combining information on new predictors, termed ‘markers’, with parameter information available for other variables from an earlier model, which was estimated from a large data source. We assess the performance of risk prediction models updated based on likelihood ratio (LR) approaches that incorporate dependence between new and old risk factors as well as approaches that assume independence (‘naive Bayes’ methods). We study the impact of estimating the LR by (i) fitting a single model to cases and non‐cases when the distribution of the new markers is in the exponential family or (ii) fitting separate models to cases and non‐cases. We also evaluate a new constrained maximum likelihood method. We study updating the risk prediction model when the new data arise from a cohort and extend available methods to accommodate updating when the new data source is a case‐control study. To create realistic correlations between predictors, we also based simulations on real data on response to antiviral therapy for hepatitis C. From these studies, we recommend the LR method fit using a single model or constrained maximum likelihood. Copyright © 2016 John Wiley & Sons, Ltd.     

How to evaluate the calibration of a disease risk prediction tool

To evaluate the calibration of a disease risk prediction tool, the quantity E/O, i.e. the ratio of the expected to the observed number of events, is usually computed. However, because of censoring, or more precisely because of individuals who drop out before the termination of the study, this quantity is generally unavailable for the complete population study and an alternative estimate has to be computed. In this paper, we present and compare four methods to do this. We show that two of the most commonly used methods generally lead to biased estimates. Our arguments are first based on some theoretic considerations. Then, we perform a simulation study to highlight the magnitude of biases. As a concluding example, we evaluate the calibration of an existing predictive model for breast cancer on the E3N–EPIC cohort. Copyright © 2009 John Wiley & Sons, Ltd.     

Semi‐varying coefficient multinomial logistic regression for disease progression risk prediction

This paper proposes a risk prediction model using semi‐varying coefficient multinomial logistic regression. We use a penalized local likelihood method to do the model selection and estimate both functional and constant coefficients in the selected model. The model can be used to improve predictive modelling when non‐linear interactions between predictors are present. We conduct a simulation study to assess our method's performance, and the results show that the model selection procedure works well with small average numbers of wrong‐selection or missing‐selection. We illustrate the use of our method by applying it to classify the patients with early rheumatoid arthritis at baseline into different risk groups in future disease progression. We use a leave‐one‐out cross‐validation method to assess its correct prediction rate and propose a recalibration framework to evaluate how reliable are the predicted risks. Copyright © 2016 John Wiley & Sons, Ltd.     

Practical implementation of frailty models in Mendelian risk prediction

There are numerous statistical models used to identify individuals at high risk of cancer due to inherited mutations. Mendelian models predict future risk of cancer by using family history with estimated cancer penetrances (age- and sex-specific risk of cancer given the genotype of the mutations) and mutation prevalences. However, there is often residual risk heterogeneity across families even after accounting for the mutations in the model, due to environmental or unobserved genetic risk factors. We aim to improve Mendelian risk prediction by incorporating a frailty model that contains a family-specific frailty vector, impacting the cancer hazard function, to account for this heterogeneity. We use a discrete uniform population frailty distribution and implement a marginalized approach that averages each family's risk predictions over the family's frailty distribution. We apply the proposed approach to improve breast cancer prediction in BRCAPRO, a Mendelian model that accounts for inherited mutations in the BRCA1 and BRCA2 genes to predict breast and ovarian cancer. We evaluate the proposed model's performance in simulations and real data from the Cancer Genetics Network and show improvements in model calibration and discrimination. We also discuss alternative approaches for incorporating frailties and their strengths and limitations.     

Clinical prediction models to predict the risk of multiple binary outcomes: a comparison of approaches

Clinical prediction models (CPMs) can predict clinically relevant outcomes or events. Typically, prognostic CPMs are derived to predict the risk of a single future outcome. However, there are many medical applications where two or more outcomes are of interest, meaning this should be more widely reflected in CPMs so they can accurately estimate the joint risk of multiple outcomes simultaneously. A potentially naïve approach to multi‐outcome risk prediction is to derive a CPM for each outcome separately, then multiply the predicted risks. This approach is only valid if the outcomes are conditionally independent given the covariates, and it fails to exploit the potential relationships between the outcomes. This paper outlines several approaches that could be used to develop CPMs for multiple binary outcomes. We consider four methods, ranging in complexity and conditional independence assumptions: namely, probabilistic classifier chain, multinomial logistic regression, multivariate logistic regression, and a Bayesian probit model. These are compared with methods that rely on conditional independence: separate univariate CPMs and stacked regression. Employing a simulation study and real‐world example, we illustrate that CPMs for joint risk prediction of multiple outcomes should only be derived using methods that model the residual correlation between outcomes. In such a situation, our results suggest that probabilistic classification chains, multinomial logistic regression or the Bayesian probit model are all appropriate choices. We call into question the development of CPMs for each outcome in isolation when multiple correlated or structurally related outcomes are of interest and recommend more multivariate approaches to risk prediction.     

Sample size considerations and predictive performance of multinomial logistic prediction models

Multinomial Logistic Regression (MLR) has been advocated for developing clinical prediction models that distinguish between three or more unordered outcomes. We present a full-factorial simulation study to examine the predictive performance of MLR models in relation to the relative size of outcome categories, number of predictors and the number of events per variable. It is shown that MLR estimated by Maximum Likelihood yields overfitted prediction models in small to medium sized data. In most cases, the calibration and overall predictive performance of the multinomial prediction model is improved by using penalized MLR. Our simulation study also highlights the importance of events per variable in the multinomial context as well as the total sample size. As expected, our study demonstrates the need for optimism correction of the predictive performance measures when developing the multinomial logistic prediction model. We recommend the use of penalized MLR when prediction models are developed in small data sets or in medium sized data sets with a small total sample size (ie, when the sizes of the outcome categories are balanced). Finally, we present a case study in which we illustrate the development and validation of penalized and unpenalized multinomial prediction models for predicting malignancy of ovarian cancer.     

The use of repeated blood pressure measures for cardiovascular risk prediction: a comparison of statistical models in the ARIC study

Many prediction models have been developed for the risk assessment and the prevention of cardiovascular disease in primary care. Recent efforts have focused on improving the accuracy of these prediction models by adding novel biomarkers to a common set of baseline risk predictors. Few have considered incorporating repeated measures of the common risk predictors. Through application to the Atherosclerosis Risk in Communities study and simulations, we compare models that use simple summary measures of the repeat information on systolic blood pressure, such as (i) baseline only; (ii) last observation carried forward; and (iii) cumulative mean, against more complex methods that model the repeat information using (iv) ordinary regression calibration; (v) risk‐set regression calibration; and (vi) joint longitudinal and survival models. In comparison with the baseline‐only model, we observed modest improvements in discrimination and calibration using the cumulative mean of systolic blood pressure, but little further improvement from any of the complex methods. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

Calibration plots for risk prediction models in the presence of competing risks

A predicted risk of 17% can be called reliable if it can be expected that the event will occur to about 17 of 100 patients who all received a predicted risk of 17%. Statistical models can predict the absolute risk of an event such as cardiovascular death in the presence of competing risks such as death due to other causes. For personalized medicine and patient counseling, it is necessary to check that the model is calibrated in the sense that it provides reliable predictions for all subjects. There are three often encountered practical problems when the aim is to display or test if a risk prediction model is well calibrated. The first is lack of independent validation data, the second is right censoring, and the third is that when the risk scale is continuous, the estimation problem is as difficult as density estimation. To deal with these problems, we propose to estimate calibration curves for competing risks models based on jackknife pseudo‐values that are combined with a nearest neighborhood smoother and a cross‐validation approach to deal with all three problems. Copyright © 2014 John Wiley & Sons, Ltd.     

Evaluating health risk models

Interest in targeted disease prevention has stimulated development of models that assign risks to individuals, using their personal covariates. We need to evaluate these models and quantify the gains achieved by expanding a model to include additional covariates. This paper reviews several performance measures and shows how they are related. Examples are used to show that appropriate performance criteria for a risk model depend upon how the model is used. Application of the performance measures to risk models for hypothetical populations and for US women at risk of breast cancer illustrate two additional points. First, model performance is constrained by the distribution of risk‐determining covariates in the population. This complicates the comparison of two models when applied to populations with different covariate distributions. Second, all summary performance measures obscure model features of relevance to its utility for the application at hand, such as performance in specific subgroups of the population. In particular, the precision gained by adding covariates to a model can be small overall, but large in certain subgroups. We propose new ways to identify these subgroups and to quantify how much they gain by measuring the additional covariates. Those with largest gains could be targeted for cost‐efficient covariate assessment. Copyright © 2010 John Wiley & Sons, Ltd.     

老年人群心血管疾病风险预测模型研究进展

风险预测模型（模型）对于老年人群心血管疾病（CVD）的一级预防具有重要意义。国内外针对老年人群构建的CVD模型共检索到15篇文献。模型的结局定义差异较大;10个模型报告时缺少方法、结果的重要信息;10个模型存在高偏倚风险;13个模型在内部验证时仅表现出中等区分度;仅有4个模型经过外部验证。老年人群CVD模型在模型算法、预测因子与结局的关联强度方面与一般人群模型存在差异,且老年人群模型的预测能力有所下降。未来仍需补充高质量的外部验证研究证据,并探索增加新的预测因子、采用竞争风险模型算法、机器学习算法、联合模型算法、改变预测时间范围等途径对模型进行优化。     

Flexible recalibration of binary clinical prediction models

Calibration in binary prediction models, that is, the agreement between model predictions and observed outcomes, is an important aspect of assessing the models' utility for characterizing risk in future data. A popular technique for assessing model calibration first proposed by D. R. Cox in 1958 involves fitting a logistic model incorporating an intercept and a slope coefficient for the logit of the estimated probability of the outcome; good calibration is evident if these parameters do not appreciably differ from 0 and 1, respectively. However, in practice, the form of miscalibration may sometimes be more complicated. In this article, we expand the Cox calibration model to allow for more general parameterizations and derive a relative measure of miscalibration between two competing models from this more flexible model. We present an example implementation using data from the US Agency for Healthcare Research and Quality. Copyright © 2012 John Wiley & Sons, Ltd.     

NDR和DIAL模型在宁波市社区2型糖尿病人群中预测心血管病发生风险的应用

目的在我国社区2型糖尿病人群中独立验证并比较基于瑞典糖尿病登记数据(NDR)建立的心血管病短期风险预测模型和糖尿病终生风险预测(DIAL)模型评估5年和10年心血管病发生风险的准确性。方法研究对象为2010年1月1日至2020年12月31日在中国鄞州电子健康档案研究中的基线无心血管病史且年龄在30～75岁的2型糖尿病队列人群。采用校准后的NDR模型评估研究对象5年心血管病风险, 采用DIAL模型评估5年和10年心血管病发生风险, 采用调整竞争风险的Kaplan-Meier法计算研究对象5年和10年心血管病实际发生风险。采用区分度C统计量、校准度χ2值和校准图评价预测模型的准确性。结果经过中位7.0年的随访, 83 503名研究对象共发生7 326例心血管病事件和2 937例非心血管病死亡事件。在5年心血管病风险预测中, NDR模型对男性和女性发病风险分别高估39.4%和8.6%, DIAL模型分别高估14.6%和50.1%。在男性5年风险预测中DIAL模型区分度优于NDR模型, 其C统计量(95%CI)分别为0.681(0.672～0.690)和0.667(0.657～0.677);女...     

Extending the c-statistic to nominal polytomous outcomes: the Polytomous Discrimination Index

Diagnostic problems in medicine are sometimes polytomous, meaning that the outcome has more than two distinct categories. For example, ovarian tumors can be benign, borderline, primary invasive, or metastatic. Extending the main measure of binary discrimination, the c‐statistic or area under the ROC curve, to nominal polytomous settings is not straightforward. This paper reviews existing measures and presents the polytomous discrimination index (PDI) as an alternative. The PDI assesses all sets of k cases consisting of one case from each outcome category. For each category i (i = 1, … ,k), it is assessed whether the risk of category i is highest for the case from category i. A score of 1∕k is given per category for which this holds, yielding a set score between 0 and 1 to indicate the level of discrimination. The PDI is the average set score and is interpreted as the probability to correctly identify a case from a randomly selected category within a set of k cases. This probability can be split up by outcome category, yielding k category‐specific values that result in the PDI when averaged. We demonstrate the measures on two diagnostic problems (residual mass histology after chemotherapy for testicular cancer; diagnosis of ovarian tumors). We compare the behavior of the measures on theoretical data, showing that PDI is more strongly influenced by simultaneous discrimination between all categories than by partial discrimination between pairs of categories. In conclusion, the PDI is attractive because it better matches the requirements of a measure to summarize polytomous discrimination. Copyright © 2012 John Wiley & Sons, Ltd.     

Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers

Predicting the probability of the occurrence of a binary outcome or condition is important in biomedical research. While assessing discrimination is an essential issue in developing and validating binary prediction models, less attention has been paid to methods for assessing model calibration. Calibration refers to the degree of agreement between observed and predicted probabilities and is often assessed by testing for lack‐of‐fit. The objective of our study was to examine the ability of graphical methods to assess the calibration of logistic regression models. We examined lack of internal calibration, which was related to misspecification of the logistic regression model, and external calibration, which was related to an overfit model or to shrinkage of the linear predictor. We conducted an extensive set of Monte Carlo simulations with a locally weighted least squares regression smoother (i.e., the loess algorithm) to examine the ability of graphical methods to assess model calibration. We found that loess‐based methods were able to provide evidence of moderate departures from linearity and indicate omission of a moderately strong interaction. Misspecification of the link function was harder to detect. Visual patterns were clearer with higher sample sizes, higher incidence of the outcome, or higher discrimination. Loess‐based methods were also able to identify the lack of calibration in external validation samples when an overfit regression model had been used. In conclusion, loess‐based smoothing methods are adequate tools to graphically assess calibration and merit wider application. © 2013 The Authors. Statistics in Medicine published by John Wiley & Sons, Ltd     

A new synthesis analysis method for building logistic regression prediction models

Synthesis analysis refers to a statistical method that integrates multiple univariate regression models and the correlation between each pair of predictors into a single multivariate regression model. The practical application of such a method could be developing a multivariate disease prediction model where a dataset containing the disease outcome and every predictor of interest is not available. In this study, we propose a new version of synthesis analysis that is specific to binary outcomes. We show that our proposed method possesses desirable statistical properties. We also conduct a simulation study to assess the robustness of the proposed method and compare it to a competing method. Copyright © 2014 John Wiley & Sons, Ltd.     

Absolute risk regression for competing risks: interpretation,link functions,and prediction

In survival analysis with competing risks, the transformation model allows different functions between the outcome and explanatory variables. However, the model's prediction accuracy and the interpretation of parameters may be sensitive to the choice of link function. We review the practical implications of different link functions for regression of the absolute risk (or cumulative incidence) of an event. Specifically, we consider models in which the regression coefficients β have the following interpretation: The probability of dying from cause D during the next t years changes with a factor exp(β) for a one unit change of the corresponding predictor variable, given fixed values for the other predictor variables. The models have a direct interpretation for the predictive ability of the risk factors. We propose some tools to justify the models in comparison with traditional approaches that combine a series of cause‐specific Cox regression models or use the Fine–Gray model. We illustrate the methods with the use of bone marrow transplant data. Copyright © 2012 John Wiley & Sons, Ltd.     

非综合征型唇腭裂的遗传预测模型研究进展

非综合征型唇腭裂(NSOC)是我国常见的出生缺陷。近年来, 随着我国生育政策相继调整两次, 与高龄生育伴发的出生缺陷防控形势日益严峻。开展NSOC风险预测将为健全出生缺陷防控链条提供重要证据。近年来, 全基因组关联研究和二代测序等发现了多个与NSOC有关的遗传位点, 为开展预测提供了有益信息。本文综述了NSOC风险预测, 特别是利用遗传信息开展风险预测的常用方法, 以期在研究设计、变量筛选、构建策略及评价方法等方面, 为进一步开发和完善NSOC等复杂出生缺陷的风险预测模型提供参考。     

Graphical calibration curves and the integrated calibration index (ICI) for survival models

In the context of survival analysis, calibration refers to the agreement between predicted probabilities and observed event rates or frequencies of the outcome within a given duration of time. We aimed to describe and evaluate methods for graphically assessing the calibration of survival models. We focus on hazard regression models and restricted cubic splines in conjunction with a Cox proportional hazards model. We also describe modifications of the Integrated Calibration Index, of E50 and of E90. In this context, this is the average (respectively, median or 90th percentile) absolute difference between predicted survival probabilities and smoothed survival frequencies. We conducted a series of Monte Carlo simulations to evaluate the performance of these calibration measures when the underlying model has been correctly specified and under different types of model mis-specification. We illustrate the utility of calibration curves and the three calibration metrics by using them to compare the calibration of a Cox proportional hazards regression model with that of a random survival forest for predicting mortality in patients hospitalized with heart failure. Under a correctly specified regression model, differences between the two methods for constructing calibration curves were minimal, although the performance of the method based on restricted cubic splines tended to be slightly better. In contrast, under a mis-specified model, the smoothed calibration curved constructed using hazard regression tended to be closer to the true calibration curve. The use of calibration curves and of these numeric calibration metrics permits for a comprehensive comparison of the calibration of competing survival models.     

Several methods to assess improvement in risk prediction models: Extension to survival analysis

Risk prediction models have been widely applied for the prediction of long‐term incidence of disease. Several parameters have been identified and estimators developed to quantify the predictive ability of models and to compare new models with traditional models. These estimators have not generally accounted for censoring in the survival data normally available for fitting the models. This paper remedies that problem. The primary parameters considered are net reclassification improvement (NRI) and integrated discrimination improvement (IDI). We have previously similarly considered a primary measure of concordance, area under the ROC curve (AUC), also called the c‐statistic. We also include here consideration of population attributable risk (PAR) and ratio of predicted risk in the top quintile of risk to that in the bottom quintile. We evaluated estimators of these various parameters both with simulation studies and also as applied to a prospective study of coronary heart disease (CHD). Our simulation studies showed that in general our estimators had little bias, and less bias and smaller variances than the traditional estimators. We have applied our methods to assessing improvement in risk prediction for each traditional CHD risk factor compared to a model without that factor. These traditional risk factors are considered valuable, yet when adding any of them to a risk prediction model that has omitted the one factor, the improvement is generally small for any of the parameters. This experience should prepare us to not expect large values of the risk prediction improvement evaluation parameters for any new risk factor to be discovered. Copyright © 2010 John Wiley & Sons, Ltd.     

Assessing the goodness of fit of personal risk models

We describe a flexible family of tests for evaluating the goodness of fit (calibration) of a pre‐specified personal risk model to the outcomes observed in a longitudinal cohort. Such evaluation involves using the risk model to assign each subject an absolute risk of developing the outcome within a given time from cohort entry and comparing subjects’ assigned risks with their observed outcomes. This comparison involves several issues. For example, subjects followed only for part of the risk period have unknown outcomes. Moreover, existing tests do not reveal the reasons for poor model fit when it occurs, which can reflect misspecification of the model's hazards for the competing risks of outcome development and death. To address these issues, we extend the model‐specified hazards for outcome and death, and use score statistics to test the null hypothesis that the extensions are unnecessary. Simulated cohort data applied to risk models whose outcome and mortality hazards agreed and disagreed with those generating the data show that the tests are sensitive to poor model fit, provide insight into the reasons for poor fit, and accommodate a wide range of model misspecification. We illustrate the methods by examining the calibration of two breast cancer risk models as applied to a cohort of participants in the Breast Cancer Family Registry. The methods can be implemented using the Risk Model Assessment Program, an R package freely available at http://stanford.edu/~ggong/rmap/ . Copyright © 2014 John Wiley & Sons, Ltd.