Flexible recalibration of binary clinical prediction models

Calibration in binary prediction models, that is, the agreement between model predictions and observed outcomes, is an important aspect of assessing the models' utility for characterizing risk in future data. A popular technique for assessing model calibration first proposed by D. R. Cox in 1958 involves fitting a logistic model incorporating an intercept and a slope coefficient for the logit of the estimated probability of the outcome; good calibration is evident if these parameters do not appreciably differ from 0 and 1, respectively. However, in practice, the form of miscalibration may sometimes be more complicated. In this article, we expand the Cox calibration model to allow for more general parameterizations and derive a relative measure of miscalibration between two competing models from this more flexible model. We present an example implementation using data from the US Agency for Healthcare Research and Quality. Copyright © 2012 John Wiley & Sons, Ltd.     

Factors affecting power of tests for multiple binary outcomes

Frequently in clinical studies a primary outcome is formulated from a vector of binary events. Several methods exist to assess treatment effects on multiple correlated binary outcomes, including comparing groups on the occurrence of at least one among the outcomes (‘collapsed composite’), on the count of outcomes observed per subject, on individual outcomes adjusting for multiplicity, or with multivariate tests postulating either common or distinct effects across outcomes. We focus on a 1‐df distinct effects test in which the estimated outcome‐specific treatment effects from a GEE model are simply averaged, and compare it with other methods on clinical and statistical grounds. Using a flexible method to simulate multivariate binary data, we show that the relative efficiencies of the assessed tests depend in a complex way on the magnitudes and variabilities of component incidences and treatment effects, as well as correlations among component events. While other tests are easily ‘driven’ by high‐frequency components, the average effect GEE test is not, since it averages the log odds ratios unweighted by the component frequencies. Thus, the average effect test is relatively more powerful than other tests when lower frequency components have stronger associations with a treatment or other predictor, but less powerful when higher frequency components are more strongly associated. In studies when relative effects are at least as important as absolute effects, or when lower frequency components are clinically most important, this test may be preferred. Two clinical trials are discussed and analyzed, and recommendations for practice are made. Copyright © 2010 John Wiley & Sons, Ltd.     

Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes

When designing a study to develop a new prediction model with binary or time-to-event outcomes, researchers should ensure their sample size is adequate in terms of the number of participants (n) and outcome events (E) relative to the number of predictor parameters (p) considered for inclusion. We propose that the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP) should be calculated to meet the following three criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥ 0.9, (ii) small absolute difference of ≤ 0.05 in the model's apparent and adjusted Nagelkerke's R² , and (iii) precise estimation of the overall risk in the population. Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p, and require prespecification of the model's anticipated Cox-Snell R² , which we show can be obtained from previous studies. The values of n and E that meet all three criteria provides the minimum sample size required for model development. Upon application of our approach, a new diagnostic model for Chagas disease requires an EPP of at least 4.8 and a new prognostic model for recurrent venous thromboembolism requires an EPP of at least 23. This reinforces why rules of thumb (eg, 10 EPP) should be avoided. Researchers might additionally ensure the sample size gives precise estimates of key predictor effects; this is especially important when key categorical predictors have few events in some categories, as this may substantially increase the numbers required.     

A comparison of bayesian and frequentist approaches to incorporating external information for the prediction of prostate cancer risk

We present the most comprehensive comparison to date of the predictive benefit of genetics in addition to currently used clinical variables, using genotype data for 33 single‐nucleotide polymorphisms (SNPs) in 1,547 Caucasian men from the placebo arm of the REduction by DUtasteride of prostate Cancer Events (REDUCE®) trial. Moreover, we conducted a detailed comparison of three techniques for incorporating genetics into clinical risk prediction. The first method was a standard logistic regression model, which included separate terms for the clinical covariates and for each of the genetic markers. This approach ignores a substantial amount of external information concerning effect sizes for these Genome Wide Association Study (GWAS)‐replicated SNPs. The second and third methods investigated two possible approaches to incorporating meta‐analysed external SNP effect estimates – one via a weighted PCa 'risk' score based solely on the meta analysis estimates, and the other incorporating both the current and prior data via informative priors in a Bayesian logistic regression model. All methods demonstrated a slight improvement in predictive performance upon incorporation of genetics. The two methods that incorporated external information showed the greatest receiver‐operating‐characteristic AUCs increase from 0.61 to 0.64. The value of our methods comparison is likely to lie in observations of performance similarities, rather than difference, between three approaches of very different resource requirements. The two methods that included external information performed best, but only marginally despite substantial differences in complexity.     

Aggregating published prediction models with individual participant data: a comparison of different approaches

During the recent decades, interest in prediction models has substantially increased, but approaches to synthesize evidence from previously developed models have failed to keep pace. This causes researchers to ignore potentially useful past evidence when developing a novel prediction model with individual participant data (IPD) from their population of interest. We aimed to evaluate approaches to aggregate previously published prediction models with new data. We consider the situation that models are reported in the literature with predictors similar to those available in an IPD dataset. We adopt a two‐stage method and explore three approaches to calculate a synthesis model, hereby relying on the principles of multivariate meta‐analysis. The former approach employs a naive pooling strategy, whereas the latter accounts for within‐study and between‐study covariance. These approaches are applied to a collection of 15 datasets of patients with traumatic brain injury, and to five previously published models for predicting deep venous thrombosis. Here, we illustrated how the generally unrealistic assumption of consistency in the availability of evidence across included studies can be relaxed. Results from the case studies demonstrate that aggregation yields prediction models with an improved discrimination and calibration in a vast majority of scenarios, and result in equivalent performance (compared with the standard approach) in a small minority of situations. The proposed aggregation approaches are particularly useful when few participant data are at hand. Assessing the degree of heterogeneity between IPD and literature findings remains crucial to determine the optimal approach in aggregating previous evidence into new prediction models. Copyright © 2012 John Wiley & Sons, Ltd.     

A method for the meta-analysis of mutually exclusive binary outcomes

Meta-analyses of multiple outcomes need to take into account the within-study correlation across the different outcomes. Here we focus on the meta-analysis of dichotomous outcomes that are mutually exclusive and exhaustive. Correlations between effect sizes for mutually exclusive outcomes are negative and can be obtained from data already available. We present both fixed-effects and random-effects methods that account for the negative correlations and yield correct simultaneous confidence intervals for both the marginal outcome-specific effect sizes and the relative effect sizes between outcomes. Formulae for the odds ratio, risk ratio, risk difference, and the differences in the arcsin-transformed risks are provided. An example of a meta-analysis of randomized trials of radiotherapy and mastectomy with axillary lymph node clearance versus only mastectomy with axillary clearance for early breast cancer is presented. The mutually exclusive outcomes of breast cancer deaths and deaths secondary to other causes are examined in separate meta-analyses, and also by taking the between-outcome correlation into account. We argue that mutually exclusive outcomes in the meta-analyses of binary data are optimally analyzed in a multinomial setting. This may also be applicable when a meta-analysis examines only one out of several mutually exclusive outcomes. For large sample sizes and/or low event counts, the covariances between outcome-specific effect sizes are small, and either ignoring them or accounting for them would result in similar estimates for any practical purpose. However, meta-analysts should explore the robustness of the findings from individual meta-analyses when mutually exclusive outcomes are assessed.     

Comparison of approaches for incorporating new information into existing risk prediction models

We compare the calibration and variability of risk prediction models that were estimated using various approaches for combining information on new predictors, termed ‘markers’, with parameter information available for other variables from an earlier model, which was estimated from a large data source. We assess the performance of risk prediction models updated based on likelihood ratio (LR) approaches that incorporate dependence between new and old risk factors as well as approaches that assume independence (‘naive Bayes’ methods). We study the impact of estimating the LR by (i) fitting a single model to cases and non‐cases when the distribution of the new markers is in the exponential family or (ii) fitting separate models to cases and non‐cases. We also evaluate a new constrained maximum likelihood method. We study updating the risk prediction model when the new data arise from a cohort and extend available methods to accommodate updating when the new data source is a case‐control study. To create realistic correlations between predictors, we also based simulations on real data on response to antiviral therapy for hepatitis C. From these studies, we recommend the LR method fit using a single model or constrained maximum likelihood. Copyright © 2016 John Wiley & Sons, Ltd.     

A new synthesis analysis method for building logistic regression prediction models

Synthesis analysis refers to a statistical method that integrates multiple univariate regression models and the correlation between each pair of predictors into a single multivariate regression model. The practical application of such a method could be developing a multivariate disease prediction model where a dataset containing the disease outcome and every predictor of interest is not available. In this study, we propose a new version of synthesis analysis that is specific to binary outcomes. We show that our proposed method possesses desirable statistical properties. We also conduct a simulation study to assess the robustness of the proposed method and compare it to a competing method. Copyright © 2014 John Wiley & Sons, Ltd.     

The use of repeated blood pressure measures for cardiovascular risk prediction: a comparison of statistical models in the ARIC study

Many prediction models have been developed for the risk assessment and the prevention of cardiovascular disease in primary care. Recent efforts have focused on improving the accuracy of these prediction models by adding novel biomarkers to a common set of baseline risk predictors. Few have considered incorporating repeated measures of the common risk predictors. Through application to the Atherosclerosis Risk in Communities study and simulations, we compare models that use simple summary measures of the repeat information on systolic blood pressure, such as (i) baseline only; (ii) last observation carried forward; and (iii) cumulative mean, against more complex methods that model the repeat information using (iv) ordinary regression calibration; (v) risk‐set regression calibration; and (vi) joint longitudinal and survival models. In comparison with the baseline‐only model, we observed modest improvements in discrimination and calibration using the cumulative mean of systolic blood pressure, but little further improvement from any of the complex methods. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

Assessing calibration of multinomial risk prediction models

Calibration, that is, whether observed outcomes agree with predicted risks, is important when evaluating risk prediction models. For dichotomous outcomes, several tools exist to assess different aspects of model calibration, such as calibration‐in‐the‐large, logistic recalibration, and (non‐)parametric calibration plots. We aim to extend these tools to prediction models for polytomous outcomes. We focus on models developed using multinomial logistic regression (MLR): outcome Y with k categories is predicted using k ? 1 equations comparing each category i (i = 2, … ,k) with reference category 1 using a set of predictors, resulting in k ? 1 linear predictors. We propose a multinomial logistic recalibration framework that involves an MLR fit where Y is predicted using the k ? 1 linear predictors from the prediction model. A non‐parametric alternative may use vector splines for the effects of the linear predictors. The parametric and non‐parametric frameworks can be used to generate multinomial calibration plots. Further, the parametric framework can be used for the estimation and statistical testing of calibration intercepts and slopes. Two illustrative case studies are presented, one on the diagnosis of malignancy of ovarian tumors and one on residual mass diagnosis in testicular cancer patients treated with cisplatin‐based chemotherapy. The risk prediction models were developed on data from 2037 and 544 patients and externally validated on 1107 and 550 patients, respectively. We conclude that calibration tools can be extended to polytomous outcomes. The polytomous calibration plots are particularly informative through the visual summary of the calibration performance. Copyright © 2014 John Wiley & Sons, Ltd.     

Using multiple risk models with preventive interventions

An ideal preventive intervention would have negligible side effects and could be applied to the entire population, thus achieving maximal preventive impact. Unfortunately, many interventions have adverse effects and beneficial effects. For example, tamoxifen reduces the risk of breast cancer by about 50% and the risk of hip fracture by 45%, but increases the risk of stroke by about 60%; other serious adverse effects include endometrial cancer and pulmonary embolus. Hence, tamoxifen should only be given to the subset of the population with high enough risks of breast cancer and hip fracture such that the preventive benefits outweigh the risks. Recommendations for preventive use of tamoxifen have been based primarily on breast cancer risk. Age-specific and race-specific rates were considered for other health outcomes, but not risk models. In this paper, we investigate the extent to which modeling not only the risk of breast cancer, but also the risk of stroke, can improve the decision to take tamoxifen. These calculations also give insight into the relative benefits of improving the discriminatory accuracy of such risk models versus improving the preventive effectiveness or reducing the adverse risks of the intervention. Depending on the discriminatory accuracies of the risk models, there may be considerable advantage to modeling the risks of more than one health outcome. Published 2012. This article is a US Government work and is in the public domain in the USA.     

Absolute risk regression for competing risks: interpretation,link functions,and prediction

In survival analysis with competing risks, the transformation model allows different functions between the outcome and explanatory variables. However, the model's prediction accuracy and the interpretation of parameters may be sensitive to the choice of link function. We review the practical implications of different link functions for regression of the absolute risk (or cumulative incidence) of an event. Specifically, we consider models in which the regression coefficients β have the following interpretation: The probability of dying from cause D during the next t years changes with a factor exp(β) for a one unit change of the corresponding predictor variable, given fixed values for the other predictor variables. The models have a direct interpretation for the predictive ability of the risk factors. We propose some tools to justify the models in comparison with traditional approaches that combine a series of cause‐specific Cox regression models or use the Fine–Gray model. We illustrate the methods with the use of bone marrow transplant data. Copyright © 2012 John Wiley & Sons, Ltd.     

A multiple‐model generalisation of updating clinical prediction models

There is growing interest in developing clinical prediction models (CPMs) to aid local healthcare decision‐making. Frequently, these CPMs are developed in isolation across different populations, with repetitive de novo derivation a common modelling strategy. However, this fails to utilise all available information and does not respond to changes in health processes through time and space. Alternatively, model updating techniques have previously been proposed that adjust an existing CPM to suit the new population, but these techniques are restricted to a single model. Therefore, we aimed to develop a generalised method for updating and aggregating multiple CPMs. The proposed “hybrid method” re‐calibrates multiple CPMs using stacked regression while concurrently revising specific covariates using individual participant data (IPD) under a penalised likelihood. The performance of the hybrid method was compared with existing methods in a clinical example of mortality risk prediction after transcatheter aortic valve implantation, and in 2 simulation studies. The simulation studies explored the effect of sample size and between‐population‐heterogeneity on the method, with each representing a situation of having multiple distinct CPMs and 1 set of IPD. When the sample size of the IPD was small, stacked regression and the hybrid method had comparable but highest performance across modelling methods. Conversely, in large IPD samples, development of a new model and the hybrid method gave the highest performance. Hence, the proposed strategy can inform the choice between utilising existing CPMs or developing a model de novo, thereby incorporating IPD, existing research, and prior (clinical) knowledge into the modelling strategy.     

Meta‐analysis and aggregation of multiple published prediction models

Published clinical prediction models are often ignored during the development of novel prediction models despite similarities in populations and intended usage. The plethora of prediction models that arise from this practice may still perform poorly when applied in other populations. Incorporating prior evidence might improve the accuracy of prediction models and make them potentially better generalizable. Unfortunately, aggregation of prediction models is not straightforward, and methods to combine differently specified models are currently lacking. We propose two approaches for aggregating previously published prediction models when a validation dataset is available: model averaging and stacked regressions. These approaches yield user‐friendly stand‐alone models that are adjusted for the new validation data. Both approaches rely on weighting to account for model performance and between‐study heterogeneity but adopt a different rationale (averaging versus combination) to combine the models. We illustrate their implementation in a clinical example and compare them with established methods for prediction modeling in a series of simulation studies. Results from the clinical datasets and simulation studies demonstrate that aggregation yields prediction models with better discrimination and calibration in a vast majority of scenarios, and results in equivalent performance (compared to developing a novel model from scratch) when validation datasets are relatively large. In conclusion, model aggregation is a promising strategy when several prediction models are available from the literature and a validation dataset is at hand. The aggregation methods do not require existing models to have similar predictors and can be applied when relatively few data are at hand. Copyright © 2014 John Wiley & Sons, Ltd.     

Practical implementation of frailty models in Mendelian risk prediction

There are numerous statistical models used to identify individuals at high risk of cancer due to inherited mutations. Mendelian models predict future risk of cancer by using family history with estimated cancer penetrances (age- and sex-specific risk of cancer given the genotype of the mutations) and mutation prevalences. However, there is often residual risk heterogeneity across families even after accounting for the mutations in the model, due to environmental or unobserved genetic risk factors. We aim to improve Mendelian risk prediction by incorporating a frailty model that contains a family-specific frailty vector, impacting the cancer hazard function, to account for this heterogeneity. We use a discrete uniform population frailty distribution and implement a marginalized approach that averages each family's risk predictions over the family's frailty distribution. We apply the proposed approach to improve breast cancer prediction in BRCAPRO, a Mendelian model that accounts for inherited mutations in the BRCA1 and BRCA2 genes to predict breast and ovarian cancer. We evaluate the proposed model's performance in simulations and real data from the Cancer Genetics Network and show improvements in model calibration and discrimination. We also discuss alternative approaches for incorporating frailties and their strengths and limitations.     

孕产妇不良妊娠结局危险因素的病例对照研究

目的：探讨孕产妇不良妊娠结局的危险因素。方法采用以医院为基础的非匹配病例对照研究,收集产妇一般情况、既往生育史、孕期异常状况及患病等资料,使用单因素χ2检验和多因素Logistic回归进行分析。结果单因素分析显示,病例组产妇高龄、职业为农民、不良妊娠史、早孕期异常、胎位异常、妊娠合并症的比例高于对照组,差异均有统计学意义（χ2值分别为9．529、13．512、10．134、4．465、11．586、31．562,均P＜0．05）。多因素非条件Logistic回归分析表明,职业为农民（ OR＝2．298,95％CI：1．202～4．396）、不良妊娠史（OR＝1．612,95％CI：1．150～2．260）、早孕期异常（OR＝1．897,95％CI：1．142～3．152）、胎位不正（OR＝3．067,95％CI：1．536～6．126）、妊娠合并症（OR＝2．539,95％CI：1．764～3．656）为不良妊娠结局的危险因素。结论鼓励育龄期妇女优生优育,加强围孕期健康教育,孕妇应定期产检,发现异常及时就诊,以减少不良妊娠结局的发生。     

Intraclass correlation coefficients and bootstrap methods of hierarchical binary outcomes

Intraclass correlation coefficients are designed to assess consistency or conformity between two or more quantitative measurements. When multistage cluster sampling is implemented, no methods are readily available to estimate intraclass correlations of binomial-distributed outcomes within a cluster. Because statistical distribution of the intraclass correlation coefficients could be complicated or unspecified, we propose using a bootstrap method to estimate the standard error and confidence interval within the framework of a multilevel generalized linear model. We compared the results derived from a parametric bootstrap method with those from a non-parametric bootstrap method and found that the non-parametric method is more robust. For non-parametric bootstrap sampling, we showed that the effectiveness of sampling on the highest level is greater than that on lower levels; to illustrate the effectiveness, we analyse survey data in China and do simulation studies.     

Several methods to assess improvement in risk prediction models: Extension to survival analysis

Risk prediction models have been widely applied for the prediction of long‐term incidence of disease. Several parameters have been identified and estimators developed to quantify the predictive ability of models and to compare new models with traditional models. These estimators have not generally accounted for censoring in the survival data normally available for fitting the models. This paper remedies that problem. The primary parameters considered are net reclassification improvement (NRI) and integrated discrimination improvement (IDI). We have previously similarly considered a primary measure of concordance, area under the ROC curve (AUC), also called the c‐statistic. We also include here consideration of population attributable risk (PAR) and ratio of predicted risk in the top quintile of risk to that in the bottom quintile. We evaluated estimators of these various parameters both with simulation studies and also as applied to a prospective study of coronary heart disease (CHD). Our simulation studies showed that in general our estimators had little bias, and less bias and smaller variances than the traditional estimators. We have applied our methods to assessing improvement in risk prediction for each traditional CHD risk factor compared to a model without that factor. These traditional risk factors are considered valuable, yet when adding any of them to a risk prediction model that has omitted the one factor, the improvement is generally small for any of the parameters. This experience should prepare us to not expect large values of the risk prediction improvement evaluation parameters for any new risk factor to be discovered. Copyright © 2010 John Wiley & Sons, Ltd.     

Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta-analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6-month mortality based on individual patient data using meta-analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.     

A comparison of approaches for simultaneous inference of fixed effects for multiple outcomes using linear mixed models

Longitudinal studies with multiple outcomes often pose challenges for the statistical analysis. A joint model including all outcomes has the advantage of incorporating the simultaneous behavior but is often difficult to fit due to computational challenges. We consider 2 alternative approaches to quantify and assess the loss in efficiency as compared with joint modelling when evaluating fixed effects. The first approach is pairwise fitting of pseudolikelihood functions for pairs of outcomes. The second approach recovers correlations between parameter estimates across multiple marginal linear mixed models. The methods are evaluated in terms of a data example both from a study on the effects of milk protein on health in young adolescents and in an extensive simulation study. We find that the 2 alternatives give similar results in settings where an exchangeability condition is met, but otherwise, pairwise fitting shows a larger loss in efficiency than the marginal models approach. Using an alternative to the joint modelling strategy will lead to some but not necessarily a large loss of efficiency for small sample sizes.