Aggregating published prediction models with individual participant data: a comparison of different approaches

During the recent decades, interest in prediction models has substantially increased, but approaches to synthesize evidence from previously developed models have failed to keep pace. This causes researchers to ignore potentially useful past evidence when developing a novel prediction model with individual participant data (IPD) from their population of interest. We aimed to evaluate approaches to aggregate previously published prediction models with new data. We consider the situation that models are reported in the literature with predictors similar to those available in an IPD dataset. We adopt a two‐stage method and explore three approaches to calculate a synthesis model, hereby relying on the principles of multivariate meta‐analysis. The former approach employs a naive pooling strategy, whereas the latter accounts for within‐study and between‐study covariance. These approaches are applied to a collection of 15 datasets of patients with traumatic brain injury, and to five previously published models for predicting deep venous thrombosis. Here, we illustrated how the generally unrealistic assumption of consistency in the availability of evidence across included studies can be relaxed. Results from the case studies demonstrate that aggregation yields prediction models with an improved discrimination and calibration in a vast majority of scenarios, and result in equivalent performance (compared with the standard approach) in a small minority of situations. The proposed aggregation approaches are particularly useful when few participant data are at hand. Assessing the degree of heterogeneity between IPD and literature findings remains crucial to determine the optimal approach in aggregating previous evidence into new prediction models. Copyright © 2012 John Wiley & Sons, Ltd.     

Combining multiple imputation and meta‐analysis with individual participant data

Multiple imputation is a strategy for the analysis of incomplete data such that the impact of the missingness on the power and bias of estimates is mitigated. When data from multiple studies are collated, we can propose both within‐study and multilevel imputation models to impute missing data on covariates. It is not clear how to choose between imputation models or how to combine imputation and inverse‐variance weighted meta‐analysis methods. This is especially important as often different studies measure data on different variables, meaning that we may need to impute data on a variable which is systematically missing in a particular study. In this paper, we consider a simulation analysis of sporadically missing data in a single covariate with a linear analysis model and discuss how the results would be applicable to the case of systematically missing data. We find in this context that ensuring the congeniality of the imputation and analysis models is important to give correct standard errors and confidence intervals. For example, if the analysis model allows between‐study heterogeneity of a parameter, then we should incorporate this heterogeneity into the imputation model to maintain the congeniality of the two models. In an inverse‐variance weighted meta‐analysis, we should impute missing data and apply Rubin's rules at the study level prior to meta‐analysis, rather than meta‐analyzing each of the multiple imputations and then combining the meta‐analysis estimates using Rubin's rules. We illustrate the results using data from the Emerging Risk Factors Collaboration. © 2013 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.     

Testing moderation in network meta‐analysis with individual participant data

Meta‐analytic methods for combining data from multiple intervention trials are commonly used to estimate the effectiveness of an intervention. They can also be extended to study comparative effectiveness, testing which of several alternative interventions is expected to have the strongest effect. This often requires network meta‐analysis (NMA), which combines trials involving direct comparison of two interventions within the same trial and indirect comparisons across trials. In this paper, we extend existing network methods for main effects to examining moderator effects, allowing for tests of whether intervention effects vary for different populations or when employed in different contexts. In addition, we study how the use of individual participant data may increase the sensitivity of NMA for detecting moderator effects, as compared with aggregate data NMA that employs study‐level effect sizes in a meta‐regression framework. A new NMA diagram is proposed. We also develop a generalized multilevel model for NMA that takes into account within‐trial and between‐trial heterogeneity and can include participant‐level covariates. Within this framework, we present definitions of homogeneity and consistency across trials. A simulation study based on this model is used to assess effects on power to detect both main and moderator effects. Results show that power to detect moderation is substantially greater when applied to individual participant data as compared with study‐level effects. We illustrate the use of this method by applying it to data from a classroom‐based randomized study that involved two sub‐trials, each comparing interventions that were contrasted with separate control groups. Copyright © 2016 John Wiley & Sons, Ltd.     

Multiple imputation for handling systematically missing confounders in meta‐analysis of individual participant data

A variable is ‘systematically missing’ if it is missing for all individuals within particular studies in an individual participant data meta‐analysis. When a systematically missing variable is a potential confounder in observational epidemiology, standard methods either fail to adjust the exposure–disease association for the potential confounder or exclude studies where it is missing. We propose a new approach to adjust for systematically missing confounders based on multiple imputation by chained equations. Systematically missing data are imputed via multilevel regression models that allow for heterogeneity between studies. A simulation study compares various choices of imputation model. An illustration is given using data from eight studies estimating the association between carotid intima media thickness and subsequent risk of cardiovascular events. Results are compared with standard methods and also with an extension of a published method that exploits the relationship between fully adjusted and partially adjusted estimated effects through a multivariate random effects meta‐analysis model. We conclude that multiple imputation provides a practicable approach that can handle arbitrary patterns of systematic missingness. Bias is reduced by including sufficient between‐study random effects in the imputation model. Copyright © 2013 John Wiley & Sons, Ltd.     

A comparison of existing methods for multiple imputation in individual participant data meta‐analysis

Multiple imputation is a popular method for addressing missing data, but its implementation is difficult when data have a multilevel structure and one or more variables are systematically missing. This systematic missing data pattern may commonly occur in meta‐analysis of individual participant data, where some variables are never observed in some studies, but are present in other hierarchical data settings. In these cases, valid imputation must account for both relationships between variables and correlation within studies. Proposed methods for multilevel imputation include specifying a full joint model and multiple imputation with chained equations (MICE). While MICE is attractive for its ease of implementation, there is little existing work describing conditions under which this is a valid alternative to specifying the full joint model. We present results showing that for multilevel normal models, MICE is rarely exactly equivalent to joint model imputation. Through a simulation study and an example using data from a traumatic brain injury study, we found that in spite of theoretical differences, MICE imputations often produce results similar to those obtained using the joint model. We also assess the influence of prior distributions in MICE imputation methods and find that when missingness is high, prior choices in MICE models tend to affect estimation of across‐study variability more than compatibility of conditional likelihoods. Copyright © 2017 John Wiley & Sons, Ltd.     

Systematically missing confounders in individual participant data meta‐analysis of observational cohort studies

One difficulty in performing meta‐analyses of observational cohort studies is that the availability of confounders may vary between cohorts, so that some cohorts provide fully adjusted analyses while others only provide partially adjusted analyses. Commonly, analyses of the association between an exposure and disease either are restricted to cohorts with full confounder information, or use all cohorts but do not fully adjust for confounding. We propose using a bivariate random‐effects meta‐analysis model to use information from all available cohorts while still adjusting for all the potential confounders. Our method uses both the fully adjusted and the partially adjusted estimated effects in the cohorts with full confounder information, together with an estimate of their within‐cohort correlation. The method is applied to estimate the association between fibrinogen level and coronary heart disease incidence using data from 154 012 participants in 31 cohorts.^? One hundred and ninety‐nine participants from the original 154 211 withdrew their consent and have been removed from this analysis. Copyright © 2009 John Wiley & Sons, Ltd.     

Multiple imputation for harmonizing longitudinal non‐commensurate measures in individual participant data meta‐analysis

There are many advantages to individual participant data meta‐analysis for combining data from multiple studies. These advantages include greater power to detect effects, increased sample heterogeneity, and the ability to perform more sophisticated analyses than meta‐analyses that rely on published results. However, a fundamental challenge is that it is unlikely that variables of interest are measured the same way in all of the studies to be combined. We propose that this situation can be viewed as a missing data problem in which some outcomes are entirely missing within some trials and use multiple imputation to fill in missing measurements. We apply our method to five longitudinal adolescent depression trials where four studies used one depression measure and the fifth study used a different depression measure. None of the five studies contained both depression measures. We describe a multiple imputation approach for filling in missing depression measures that makes use of external calibration studies in which both depression measures were used. We discuss some practical issues in developing the imputation model including taking into account treatment group and study. We present diagnostics for checking the fit of the imputation model and investigate whether external information is appropriately incorporated into the imputed values. Copyright © 2015 John Wiley & Sons, Ltd.     

Handling incomplete correlated continuous and binary outcomes in meta‐analysis of individual participant data

Meta‐analysis of individual participant data (IPD) is increasingly utilised to improve the estimation of treatment effects, particularly among different participant subgroups. An important concern in IPD meta‐analysis relates to partially or completely missing outcomes for some studies, a problem exacerbated when interest is on multiple discrete and continuous outcomes. When leveraging information from incomplete correlated outcomes across studies, the fully observed outcomes may provide important information about the incompleteness of the other outcomes. In this paper, we compare two models for handling incomplete continuous and binary outcomes in IPD meta‐analysis: a joint hierarchical model and a sequence of full conditional mixed models. We illustrate how these approaches incorporate the correlation across the multiple outcomes and the between‐study heterogeneity when addressing the missing data. Simulations characterise the performance of the methods across a range of scenarios which differ according to the proportion and type of missingness, strength of correlation between outcomes and the number of studies. The joint model provided confidence interval coverage consistently closer to nominal levels and lower mean squared error compared with the fully conditional approach across the scenarios considered. Methods are illustrated in a meta‐analysis of randomised controlled trials comparing the effectiveness of implantable cardioverter‐defibrillator devices alone to implantable cardioverter‐defibrillator combined with cardiac resynchronisation therapy for treating patients with chronic heart failure. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

Antibiotic treatment of bacterial vaginosis to prevent preterm delivery: Systematic review and individual participant data meta-analysis

Background

Bacterial vaginosis (BV) increases preterm delivery (PTD) risk, but treatment trials showed mixed results in preventing PTD.

Objectives

Determine, using individual participant data (IPD), whether BV treatment during pregnancy reduced PTD or prolonged time-to-delivery.

Data Sources

Cochrane Systematic Review (2013), MEDLINE, EMBASE, journal searches, and searches (January 2013–September 2022) (“bacterial vaginosis AND pregnancy”) of (i) clinicaltrials.gov ; (ii) Cochrane Central Register of Controlled Trials; (iii) World Health Organization International Clinical Trials Registry Platform Portal; and (iv) Web of Science (“bacterial vaginosis”).

Study Selection and Data Extraction

Studies randomising asymptomatic pregnant individuals with BV to antibiotics or control, measuring delivery gestation. Extraction was from original data files. Bias risk was assessed using the Cochrane tool. Analysis used “one-step” logistic and Cox random effect models, adjusting gestation at randomisation and PTD history; heterogeneity by I². Subgroup analysis tested interactions with treatment. In sensitivity analyses, studies not providing IPD were incorporated by “multiple random-donor hot-deck” imputation, using IPD studies as donors.

Results

There were 121 references (96 studies) with 23 eligible trials (11,979 participants); 13 studies (6915 participants) provided IPD; 12 (6115) were incorporated. Results from 9 (4887 participants) not providing IPD were imputed. Odds ratios for PTD for metronidazole and clindamycin versus placebo were 1.00 (95% CI 0.84, 1.17), I² = 62%, and 0.59 (95% CI 0.42, 0.82), I² = 0 before; and 0.95 (95% CI 0.81, 1.11), I² = 59%, and 0.90 (95% CI: 0.72, 1.12), I² = 0, after imputation. Time-to-delivery did not differ from null with either treatment. Including imputed IPD, there was no evidence that either drug was more effective when administered earlier, or among those with a PTD history.

Conclusions

Clindamycin, but not metronidazole, was beneficial in studies providing IPD, but after imputing data from missing IPD studies, treatment of BV during pregnancy did not reduce PTD, nor prolong pregnancy, in any subgroup or when started earlier in gestation.     

Evaluation of biomarkers for treatment selection using individual participant data from multiple clinical trials

Biomarkers that predict treatment effects may be used to guide treatment decisions, thus improving patient outcomes. A meta‐analysis of individual participant data (IPD) is potentially more powerful than a single‐study data analysis in evaluating markers for treatment selection. Our study was motivated by the IPD that were collected from 2 randomized controlled trials of hypertension and preeclampsia among pregnant women to evaluate the effect of labor induction over expectant management of the pregnancy in preventing progression to severe maternal disease. The existing literature on statistical methods for biomarker evaluation in IPD meta‐analysis have evaluated a marker's performance in terms of its ability to predict risk of disease outcome, which do not directly apply to the treatment selection problem. In this study, we propose a statistical framework for evaluating a marker for treatment selection given IPD from a small number of individual clinical trials. We derive marker‐based treatment rules by minimizing the average expected outcome across studies. The application of the proposed methods to the IPD from 2 studies in women with hypertension in pregnancy is presented.     

Sex-dependent immune responses to infant vaccination: an individual participant data meta-analysis of antibody and memory B cells

BackgroundBiological sex can be an important source of variation in infection and immunity and sex-dependent differences in immune response to vaccination have been reported in some studies.MethodsWe conducted an individual participant data meta-analysis of vaccine trials from one research centre, in which vaccines were administered to children under three years of age and immunological parameters measured. Log-transformed antigen-specific antibody and memory B cell results were meta-analysed and differences between girls and boys reported as geometric mean ratios.ResultsAntibody and memory B cell data were available from nine trials and 2378 children. Statistically significant differences between girls and boys were observed for diphtheria toxoid, capsular group A, W, and Y meningococcal, and pneumococcal vaccines. No sex-differences were observed for responses to Haemophilus influenzae type b, capsular group C meningococcal or tetanus toxoid vaccines.ConclusionsIn young children, immune responses to vaccines were consistently higher or equivalent in girls compared with boys. In no instance were responses in boys significantly higher than girls. While these data do not indicate differences in protection conferred by immunisation in boys and girls, they do support further consideration of biological sex in planning of clinical trials of vaccines.     

Imputation of systematically missing predictors in an individual participant data meta‐analysis: a generalized approach using MICE

Individual participant data meta‐analyses (IPD‐MA) are increasingly used for developing and validating multivariable (diagnostic or prognostic) risk prediction models. Unfortunately, some predictors or even outcomes may not have been measured in each study and are thus systematically missing in some individual studies of the IPD‐MA. As a consequence, it is no longer possible to evaluate between‐study heterogeneity and to estimate study‐specific predictor effects, or to include all individual studies, which severely hampers the development and validation of prediction models. Here, we describe a novel approach for imputing systematically missing data and adopt a generalized linear mixed model to allow for between‐study heterogeneity. This approach can be viewed as an extension of Resche‐Rigon's method (Stat Med 2013), relaxing their assumptions regarding variance components and allowing imputation of linear and nonlinear predictors. We illustrate our approach using a case study with IPD‐MA of 13 studies to develop and validate a diagnostic prediction model for the presence of deep venous thrombosis. We compare the results after applying four methods for dealing with systematically missing predictors in one or more individual studies: complete case analysis where studies with systematically missing predictors are removed, traditional multiple imputation ignoring heterogeneity across studies, stratified multiple imputation accounting for heterogeneity in predictor prevalence, and multilevel multiple imputation (MLMI) fully accounting for between‐study heterogeneity. We conclude that MLMI may substantially improve the estimation of between‐study heterogeneity parameters and allow for imputation of systematically missing predictors in IPD‐MA aimed at the development and validation of prediction models. Copyright © 2015 John Wiley & Sons, Ltd.     

Meta‐analysis of genome‐wide association studies: no efficiency gain in using individual participant data

To identify genetic variants with modest effects on complex human diseases, a growing number of networks or consortia are created for sharing data from multiple genome‐wide association studies on the same disease or related disorders. A central question in this enterprise is whether to obtain summary results or individual participant data from relevant studies. We show theoretically and numerically that meta‐analysis of summary results is statistically as efficient as joint analysis of individual participant data (provided that both analyses are performed properly under the same modeling assumptions). We illustrate this equivalence with case‐control data from the Finland‐United States Investigation of NIDDM Genetics (FUSION) study. Collating only summary results will increase the number and representativeness of available studies, simplify data collection and analysis, reduce resource utilization, and accelerate discovery. Genet. Epidemiol. 34:60–66, 2010. © 2009 Wiley‐Liss, Inc.     

Meta-analysis of non-linear exposure-outcome relationships using individual participant data: A comparison of two methods

Non-linear exposure-outcome relationships such as between body mass index (BMI) and mortality are common. They are best explored as continuous functions using individual participant data from multiple studies. We explore two two-stage methods for meta-analysis of such relationships, where the confounder-adjusted relationship is first estimated in a non-linear regression model in each study, then combined across studies. The “metacurve” approach combines the estimated curves using multiple meta-analyses of the relative effect between a given exposure level and a reference level. The “mvmeta” approach combines the estimated model parameters in a single multivariate meta-analysis. Both methods allow the exposure-outcome relationship to differ across studies. Using theoretical arguments, we show that the methods differ most when covariate distributions differ across studies; using simulated data, we show that mvmeta gains precision but metacurve is more robust to model mis-specification. We then compare the two methods using data from the Emerging Risk Factors Collaboration on BMI, coronary heart disease events, and all-cause mortality (>80 cohorts, >18 000 events). For each outcome, we model BMI using fractional polynomials of degree 2 in each study, with adjustment for confounders. For metacurve, the powers defining the fractional polynomials may be study-specific or common across studies. For coronary heart disease, metacurve with common powers and mvmeta correctly identify a small increase in risk in the lowest levels of BMI, but metacurve with study-specific powers does not. For all-cause mortality, all methods identify a steep U-shape. The metacurve and mvmeta methods perform well in combining complex exposure-disease relationships across studies.     

Meta‐analysis of randomised trials with a continuous outcome according to baseline imbalance and availability of individual participant data

We describe methods for meta‐analysis of randomised trials where a continuous outcome is of interest, such as blood pressure, recorded at both baseline (pre treatment) and follow‐up (post treatment). We used four examples for illustration, covering situations with and without individual participant data (IPD) and with and without baseline imbalance between treatment groups in each trial. Given IPD, meta‐analysts can choose to synthesise treatment effect estimates derived using analysis of covariance (ANCOVA), a regression of just final scores, or a regression of the change scores. When there is baseline balance in each trial, treatment effect estimates derived using ANCOVA are more precise and thus preferred. However, we show that meta‐analysis results for the summary treatment effect are similar regardless of the approach taken. Thus, without IPD, if trials are balanced, reviewers can happily utilise treatment effect estimates derived from any of the approaches. However, when some trials have baseline imbalance, meta‐analysts should use treatment effect estimates derived from ANCOVA, as this adjusts for imbalance and accounts for the correlation between baseline and follow‐up; we show that the other approaches can give substantially different meta‐analysis results. Without IPD and with unavailable ANCOVA estimates, reviewers should limit meta‐analyses to those trials with baseline balance. Trowman's method to adjust for baseline imbalance without IPD performs poorly in our examples and so is not recommended. Finally, we extend the ANCOVA model to estimate the interaction between treatment effect and baseline values and compare options for estimating this interaction given only aggregate data. Copyright © 2013 John Wiley & Sons, Ltd.     

A framework for developing,implementing, and evaluating clinical prediction models in an individual participant data meta‐analysis

The use of individual participant data (IPD) from multiple studies is an increasingly popular approach when developing a multivariable risk prediction model. Corresponding datasets, however, typically differ in important aspects, such as baseline risk. This has driven the adoption of meta‐analytical approaches for appropriately dealing with heterogeneity between study populations. Although these approaches provide an averaged prediction model across all studies, little guidance exists about how to apply or validate this model to new individuals or study populations outside the derivation data. We consider several approaches to develop a multivariable logistic regression model from an IPD meta‐analysis (IPD‐MA) with potential between‐study heterogeneity. We also propose strategies for choosing a valid model intercept for when the model is to be validated or applied to new individuals or study populations. These strategies can be implemented by the IPD‐MA developers or future model validators. Finally, we show how model generalizability can be evaluated when external validation data are lacking using internal–external cross‐validation and extend our framework to count and time‐to‐event data. In an empirical evaluation, our results show how stratified estimation allows study‐specific model intercepts, which can then inform the intercept to be used when applying the model in practice, even to a population not represented by included studies. In summary, our framework allows the development (through stratified estimation), implementation in new individuals (through focused intercept choice), and evaluation (through internal–external validation) of a single, integrated prediction model from an IPD‐MA in order to achieve improved model performance and generalizability. Copyright © 2013 John Wiley & Sons, Ltd.     

Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta-analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6-month mortality based on individual patient data using meta-analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.