Uncertainty method improved on best-worst case analysis in a binary meta-analysis

BACKGROUND: Most systematic reviewers aim to perform an intention-to-treat meta-analysis, including all randomized participants from each trial. This is not straightforward in practice: reviewers must decide how to handle missing outcome data in the contributing trials. OBJECTIVE: To investigate methods of allowing for uncertainty due to missing data in a meta-analysis. STUDY DESIGN AND SETTING: The Cochrane Library was surveyed to assess current use of imputation methods. We developed a methodology for incorporating uncertainty, with weights assigned to trials based on uncertainty interval widths. The uncertainty interval for a trial incorporates both sampling error and the potential impact of missing data. We evaluated the performance of this method using simulated data. RESULTS: The survey showed that complete-case analysis is commonly considered alongside best-worst case analysis. Best-worst case analysis gives an interval for the treatment effect that includes all of the uncertainty due to missing data. Unless there are few missing data, this interval is very wide. Simulations show that the uncertainty method consistently has better power and narrower interval widths than best-worst case analysis. CONCLUSION: The uncertainty method performs consistently better than best-worst case imputation and should be considered along with complete-case analysis whenever missing data are a concern.     

Advances in Statistical Methods for Substance Abuse Prevention Research

The paper describes advances in statistical methods for prevention research with a particular focus on substance abuse prevention. Standard analysis methods are extended to the typical research designs and characteristics of the data collected in prevention research. Prevention research often includes longitudinal measurement, clustering of data in units such as schools or clinics, missing data, and categorical as well as continuous outcome variables. Statistical methods to handle these features of prevention data are outlined. Developments in mediation, moderation, and implementation analysis allow for the extraction of more detailed information from a prevention study. Advancements in the interpretation of prevention research results include more widespread calculation of effect size and statistical power, the use of confidence intervals as well as hypothesis testing, detailed causal analysis of research findings, and meta-analysis. The increased availability of statistical software has contributed greatly to the use of new methods in prevention research. It is likely that the Internet will continue to stimulate the development and application of new methods.     

Missing data in a multi-item instrument were best handled by multiple imputation at the item score level

ObjectivesRegardless of the proportion of missing values, complete-case analysis is most frequently applied, although advanced techniques such as multiple imputation (MI) are available. The objective of this study was to explore the performance of simple and more advanced methods for handling missing data in cases when some, many, or all item scores are missing in a multi-item instrument.Study Design and SettingReal-life missing data situations were simulated in a multi-item variable used as a covariate in a linear regression model. Various missing data mechanisms were simulated with an increasing percentage of missing data. Subsequently, several techniques to handle missing data were applied to decide on the most optimal technique for each scenario. Fitted regression coefficients were compared using the bias and coverage as performance parameters.ResultsMean imputation caused biased estimates in every missing data scenario when data are missing for more than 10% of the subjects. Furthermore, when a large percentage of subjects had missing items (>25%), MI methods applied to the items outperformed methods applied to the total score.ConclusionWe recommend applying MI to the item scores to get the most accurate regression model estimates. Moreover, we advise not to use any form of mean imputation to handle missing data.     

Dealing with missing outcome data in randomized trials and observational studies

Although missing outcome data are an important problem in randomized trials and observational studies, methods to address this issue can be difficult to apply. Using simulated data, the authors compared 3 methods to handle missing outcome data: 1) complete case analysis; 2) single imputation; and 3) multiple imputation (all 3 with and without covariate adjustment). Simulated scenarios focused on continuous or dichotomous missing outcome data from randomized trials or observational studies. When outcomes were missing at random, single and multiple imputations yielded unbiased estimates after covariate adjustment. Estimates obtained by complete case analysis with covariate adjustment were unbiased as well, with coverage close to 95%. When outcome data were missing not at random, all methods gave biased estimates, but handling missing outcome data by means of 1 of the 3 methods reduced bias compared with a complete case analysis without covariate adjustment. Complete case analysis with covariate adjustment and multiple imputation yield similar estimates in the event of missing outcome data, as long as the same predictors of missingness are included. Hence, complete case analysis with covariate adjustment can and should be used as the analysis of choice more often. Multiple imputation, in addition, can accommodate the missing-not-at-random scenario more flexibly, making it especially suited for sensitivity analyses.     

What to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data

OBJECTIVES: To compare the performance of different meta-analysis methods for pooling odds ratios when applied to sparse event data with emphasis on the use of continuity corrections. BACKGROUND: Meta-analysis of side effects from RCTs or risk factors for rare diseases in epidemiological studies frequently requires the synthesis of data with sparse event rates. Combining such data can be problematic when zero events exist in one or both arms of a study as continuity corrections are often needed, but, these can influence results and conclusions. METHODS: A simulation study was undertaken comparing several meta-analysis methods for combining odds ratios (using various classical and Bayesian methods of estimation) on sparse event data. Where required, the routine use of a constant and two alternative continuity corrections; one based on a function of the reciprocal of the opposite group arm size; and the other an empirical estimate of the pooled effect size from the remaining studies in the meta-analysis, were also compared. A number of meta-analysis scenarios were simulated and replicated 1000 times, varying the ratio of the study arm sizes. RESULTS: Mantel-Haenszel summary estimates using the alternative continuity correction factors gave the least biased results for all group size imbalances. Logistic regression was virtually unbiased for all scenarios and gave good coverage properties. The Peto method provided unbiased results for balanced treatment groups but bias increased with the ratio of the study arm sizes. The Bayesian fixed effect model provided good coverage for all group size imbalances. The two alternative continuity corrections outperformed the constant correction factor in nearly all situations. The inverse variance method performed consistently badly, irrespective of the continuity correction used. CONCLUSIONS: Many routinely used summary methods provide widely ranging estimates when applied to sparse data with high imbalance between the size of the studies' arms. A sensitivity analysis using several methods and continuity correction factors is advocated for routine practice.     

Unpredictable bias when using the missing indicator method or complete case analysis for missing confounder values: an empirical example

ObjectiveMissing indicator method (MIM) and complete case analysis (CC) are frequently used to handle missing confounder data. Using empirical data, we demonstrated the degree and direction of bias in the effect estimate when using these methods compared with multiple imputation (MI).Study Design and SettingFrom a cohort study, we selected an exposure (marital status), outcome (depression), and confounders (age, sex, and income). Missing values in “income” were created according to different patterns of missingness: missing values were created completely at random and depending on exposure and outcome values. Percentages of missing values ranged from 2.5% to 30%.ResultsWhen missing values were completely random, MIM gave an overestimation of the odds ratio, whereas CC and MI gave unbiased results. MIM and CC gave under- or overestimations when missing values depended on observed values. Magnitude and direction of bias depended on how the missing values were related to exposure and outcome. Bias increased with increasing percentage of missing values.ConclusionMIM should not be used in handling missing confounder data because it gives unpredictable bias of the odds ratio even with small percentages of missing values. CC can be used when missing values are completely random, but it gives loss of statistical power.     

Imputation of missing values is superior to complete case analysis and the missing-indicator method in multivariable diagnostic research: a clinical example

BACKGROUND AND OBJECTIVES: To illustrate the effects of different methods for handling missing data--complete case analysis, missing-indicator method, single imputation of unconditional and conditional mean, and multiple imputation (MI)--in the context of multivariable diagnostic research aiming to identify potential predictors (test results) that independently contribute to the prediction of disease presence or absence. METHODS: We used data from 398 subjects from a prospective study on the diagnosis of pulmonary embolism. Various diagnostic predictors or tests had (varying percentages of) missing values. Per method of handling these missing values, we fitted a diagnostic prediction model using multivariable logistic regression analysis. RESULTS: The receiver operating characteristic curve area for all diagnostic models was above 0.75. The predictors in the final models based on the complete case analysis, and after using the missing-indicator method, were very different compared to the other models. The models based on MI did not differ much from the models derived after using single conditional and unconditional mean imputation. CONCLUSION: In multivariable diagnostic research complete case analysis and the use of the missing-indicator method should be avoided, even when data are missing completely at random. MI methods are known to be superior to single imputation methods. For our example study, the single imputation methods performed equally well, but this was most likely because of the low overall number of missing values.     

Limitations in Using Multiple Imputation to Harmonize Individual Participant Data for Meta-Analysis

Individual participant data (IPD) meta-analysis is a meta-analysis in which the individual-level data for each study are obtained and used for synthesis. A common challenge in IPD meta-analysis is when variables of interest are measured differently in different studies. The term harmonization has been coined to describe the procedure of placing variables on the same scale in order to permit pooling of data from a large number of studies. Using data from an IPD meta-analysis of 19 adolescent depression trials, we describe a multiple imputation approach for harmonizing 10 depression measures across the 19 trials by treating those depression measures that were not used in a study as missing data. We then apply diagnostics to address the fit of our imputation model. Even after reducing the scale of our application, we were still unable to produce accurate imputations of the missing values. We describe those features of the data that made it difficult to harmonize the depression measures and provide some guidelines for using multiple imputation for harmonization in IPD meta-analysis.     

Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study

Standard network meta-analysis and indirect comparisons combine aggregate data from multiple studies on treatments of interest, assuming that any factors that interact with treatment effects (effect modifiers) are balanced across populations. Population adjustment methods such as multilevel network meta-regression (ML-NMR), matching-adjusted indirect comparison (MAIC), and simulated treatment comparison (STC) relax this assumption using individual patient data from one or more studies, and are becoming increasingly prevalent in health technology appraisals and the applied literature. Motivated by an applied example and two recent reviews of applications, we undertook an extensive simulation study to assess the performance of these methods in a range of scenarios under various failures of assumptions. We investigated the impact of varying sample size, missing effect modifiers, strength of effect modification and validity of the shared effect modifier assumption, validity of extrapolation and varying between-study overlap, and different covariate distributions and correlations. ML-NMR and STC performed similarly, eliminating bias when the requisite assumptions were met. Serious concerns are raised for MAIC, which performed poorly in nearly all simulation scenarios and may even increase bias compared with standard indirect comparisons. All methods incur bias when an effect modifier is missing, highlighting the necessity of careful selection of potential effect modifiers prior to analysis. When all effect modifiers are included, ML-NMR and STC are robust techniques for population adjustment. ML-NMR offers additional advantages over MAIC and STC, including extending to larger treatment networks and producing estimates in any target population, making this an attractive choice in a variety of scenarios.     

Limitations in Using Multiple Imputation to Harmonize Individual Participant Data for Meta-Analysis

Individual participant data (IPD) meta-analysis is a meta-analysis in which the individual-level data for each study are obtained and used for synthesis. A common challenge in IPD meta-analysis is when variables of interest are measured differently in different studies. The term harmonization has been coined to describe the procedure of placing variables on the same scale in order to permit pooling of data from a large number of studies. Using data from an IPD meta-analysis of 19 adolescent depression trials, we describe a multiple imputation approach for harmonizing 10 depression measures across the 19 trials by treating those depression measures that were not used in a study as missing data. We then apply diagnostics to address the fit of our imputation model. Even after reducing the scale of our application, we were still unable to produce accurate imputations of the missing values. We describe those features of the data that made it difficult to harmonize the depression measures and provide some guidelines for using multiple imputation for harmonization in IPD meta-analysis.

     

Allowing for uncertainty due to missing data in meta-analysis--part 1: two-stage methods

Analysis of a randomized trial with missing outcome data involves untestable assumptions, such as the missing at random (MAR) assumption. Estimated treatment effects are potentially biased if these assumptions are wrong. We quantify the degree of departure from the MAR assumption by the informative missingness odds ratio (IMOR). We incorporate prior beliefs about the IMOR in a Bayesian pattern-mixture model and derive a point estimate and standard error that take account of the uncertainty about the IMOR. In meta-analysis, this model should be used for four separate sensitivity analyses which explore the impact of IMORs that either agree or contrast across trial arms on pooled results via their effects on point estimates or on standard errors. We also propose a variance inflation factor that can be used to assess the influence of trials with many missing outcomes on the meta-analysis. We illustrate the methods using a meta-analysis on psychiatric interventions in deliberate self-harm.     

Imputing variance estimates do not alter the conclusions of a meta-analysis with continuous outcomes: a case study of changes in renal function after living kidney donation

OBJECTIVE: To assess how different imputation methods used to account for missing variance data in primary studies influence tests of heterogeneity and pooled results from a meta-analysis with continuous outcomes. STUDY DESIGN AND SETTING: Point and variance estimates for changes in serum creatinine, glomerular filtration rate, systolic blood pressure, and diastolic blood pressure were variably reported among 48 primary longitudinal studies of living kidney donors (71%-78% of point estimates were reported, 8%-13% of variance data were reported). We compared the results of meta-analysis, which either were restricted to available data or used four methods to impute missing variance data. These methods used reported P-values, reported nonparametric summaries, results from other similar studies using multiple imputation, or results from estimated correlation coefficients. RESULTS: Significant heterogeneity was present in all four outcomes regardless of the imputation methods applied. The random effects point estimates and 95% confidence intervals varied little across imputation methods, and the differences were not clinically significant. CONCLUSIONS: Different methods to impute the variance data in the primary studies did not alter the conclusions from this meta-analysis of continuous outcomes. Such reproducibility increases confidence in the results. However, as with most meta-analyses, there was no gold standard of truth, and results must be interpreted judiciously. The generalization of these findings to other meta-analyses, which differ in outcomes, missing data, or between-study heterogeneity, requires further consideration.     

Identification of significant host factors for HIV dynamics modelled by non-linear mixed-effects models

Non-linear mixed-effects models are powerful tools for modelling HIV viral dynamics. In AIDS clinical trials, the viral load measurements for each subject are often sparse. In such cases, linearization procedures are usually used for inferences. Under such linearization procedures, however, standard covariate selection methods based on the approximate likelihood, such as the likelihood ratio test, may not be reliable. In order to identify significant host factors for HIV dynamics, in this paper we consider two alternative approaches for covariate selection: one is based on individual non-linear least square estimates and the other is based on individual empirical Bayes estimates. Our simulation study shows that, if the within-individual data are sparse and the between-individual variation is large, the two alternative covariate selection methods are more reliable than the likelihood ratio test, and the more powerful method based on individual empirical Bayes estimates is especially preferable. We also consider the missing data in covariates. The commonly used missing data methods may lead to misleading results. We recommend a multiple imputation method to handle missing covariates. A real data set from an AIDS clinical trial is analysed based on various covariate selection methods and missing data methods.     

Haplotype analysis in the presence of informatively missing genotype data

It is common to have missing genotypes in practical genetic studies, but the exact underlying missing data mechanism is generally unknown to the investigators. Although some statistical methods can handle missing data, they usually assume that genotypes are missing at random, that is, at a given marker, different genotypes and different alleles are missing with the same probability. These include those methods on haplotype frequency estimation and haplotype association analysis. However, it is likely that this simple assumption does not hold in practice, yet few studies to date have examined the magnitude of the effects when this simplifying assumption is violated. In this study, we demonstrate that the violation of this assumption may lead to serious bias in haplotype frequency estimates, and haplotype association analysis based on this assumption can induce both false-positive and false-negative evidence of association. To address this limitation in the current methods, we propose a general missing data model to characterize missing data patterns across a set of two or more markers simultaneously. We prove that haplotype frequencies and missing data probabilities are identifiable if and only if there is linkage disequilibrium between these markers under our general missing data model. Simulation studies on the analysis of haplotypes consisting of two single nucleotide polymorphisms illustrate that our proposed model can reduce the bias both for haplotype frequency estimates and association analysis due to incorrect assumption on the missing data mechanism. Finally, we illustrate the utilities of our method through its application to a real data set.     

Sequential analysis of latent variables using mixed-effect latent variable models: Impact of non-informative and informative missing data

Sequential methods allowing for early stopping of clinical trials are widely used in various therapeutic areas. These methods allow for the analysis of different types of endpoints (quantitative, qualitative, time to event) and often provide, in average, substantial reductions in sample size as compared with single-stage designs while maintaining pre-specified type I and II errors. Sequential methods are also used when analysing particular endpoints that cannot be directly measured, such as depression, quality of life, or cognitive functioning, which are often measured through questionnaires. These types of endpoints are usually referred to as latent variables and should be analysed with latent variable models. In addition, in most clinical trials studying such latent variables, incomplete data are not uncommon and the missing data process might also be non-ignorable. We investigated the impact of informative or non-informative missing data on the statistical properties of the double triangular test (DTT), combined with the mixed-effects Rasch model (MRM) for dichotomous responses or the traditional method based on observed patient's scores (S) to the questionnaire. The achieved type I errors for the DTT were usually close to the target value of 0.05 for both methods, but increased slightly for the MRM when informative missing data were present. The DTT was very close to the nominal power of 0.95 when the MRM was used, but substantially underpowered with the S method (reduction of about 23 per cent), irrespective of whether informative missing data were present or not. Moreover, the DTT using the MRM allowed for reaching a conclusion (under H(0) or H(1)) with fewer patients than the S method, the average sample number for the latter increasing importantly when the proportion of missing data increased. Incorporating MRM in sequential analysis of latent variables might provide a more powerful method than the traditional S method, even in the presence of non-informative or informative missing data.     

EMLasso: logistic lasso with missing data

In clinical settings, missing data in the covariates occur frequently. For example, some markers are expensive or hard to measure. When this sort of data is used for model selection, the missingness is often resolved through a complete case analysis or a form of single imputation. An alternative sometimes comes in the form of leaving the most damaged covariates out. All these strategies jeopardise the goal of model selection. In earlier work, we have applied the logistic Lasso in combination with multiple imputation to obtain results in such settings, but we only provided heuristic arguments to advocate the method. In this paper, we propose an improved method that builds on firm statistical arguments and that is developed along the lines of the stochastic expectation–maximisation algorithm. We show that our method can be used to handle missing data in both categorical and continuous predictors, as well as in a nonpenalised regression. We demonstrate the method by applying it to data of 273 lung cancer patients. The objective is to select a model for the prediction of acute dysphagia, starting from a large set of potential predictors, including clinical and treatment covariates as well as a set of single‐nucleotide polymorphisms. Copyright © 2013 John Wiley & Sons, Ltd.     

Joint mixed‐effects models for causal inference with longitudinal data

Causal inference with observational longitudinal data and time‐varying exposures is complicated due to the potential for time‐dependent confounding and unmeasured confounding. Most causal inference methods that handle time‐dependent confounding rely on either the assumption of no unmeasured confounders or the availability of an unconfounded variable that is associated with the exposure (eg, an instrumental variable). Furthermore, when data are incomplete, validity of many methods often depends on the assumption of missing at random. We propose an approach that combines a parametric joint mixed‐effects model for the study outcome and the exposure with g‐computation to identify and estimate causal effects in the presence of time‐dependent confounding and unmeasured confounding. G‐computation can estimate participant‐specific or population‐average causal effects using parameters of the joint model. The joint model is a type of shared parameter model where the outcome and exposure‐selection models share common random effect(s). We also extend the joint model to handle missing data and truncation by death when missingness is possibly not at random. We evaluate the performance of the proposed method using simulation studies and compare the method to both linear mixed‐ and fixed‐effects models combined with g‐computation as well as to targeted maximum likelihood estimation. We apply the method to an epidemiologic study of vitamin D and depressive symptoms in older adults and include code using SAS PROC NLMIXED software to enhance the accessibility of the method to applied researchers.     

Development and validation of a prediction model with missing predictor data: a practical approach

ObjectiveTo illustrate the sequence of steps needed to develop and validate a clinical prediction model, when missing predictor values have been multiply imputed.Study Design and SettingWe used data from consecutive primary care patients suspected of deep venous thrombosis (DVT) to develop and validate a diagnostic model for the presence of DVT. Missing values were imputed 10 times with the MICE conditional imputation method. After the selection of predictors and transformations for continuous predictors according to three different methods, we estimated regression coefficients and performance measures.ResultsThe three methods to select predictors and transformations of continuous predictors showed similar results. Rubin's rules could easily be applied to estimate regression coefficients and performance measures, once predictors and transformations were selected.ConclusionWe provide a practical approach for model development and validation with multiply imputed data.     

Comparison of imputation and modelling methods in the analysis of a physical activity trial with missing outcomes

BACKGROUND: Longitudinal studies almost always have some individuals with missing outcomes. Inappropriate handling of the missing data in the analysis can result in misleading conclusions. Here we review a wide range of methods to handle missing outcomes in single and repeated measures data and discuss which methods are most appropriate. METHODS: Using data from a randomized controlled trial to compare two interventions for increasing physical activity, we compare complete-case analysis; ad hoc imputation techniques such as last observation carried forward and worst-case; model-based imputation; longitudinal models with random effects; and recently proposed joint models for repeated measures data and non-ignorable dropout. RESULTS: Estimated intervention effects from ad hoc imputation methods vary widely. Standard multiple imputation and longitudinal modelling agree closely, as they should. Modifying the modelling method to allow for non-ignorable dropout had little effect on estimated intervention effects, but imputing using a common imputation model in both groups gave more conservative results. CONCLUSIONS: Results from ad hoc imputation methods should be avoided in favour of methods with more plausible assumptions although they may be computationally more complex. Although standard multiple imputation methods and longitudinal modelling methods are equivalent for estimating the treatment effect, the two approaches suggest different ways of relaxing the assumptions, and the choice between them depends on contextual knowledge.     

Analyzing weight loss intervention studies with missing data: Which methods should be used?

ObjectiveMissing data due to study dropout is common in weight loss trials and several statistical methods exist to account for it. The aim of this study was to identify methods in the literature and to compare the effects of methods of analysis using simulated data sets.MethodsLiterature was obtained for a 1-y period to identify analytical methods used in reporting weight loss trials. A comparison of methods with large or small between-group weight loss, and missing data that was, or was not, missing randomly was conducted in simulated data sets based on previous research.ResultsTwenty-seven studies, some with multiple analyses, were retrieved. Complete case analysis (n = 17), last observation carried forward (n = 6), baseline carried forward (n = 4), maximum likelihood (n = 6), and multiple imputation (n = 2) were the common methods of accounting for missing data. When comparing methods on simulated data, all demonstrated a significant effect when the between-group weight loss was large (P < 0.001, interaction term) regardless of whether the data was missing completely at random. When the weight loss interaction was small, the method used for analysis gave considerably different results with mixed models (P = 0.180) and multiple imputations (P = 0.125) closest to the full data model (P = 0.033).ConclusionThe simulation analysis showed that when data were not missing at random, treatment effects were small, and the amount of missing data was substantial, the analysis method had an effect on the significance of the outcome. Careful attention must be paid when analyzing or appraising studies with missing data and small effects to ensure appropriate conclusions are drawn.