Robust extraction of covariate information to improve estimation efficiency in randomized trials

In randomized trials, investigators typically rely upon an unadjusted estimate of the mean outcome within each treatment arm to draw causal inferences. Statisticians have underscored the gain in efficiency that can be achieved from covariate adjustment in randomized trials with a focus on problems involving linear models. Despite recent theoretical advances, there has been a reluctance to adjust for covariates based on two primary reasons: (i) covariate-adjusted estimates based on conditional logistic regression models have been shown to be less precise and (ii) concern over the opportunity to manipulate the model selection process for covariate adjustments to obtain favorable results. In this paper, we address these two issues and summarize recent theoretical results on which is based a proposed general methodology for covariate adjustment under the framework of targeted maximum likelihood estimation in trials with two arms where the probability of treatment is 50%. The proposed methodology provides an estimate of the true causal parameter of interest representing the population-level treatment effect. It is compared with the estimates based on conditional logistic modeling, which only provide estimates of subgroup-level treatment effects rather than marginal (unconditional) treatment effects. We provide a clear criterion for determining whether a gain in efficiency can be achieved with covariate adjustment over the unadjusted method. We illustrate our strategy using a resampled clinical trial dataset from a placebo controlled phase 4 study. Results demonstrate that gains in efficiency can be achieved even with binary outcomes through covariate adjustment leading to increased statistical power.     

Covariate adjustment in clinical trials with non-ignorable missing data and non-compliance

Estimating causal effects in psychiatric clinical trials is often complicated by treatment non-compliance and missing outcomes. While new estimators have recently been proposed to address these problems, they do not allow for inclusion of continuous covariates. We propose estimators that adjust for continuous covariates in addition to non-compliance and missing data. Using simulations, we compare mean squared errors for the new estimators with those of previously established estimators. We then illustrate our findings in a study examining the efficacy of clozapine versus haloperidol in the treatment of refractory schizophrenia. For data with continuous or binary outcomes in the presence of non-compliance, non-ignorable missing data, and a covariate effect, the new estimators generally performed better than the previously established estimators. In the clozapine trial, the new estimators gave point and interval estimates similar to established estimators. We recommend the new estimators as they are unbiased even when outcomes are not missing at random and they are more efficient than established estimators in the presence of covariate effects under the widest variety of circumstances.     

Inverse probability weighting for covariate adjustment in randomized studies

Covariate adjustment in randomized clinical trials has the potential benefit of precision gain. It also has the potential pitfall of reduced objectivity as it opens the possibility of selecting a ‘favorable’ model that yields strong treatment benefit estimate. Although there is a large volume of statistical literature targeting on the first aspect, realistic solutions to enforce objective inference and improve precision are rare. As a typical randomized trial needs to accommodate many implementation issues beyond statistical considerations, maintaining the objectivity is at least as important as precision gain if not more, particularly from the perspective of the regulatory agencies. In this article, we propose a two‐stage estimation procedure based on inverse probability weighting to achieve better precision without compromising objectivity. The procedure is designed in a way such that the covariate adjustment is performed before seeing the outcome, effectively reducing the possibility of selecting a ‘favorable’ model that yields a strong intervention effect. Both theoretical and numerical properties of the estimation procedure are presented. Application of the proposed method to a real data example is presented. Copyright © 2013 John Wiley & Sons, Ltd.     

A note on non-parametric ANCOVA for covariate adjustment in randomized clinical trials

Koch et al. recently (1998) proposed two covariate-adjusted approaches for the comparison of continuous, ordinal and binary responses in a randomized clinical trial (Statist. Med. 1998; 17: 1863-1892). The first is a randomization approach while the second assumes that the study is a sample of a population. Here, we study the second approach and consider the simplest cases of two treatments with a continuous response and with a binary response. Koch's second approach will be compared with the classical ANCOVA for a continuous response. From this relationship we demonstrate that Koch's method cannot preserve the probability of the type I error. Simulations with continuous responses as well as with binary outcomes confirm the aforementioned theoretical result on the performance of Koch's method under the null hypothesis of no treatment effect. However, this poses only a problem for relatively small to moderate sample sizes. Further, as specified in the original paper of Koch et al., the first approach does preserve the type I error for any sample size, as the P-values can be reported in an exact manner (Statist. Med. 1998; 17: 1863-1892). Finally, we propose a correction factor for Koch's test statistic that better preserves the type I error.     

Covariate adjustment in family-based association studies

Family-based tests of association between a candidate locus and a disease evaluate how often a variant allele at the locus is transmitted from parents to offspring. These tests assume that in the absence of association, an affected offspring is equally likely to have inherited either one of the two homologous alleles carried by a parent. However, transmission distortion was documented in families in which the offspring are unselected for phenotype. Moreover, if offspring genotypes are associated with a risk factor for the disease, transmission distortion to affected offspring can occur in the absence of a causal relation between gene and disease risk. We discuss the appropriateness of adjusting for established risk factors when evaluating association in family-based studies. We present methods for adjusting the transmission/disequilibrium test for risk factors when warranted, and we apply them to data on CYP19 (aromatase) genotypes in nuclear families with multiple cases of breast cancer. Simulations show that when genotypes are correlated with risk factors, the unadjusted test statistics have inflated size, while the adjusted ones do not. The covariate-adjusted tests are less powerful than the unadjusted ones, suggesting the need to check the relationship between genotypes and known risk factors to verify that adjustment is needed. The adjusted tests are most useful for data containing a large proportion of families that lack disease-discordant sibships, i.e., data for which multiple logistic regression of matched sibships would have little power. Software for performing the covariate-adjusted tests is available at http://www.stanford.edu/dept/HRP/epidemiology/COVTDT.     

Variable selection for covariate‐adjusted semiparametric inference in randomized clinical trials

Extensive baseline covariate information is routinely collected on participants in randomized clinical trials, and it is well recognized that a proper covariate‐adjusted analysis can improve the efficiency of inference on the treatment effect. However, such covariate adjustment has engendered considerable controversy, as post hoc selection of covariates may involve subjectivity and may lead to biased inference, whereas prior specification of the adjustment may exclude important variables from consideration. Accordingly, how to select covariates objectively to gain maximal efficiency is of broad interest. We propose and study the use of modern variable selection methods for this purpose in the context of a semiparametric framework, under which variable selection in modeling the relationship between outcome and covariates is separated from estimation of the treatment effect, circumventing the potential for selection bias associated with standard analysis of covariance methods. We demonstrate that such objective variable selection techniques combined with this framework can identify key variables and lead to unbiased and efficient inference on the treatment effect. A critical issue in finite samples is validity of estimators of uncertainty, such as standard errors and confidence intervals for the treatment effect. We propose an approach to estimation of sampling variation of estimated treatment effect and show its superior performance relative to that of existing methods. Copyright © 2012 John Wiley & Sons, Ltd.     

Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements

OBJECTIVE: Randomized controlled trials (RCTs) with dichotomous outcomes may be analyzed with or without adjustment for baseline characteristics (covariates). We studied type I error, power, and potential reduction in sample size with several covariate adjustment strategies. STUDY DESIGN AND SETTING: Logistic regression analysis was applied to simulated data sets (n=360) with different treatment effects, covariate effects, outcome incidences, and covariate prevalences. Treatment effects were estimated with or without adjustment for a single dichotomous covariate. Strategies included always adjusting for the covariate ("prespecified"), or only when the covariate was predictive or imbalanced. RESULTS: We found that the type I error was generally at the nominal level. The power was highest with prespecified adjustment. The potential reduction in sample size was higher with stronger covariate effects (from 3 to 46%, at 50% outcome incidence and covariate prevalence) and independent of the treatment effect. At lower outcome incidences and/or covariate prevalences, the reduction was lower. CONCLUSION: We conclude that adjustment for a predictive baseline characteristic may lead to a potentially important increase in power of analyses of treatment effect. Adjusted analysis should, hence, be considered more often for RCTs with dichotomous outcomes.     

Correcting for non-compliance of repeated binary outcomes in randomized clinical trials: randomized analysis approach

We develop the randomized analysis for repeated binary outcomes with non-compliance. A break randomization-based semi-parametric estimation procedure for both the causal risk difference and the causal risk ratio is proposed for repeated binary data. Although we assume the simple structural models for potential outcomes, we choose to avoid making any assumptions about comparability beyond those implied by randomization at time zero. The proposed methods can incorporate non-compliance information, while preserving the validity of the test of the null hypothesis, and even in the presence of non-random non-compliance can give the estimate of the causal effect that treatment would have if all individuals complied with their assigned treatment. The methods are applied to data from a randomized clinical trial for reduction of febrile neutropenia events among acute myeloid leukaemia patients, in which a prophylactic use of macrophage colony-stimulating factor (M-CSF) was compared to placebo during the courses of intensive chemotherapies.     

Increasing the power of the Mann‐Whitney test in randomized experiments through flexible covariate adjustment

The Mann‐Whitney U test is frequently used to evaluate treatment effects in randomized experiments with skewed outcome distributions or small sample sizes. It may lack power, however, because it ignores the auxiliary baseline covariate information that is routinely collected. Wald and score tests in so‐called probabilistic index models generalize the Mann‐Whitney U test to enable adjustment for covariates, but these may lack robustness by demanding correct model specification and do not lend themselves to small sample inference. Using semiparametric efficiency theory, we here propose an alternative extension of the Mann‐Whitney U test, which increases its power by exploiting covariate information in an objective way and which lends itself to permutation inference. Simulation studies and an application to an HIV clinical trial show that the proposed permutation test attains the nominal Type I error rate and can be drastically more powerful than the classical Mann‐Whitney U test. Copyright © 2014 John Wiley & Sons, Ltd.     

Sample size estimation for stratified individual and cluster randomized trials with binary outcomes

Individual randomized trials (IRTs) and cluster randomized trials (CRTs) with binary outcomes arise in a variety of settings and are often analyzed by logistic regression (fitted using generalized estimating equations for CRTs). The effect of stratification on the required sample size is less well understood for trials with binary outcomes than for continuous outcomes. We propose easy-to-use methods for sample size estimation for stratified IRTs and CRTs and demonstrate the use of these methods for a tuberculosis prevention CRT currently being planned. For both IRTs and CRTs, we also identify the ratio of the sample size for a stratified trial vs a comparably powered unstratified trial, allowing investigators to evaluate how stratification will affect the required sample size when planning a trial. For CRTs, these can be used when the investigator has estimates of the within-stratum intracluster correlation coefficients (ICCs) or by assuming a common within-stratum ICC. Using these methods, we describe scenarios where stratification may have a practically important impact on the required sample size. We find that in the two-stratum case, for both IRTs and for CRTs with very small cluster sizes, there are unlikely to be plausible scenarios in which an important sample size reduction is achieved when the overall probability of a subject experiencing the event of interest is low. When the probability of events is not small, or when cluster sizes are large, however, there are scenarios where practically important reductions in sample size result from stratification.     

Using longitudinal targeted maximum likelihood estimation in complex settings with dynamic interventions

Longitudinal targeted maximum likelihood estimation (LTMLE) has very rarely been used to estimate dynamic treatment effects in the context of time-dependent confounding affected by prior treatment when faced with long follow-up times, multiple time-varying confounders, and complex associational relationships simultaneously. Reasons for this include the potential computational burden, technical challenges, restricted modeling options for long follow-up times, and limited practical guidance in the literature. However, LTMLE has desirable asymptotic properties, ie, it is doubly robust, and can yield valid inference when used in conjunction with machine learning. It also has the advantage of easy-to-calculate analytic standard errors in contrast to the g-formula, which requires bootstrapping. We use a topical and sophisticated question from HIV treatment research to show that LTMLE can be used successfully in complex realistic settings, and we compare results to competing estimators. Our example illustrates the following practical challenges common to many epidemiological studies: (1) long follow-up time (30 months); (2) gradually declining sample size; (3) limited support for some intervention rules of interest; (4) a high-dimensional set of potential adjustment variables, increasing both the need and the challenge of integrating appropriate machine learning methods; and (5) consideration of collider bias. Our analyses, as well as simulations, shed new light on the application of LTMLE in complex and realistic settings: We show that (1) LTMLE can yield stable and good estimates, even when confronted with small samples and limited modeling options; (2) machine learning utilized with a small set of simple learners (if more complex ones cannot be fitted) can outperform a single, complex model, which is tailored to incorporate prior clinical knowledge; and (3) performance can vary considerably depending on interventions and their support in the data, and therefore critical quality checks should accompany every LTMLE analysis. We provide guidance for the practical application of LTMLE.     

Targeted maximum likelihood estimation for a binary treatment: A tutorial

When estimating the average effect of a binary treatment (or exposure) on an outcome, methods that incorporate propensity scores, the G‐formula, or targeted maximum likelihood estimation (TMLE) are preferred over naïve regression approaches, which are biased under misspecification of a parametric outcome model. In contrast propensity score methods require the correct specification of an exposure model. Double‐robust methods only require correct specification of either the outcome or the exposure model. Targeted maximum likelihood estimation is a semiparametric double‐robust method that improves the chances of correct model specification by allowing for flexible estimation using (nonparametric) machine‐learning methods. It therefore requires weaker assumptions than its competitors. We provide a step‐by‐step guided implementation of TMLE and illustrate it in a realistic scenario based on cancer epidemiology where assumptions about correct model specification and positivity (ie, when a study participant had 0 probability of receiving the treatment) are nearly violated. This article provides a concise and reproducible educational introduction to TMLE for a binary outcome and exposure. The reader should gain sufficient understanding of TMLE from this introductory tutorial to be able to apply the method in practice. Extensive R‐code is provided in easy‐to‐read boxes throughout the article for replicability. Stata users will find a testing implementation of TMLE and additional material in the Appendix S1 and at the following GitHub repository: https://github.com/migariane/SIM‐TMLE‐tutorial     

Methods for covariate adjustment in cost-effectiveness analysis that use cluster randomised trials

Statistical methods have been developed for cost-effectiveness analyses of cluster randomised trials (CRTs) where baseline covariates are balanced. However, CRTs may show systematic differences in individual and cluster-level covariates between the treatment groups. This paper presents three methods to adjust for imbalances in observed covariates: seemingly unrelated regression with a robust standard error, a 'two-stage' bootstrap approach combined with seemingly unrelated regression and multilevel models. We consider the methods in a cost-effectiveness analysis of a CRT with covariate imbalance, unequal cluster sizes and a prognostic relationship that varied by treatment group. The cost-effectiveness results differed according to the approach for covariate adjustment. A simulation study then assessed the relative performance of methods for addressing systematic imbalance in baseline covariates. The simulations extended the case study and considered scenarios with different levels of confounding, cluster size variation and few clusters. Performance was reported as bias, root mean squared error and CI coverage of the incremental net benefit. Even with low levels of confounding, unadjusted methods were biased, but all adjusted methods were unbiased. Multilevel models performed well across all settings, and unlike the other methods, reported CI coverage close to nominal levels even with few clusters of unequal sizes.     

Precision of maximum likelihood estimation in adaptive designs

There has been increasing interest in trials that allow for design adaptations like sample size reassessment or treatment selection at an interim analysis. Ignoring the adaptive and multiplicity issues in such designs leads to an inflation of the type 1 error rate, and treatment effect estimates based on the maximum likelihood principle become biased. Whereas the methodological issues concerning hypothesis testing are well understood, it is not clear how to deal with parameter estimation in designs were adaptation rules are not fixed in advanced so that, in practice, the maximum likelihood estimate (MLE) is used. It is therefore important to understand the behavior of the MLE in such designs. The investigation of Bias and mean squared error (MSE) is complicated by the fact that the adaptation rules need not be fully specified in advance and, hence, are usually unknown. To investigate Bias and MSE under such circumstances, we search for the sample size reassessment and selection rules that lead to the maximum Bias or maximum MSE. Generally, this leads to an overestimation of Bias and MSE, which can be reduced by imposing realistic constraints on the rules like, for example, a maximum sample size. We consider designs that start with k treatment groups and a common control and where selection of a single treatment and control is performed at the interim analysis with the possibility to reassess each of the sample sizes. We consider the case of unlimited sample size reassessments as well as several realistically restricted sample size reassessment rules. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

Improving small-sample inference in group randomized trials with binary outcomes

Group Randomized Trials (GRTs) randomize groups of people to treatment or control arms instead of individually randomizing subjects. When each subject has a binary outcome, over-dispersed binomial data may result, quantified as an intra-cluster correlation (ICC). Typically, GRTs have a small number, bin, of independent clusters, each of which can be quite large. Treating the ICC as a nuisance parameter, inference for a treatment effect can be done using quasi-likelihood with a logistic link. A Wald statistic, which, under standard regularity conditions, has an asymptotic standard normal distribution, can be used to test for a marginal treatment effect. However, we have found in our setting that the Wald statistic may have a variance less than 1, resulting in a test size smaller than its nominal value. This problem is most apparent when marginal probabilities are close to 0 or 1, particularly when n is small and the ICC is not negligible. When the ICC is known, we develop a method for adjusting the estimated standard error appropriately such that the Wald statistic will approximately have a standard normal distribution. We also propose ways to handle non-nominal test sizes when the ICC is estimated. We demonstrate the utility of our methods through simulation results covering a variety of realistic settings for GRTs.     

Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: a principled yet flexible approach

There is considerable debate regarding whether and how covariate-adjusted analyses should be used in the comparison of treatments in randomized clinical trials. Substantial baseline covariate information is routinely collected in such trials, and one goal of adjustment is to exploit covariates associated with outcome to increase precision of estimation of the treatment effect. However, concerns are routinely raised over the potential for bias when the covariates used are selected post hoc and the potential for adjustment based on a model of the relationship between outcome, covariates, and treatment to invite a 'fishing expedition' for that leading to the most dramatic effect estimate. By appealing to the theory of semiparametrics, we are led naturally to a characterization of all treatment effect estimators and to principled, practically feasible methods for covariate adjustment that yield the desired gains in efficiency and that allow covariate relationships to be identified and exploited while circumventing the usual concerns. The methods and strategies for their implementation in practice are presented. Simulation studies and an application to data from an HIV clinical trial demonstrate the performance of the techniques relative to the existing methods. Copyright (c) 2007 John Wiley & Sons, Ltd.     

Maximum likelihood estimation with missing outcomes: From simplicity to complexity

Many clinical or prevention studies involve missing or censored outcomes. Maximum likelihood (ML) methods provide a conceptually straightforward approach to estimation when the outcome is partially missing. Methods of implementing ML methods range from the simple to the complex, depending on the type of data and the missing-data mechanism. Simple ML methods for ignorable missing-data mechanisms (when data are missing at random) include complete-case analysis, complete-case analysis with covariate adjustment, survival analysis with covariate adjustment, and analysis via propensity-to-be-missing scores. More complex ML methods for ignorable missing-data mechanisms include the analysis of longitudinal dropouts via a marginal model for continuous data or a conditional model for categorical data. A moderately complex ML method for categorical data with a saturated model and either ignorable or nonignorable missing-data mechanisms is a perfect fit analysis, an algebraic method involving closed-form estimates and variances. A complex and flexible ML method with categorical data and either ignorable or nonignorable missing-data mechanisms is the method of composite linear models, a matrix method requiring specialized software. Except for the method of composite linear models, which can involve challenging matrix specifications, the implementation of these ML methods ranges in difficulty from easy to moderate.     

A covariate adjusted two-stage allocation design for binary responses in randomized clinical trials

In the present work, we develop a two-stage allocation rule for binary response using the log-odds ratio within the Bayesian framework allowing the current allocation to depend on the covariate value of the current subject. We study, both numerically and theoretically, several exact and limiting properties of this design. The applicability of the proposed methodology is illustrated by using some data set. We compare this rule with some of the existing rules by computing various performance measures.     

A simulation study of odds ratio estimation for binary outcomes from cluster randomized trials

We used simulation to compare accuracy of estimation and confidence interval coverage of several methods for analysing binary outcomes from cluster randomized trials. The following methods were used to estimate the population-averaged intervention effect on the log-odds scale: marginal logistic regression models using generalized estimating equations with information sandwich estimates of standard error (GEE); unweighted cluster-level mean difference (CL/U); weighted cluster-level mean difference (CL/W) and cluster-level random effects linear regression (CL/RE). Methods were compared across trials simulated with different numbers of clusters per trial arm, numbers of subjects per cluster, intraclass correlation coefficients (rho), and intervention versus control arm proportions. Two thousand data sets were generated for each combination of design parameter values. The results showed that the GEE method has generally acceptable properties, including close to nominal levels of confidence interval coverage, when a simple adjustment is made for data with relatively few clusters. CL/U and CL/W have good properties for trials where the number of subjects per cluster is sufficiently large and rho is sufficiently small. CL/RE also has good properties in this situation provided a t-distribution multiplier is used for confidence interval calculation in studies with small numbers of clusters. For studies where the number of subjects per cluster is small and rho is large all cluster-level methods may perform poorly for studies with between 10 and 50 clusters per trial arm.