A comparison of intent-to-treat and per-protocol results in antibiotic non-inferiority trials

While the intent-to-treat (ITT) analysis is widely accepted for superiority trials, there remains debate about its role in non-inferiority trials. It is often said that the ITT tends to be anti-conservative in the demonstration of non-inferiority. This concern has led to some reliance on per-protocol (PP) analyses that exclude patients on the basis of post-baseline events, despite the inherent bias of such analyses. We compare ITT and PP results from antibiotic trials presented to the public at the FDA's Anti-infective Drug Advisory Committee from 1999 to 2003. While the number of available trials is too small to produce clear conclusions, these data did not support the assumption that the ITT would lead to smaller treatment difference than the PP, in the setting of antibiotic trials. Possible explanations are discussed.     

A comparison of the results of intent‐to‐treat,per‐protocol,and g‐estimation in the presence of non‐random treatment changes in a time‐to‐event non‐inferiority trial

While intent‐to‐treat (ITT) analysis is widely accepted for superiority trials, there remains debate about its role in non‐inferiority trials. It has often been said that ITT analysis tends to be anti‐conservative in demonstrating non‐inferiority, suggesting that per‐protocol (PP) analysis may be preferable for non‐inferiority trials, despite the inherent bias of such analyses. We propose using randomization‐based g‐estimation analyses that more effectively preserve the integrity of randomization than do the more widely used PP analyses. Simulation studies were conducted to investigate the impacts of different types of treatment changes on the conservatism or anti‐conservatism of analyses using the ITT, PP, and g‐estimation methods in a time‐to‐event outcome. The ITT results were anti‐conservative for all simulations. Anti‐conservativeness increased with the percentage of treatment change and was more pronounced for outcome‐dependent treatment changes. PP analysis, in which treatment‐switching cases were censored at the time of treatment change, maintained type I error near the nominal level for independent treatment changes, whereas for outcome‐dependent cases, PP analysis was either conservative or anti‐conservative depending on the mechanism underlying the percentage of treatment changes. G‐estimation analysis maintained type I error near the nominal level even for outcome‐dependent treatment changes, although information on unmeasured covariates is not used in the analysis. Thus, randomization‐based g‐estimation analyses should be used to supplement the more conventional ITT and PP analyses, especially for non‐inferiority trials. Copyright © 2010 John Wiley & Sons, Ltd.     

Correcting for non‐compliance in randomized non‐inferiority trials with active and placebo control using structural models

The three‐arm clinical trial design, which includes a test treatment, an active reference, and placebo control, is the gold standard for the assessment of non‐inferiority. In the presence of non‐compliance, one common concern is that an intent‐to‐treat (ITT) analysis (which is the standard approach to non‐inferiority trials), tends to increase the chances of erroneously concluding non‐inferiority, suggesting that the per‐protocol (PP) analysis may be preferable for non‐inferiority trials despite its inherent bias. The objective of this paper was to develop statistical methodology for dealing with non‐compliance in three‐arm non‐inferiority trials for censored, time‐to‐event data. Changes in treatment were here considered the only form of non‐compliance. An approach using a three‐arm rank preserving structural failure time model and G‐estimation analysis is here presented. Using simulations, the impact of non‐compliance on non‐inferiority trials was investigated in detail using ITT, PP analyses, and the present proposed method. Results indicate that the proposed method shows good characteristics, and that neither ITT nor PP analyses can always guarantee the validity of the non‐inferiority conclusion. A Statistical Analysis System program for the implementation of the proposed test procedure is available from the authors upon request. Copyright © 2014 John Wiley & Sons, Ltd.     

Regaining power lost by non-compliance via full probability modelling

Non-compliance, or non-receipt of randomized intervention, is a common problem in randomized controlled trials. An intention-to-treat (ITT) analysis, which compares individuals as randomized, under-estimates the efficacy of the intervention and leads to a loss of power. We explore the possibility of regaining some of this power in a setting with all-or-nothing compliance, without making any assumptions about the comparability of compliers and non-compliers. Efficacy may be specified as the complier average causal effect (CACE), which is the difference in mean outcome among compliers. Compliance is only partially observed, but under an exclusion restriction assumption, the CACE may be estimated using maximum likelihood. In order to quantify the possible gain in power, we derive an expression for the asymptotic relative efficiency (ARE) of the CACE relative to the ITT effect with a Normally distributed outcome. Under the assumption of a common CACE across covariate strata, the CACE estimate is at least as powerful as ITT analysis. The inclusion of covariates that predict compliance enables an additional gain in power, which is investigated algebraically. Using data from three clinical trials, we obtain values of the ARE ranging up to 1.05 due to covariates alone, and 1.13 due to CACE modelling alone, corresponding to gains in power of up to 5 per cent. This implies that a large gain in power obtained using as-treated or per-protocol analyses is likely to be due to the strong and often implausible assumptions such analyses require to be valid.     

Covariate adjustment in clinical trials with non-ignorable missing data and non-compliance

Estimating causal effects in psychiatric clinical trials is often complicated by treatment non-compliance and missing outcomes. While new estimators have recently been proposed to address these problems, they do not allow for inclusion of continuous covariates. We propose estimators that adjust for continuous covariates in addition to non-compliance and missing data. Using simulations, we compare mean squared errors for the new estimators with those of previously established estimators. We then illustrate our findings in a study examining the efficacy of clozapine versus haloperidol in the treatment of refractory schizophrenia. For data with continuous or binary outcomes in the presence of non-compliance, non-ignorable missing data, and a covariate effect, the new estimators generally performed better than the previously established estimators. In the clozapine trial, the new estimators gave point and interval estimates similar to established estimators. We recommend the new estimators as they are unbiased even when outcomes are not missing at random and they are more efficient than established estimators in the presence of covariate effects under the widest variety of circumstances.     

Three‐sided hypothesis testing: Simultaneous testing of superiority,equivalence and inferiority

We propose three‐sided testing, a testing framework for simultaneous testing of inferiority, equivalence and superiority in clinical trials, controlling for multiple testing using the partitioning principle. Like the usual two‐sided testing approach, this approach is completely symmetric in the two treatments compared. Still, because the hypotheses of inferiority and superiority are tested with one‐sided tests, the proposed approach has more power than the two‐sided approach to infer non‐inferiority or non‐superiority. Applied to the classical point null hypothesis of equivalence, the three‐sided testing approach shows that it is sometimes possible to make an inference on the sign of the parameter of interest, even when the null hypothesis itself could not be rejected. Relationships with confidence intervals are explored, and the effectiveness of the three‐sided testing approach is demonstrated in a number of recent clinical trials. Copyright © 2010 John Wiley & Sons, Ltd.     

Interval estimation of the risk difference in non-compliance randomized trials with repeated binary measurements

In a randomized clinical trial (RCT), we often encounter non-compliance with the treatment protocol for a subset of patients. The intention-to-treat (ITT) analysis is probably the most commonly used method in a RCT with non-compliance. However, the ITT analysis estimates 'the programmatic effectiveness' rather than 'the biological efficacy'. In this paper, we focus attention on the latter index and consider use of the risk difference (RD) to measure the effect of a treatment. Based on a simple additive risk model proposed elsewhere, we develop four asymptotic interval estimators of the RD for repeated binary measurements in a RCT with non-compliance. We apply Monte Carlo simulation to evaluate and compare the finite-sample performance of these interval estimators in a variety of situations. We find that all interval estimators considered here can perform well with respect to the coverage probability. We further find that the interval estimator using a tanh(-1)(x) transformation is probably more precise than the others, while the interval estimator derived from a randomization-based approach may cause a slight loss of precision. When the number of patients per treatment is large and the probability of compliance to an assigned treatment is high, we find that all interval estimators discussed here are essentially equivalent. Finally, we illustrate use of these interval estimators with data simulated from a trial of using macrophage colony-stimulating factor to reduce febrile neutropenia incidence in acute myeloid leukaemia patients.     

Intention-to-treat analysis in cluster randomized trials with noncompliance

In cluster randomized trials (CRTs), individuals belonging to the same cluster are very likely to resemble one another, not only in terms of outcomes but also in terms of treatment compliance behavior. Although the impact of resemblance in outcomes is well acknowledged, little attention has been given to the possible impact of resemblance in compliance behavior. This study defines compliance intraclass correlation as the level of resemblance in compliance behavior among individuals within clusters. On the basis of Monte Carlo simulations, it is demonstrated how compliance intraclass correlation affects power to detect intention-to-treat (ITT) effect in the CRT setting. As a way of improving power to detect ITT effect in CRTs accompanied by noncompliance, this study employs an estimation method, where ITT effect estimates are obtained based on compliance-type-specific treatment effect estimates. A multilevel mixture analysis using an ML-EM estimation method is used for this estimation.     

Choice of lambda-margin and dependency of non-inferiority trials

For a two-arm active control clinical trial designed to test for non-inferiority of the test treatment compared with the active control standard treatment, data of historical studies are often utilized. For example, with a cross-trial comparison approach (also called synthetic approach or lambda-margin approach), the trial is conducted to test the hypothesis that the mean difference or the ratio between the current test product and the active control is no larger than a certain portion of the mean difference or the ratio of the active control and placebo obtained in the historical data when the positive response indicates treatment effectiveness. The regulatory agency usually requires that the clinical trials of two different test treatments are independent in most regular cases. It also requires, in general, two independent trials of the same test treatment in order to provide confirmatory evidence of the efficacy of the test product. In this article, we derived the relationship between the correlation of the test statistics of two trials with the choice of lambda (the percentage to preserve), the sample sizes and variances under the normality assumption. We showed that the smaller a lambda, the higher the correlation between the two non-inferiority tests. It is further shown that when an 80 per cent or larger lambda is used, the correlation can be controlled to be less than 10 per cent if the variances of the response variables in the current trial are not much smaller than those of the historical studies.     

Bio‐creep in non‐inferiority clinical trials

After a non‐inferiority clinical trial, a new therapy may be accepted as effective, even if its treatment effect is slightly smaller than the current standard. It is therefore possible that, after a series of trials where the new therapy is slightly worse than the preceding drugs, an ineffective or harmful therapy might be incorrectly declared efficacious; this is known as ‘bio‐creep’. Several factors may influence the rate at which bio‐creep occurs, including the distribution of the effects of the new agents being tested and how that changes over time, the choice of active comparator, the method used to account for the variability of the estimate of the effect of the active comparator, and changes in the effect of the active comparator from one trial to the next (violations of the constancy assumption). We performed a simulation study to examine which of these factors might lead to bio‐creep and found that bio‐creep was rare, except when the constancy assumption was violated. Copyright © 2010 John Wiley & Sons, Ltd.     

Choosing the analysis population in non-inferiority studies: per protocol or intent-to-treat

For superiority trials, the intent-to-treat population (ITT) is considered the primary analysis population because it tends to avoid the over-optimistic estimates of efficacy that results from a per-protocol (PP) population. However, the roles of the ITT population and PP population in non-inferiority studies are not clearly defined as in superiority trials. In this paper, a simulation study is conducted to systematically investigate the impact of different types of missingness and protocol violations on the conservatism or anticonservatism of analyses based on the ITT and the PP population in non-inferiority trials. We find that conservatism or anticonservatism of the PP or ITT analysis depends on many factors, including the type of protocol deviation and missingness, the treatment trajectory (for longitudinal study) and the method of handling missing data in ITT population. The requirement that non-inferiority be shown for both PP and ITT populations does not necessarily guarantee the validity of a non-inferiority conclusion and a sufficiently powered PP analysis is not necessarily powered for ITT analysis. It is important to assess the potential types and rates of protocol deviation and missingness that might occur in a non-inferiority trial and to obtain some prior knowledge regarding the treatment trajectory of the test treatment versus the active control at the design stage so that a proper analysis plan and appropriate power estimation can be carried out. In general, for the types of protocol violations and missingness considered, we find that hybrid ITT/PP analysis, which excludes non-compliant patients as in the PP analysis and properly addresses the impact of non-trivial missing data as in the MLE-based ITT analysis, is more promising by way of providing reliable non-inferiority tests.     

Bayesian inference for randomized clinical trials with treatment failures

During the course of a clinical trial, subjects may experience treatment failure. For ethical reasons, it is necessary to administer emergency or rescue medications for such subjects. However, the rescue medications may bias the set of response measurements. This bias is of particular concern if a subject has been randomized to the control group, and the rescue medications improve the subject's condition. The standard approach to analysing data from a clinical trial is to perform an intent-to-treat (ITT) analysis, wherein the data are analysed according to treatment randomization. Supplementary analyses may be performed in addition to the ITT analysis to account for the effect of treatment failures and rescue medications. A Bayesian, counterfactual approach, which uses the data augmentation (DA) algorithm, is proposed for supplemental analysis. A simulation study is conducted to compare the operating characteristics of this procedure with a likelihood-based, counterfactual approach based on the EM algorithm. An example from the Asthma Clinical Research Network (ACRN) is used to illustrate the Bayesian procedure.     

Sample size determination for assessing equivalence based on proportion ratio under a randomized trial with non-compliance and missing outcomes

When a generic drug is developed, it is important to assess the equivalence of therapeutic efficacy between the new and the standard drugs. Although the number of publications on testing equivalence and its relevant sample size determination is numerous, the discussion on sample size determination for a desired power of detecting equivalence under a randomized clinical trial (RCT) with non-compliance and missing outcomes is limited. In this paper, we derive under the compound exclusion restriction model the maximum likelihood estimator (MLE) for the ratio of probabilities of response among compliers between two treatments in a RCT with both non-compliance and missing outcomes. Using the MLE with the logarithmic transformation, we develop an asymptotic test procedure for assessing equivalence and find that this test procedure can perform well with respect to type I error based on Monte Carlo simulation. We further develop a sample size calculation formula for a desired power of detecting equivalence at a nominal alpha-level. To evaluate the accuracy of the sample size calculation formula, we apply Monte Carlo simulation again to calculate the simulated power of the proposed test procedure corresponding to the resulting sample size for a desired power of 80 per cent at 0.05 level in a variety of situations. We also include a discussion on determining the optimal ratio of sample size allocation subject to a desired power to minimize a linear cost function and provide a sensitivity analysis of the sample size formula developed here under an alterative model with missing at random.     

A causal model for longitudinal randomised trials with time‐dependent non‐compliance

In the presence of non‐compliance, conventional analysis by intention‐to‐treat provides an unbiased comparison of treatment policies but typically under‐estimates treatment efficacy. With all‐or‐nothing compliance, efficacy may be specified as the complier‐average causal effect (CACE), where compliers are those who receive intervention if and only if randomised to it. We extend the CACE approach to model longitudinal data with time‐dependent non‐compliance, focusing on the situation in which those randomised to control may receive treatment and allowing treatment effects to vary arbitrarily over time. Defining compliance type to be the time of surgical intervention if randomised to control, so that compliers are patients who would not have received treatment at all if they had been randomised to control, we construct a causal model for the multivariate outcome conditional on compliance type and randomised arm. This model is applied to the trial of alternative regimens for glue ear treatment evaluating surgical interventions in childhood ear disease, where outcomes are measured over five time points, and receipt of surgical intervention in the control arm may occur at any time. We fit the models using Markov chain Monte Carlo methods to obtain estimates of the CACE at successive times after receiving the intervention. In this trial, over a half of those randomised to control eventually receive intervention. We find that surgery is more beneficial than control at 6months, with a small but non‐significant beneficial effect at 12months. © 2015 The Authors. Statistics in Medicine Published by JohnWiley & Sons Ltd.     

Adjusting treatment comparisons to account for non-randomized interventions: an example from an angina trial

In a clinical trial where some subjects receive one or more non-randomized interventions during follow-up, primary interest is in the effect of the overall treatment strategies as implemented, but it may also be of interest to adjust treatment comparisons for non-randomized interventions. We consider non-randomized interventions, especially surgical procedures, which only occur when the outcome would otherwise have been poor. Focusing on an outcome measured repeatedly over time, we describe the variety of questions that may be addressed by an adjusted analysis. The adjusted analyses involve new outcome variables defined in terms of the observed outcomes and the history of non-randomized intervention. We also show how to check the assumption that the outcome would otherwise have been poor, and how to do a sensitivity analysis. We apply these methods to a clinical trial comparing initial angioplasty with medical management in patients with angina. We find that the initial benefit of a single angioplasty in reducing angina tends to disappear with time, but a policy of additional interventions as required yields a benefit that is maintained over 4 years. Such methods may be of interest to many pragmatic randomized trials in which the effects of the initial randomized treatments and the effects of the overall treatment strategies as implemented are both of interest.     

Model misspecification sensitivity analysis in estimating causal effects of interventions with non-compliance

Randomized trials often face complications in assessing the effect of treatment because of study participants' non-compliance. If compliance type is observed in both the treatment and control conditions, the causal effect of treatment can be estimated for a targeted subpopulation of interest based on compliance type. However, in practice, compliance type is not observed completely. Given this missing compliance information, the complier average causal effect (CACE) estimation approach provides a way to estimate differential effects of treatments by imposing the exclusion restriction for non-compliers. Under the exclusion restriction, the CACE approach estimates the effect of treatment assignment for compliers, but disallows the effect of treatment assignment for non-compliers. The exclusion restriction plays a key role in separating outcome distributions based on compliance type. However, the CACE estimate can be substantially biased if the assumption is violated. This study examines the bias mechanism in the estimation of CACE when the assumption of the exclusion restriction is violated. How covariate information affects the sensitivity of the CACE estimate to violation of the exclusion restriction assumption is also examined.     

Therapeutic equivalence: fallacies and falsification

The number of studies designed specifically to demonstrate therapeutic equivalence or alternatively non-inferiority of pharmaceutical treatments has increased dramatically in recent years, during which time awareness of the methodological issues has increased. Regulatory authorities have been quick to recognize the need for specific support and have either published or initiated the creation of relevant guidance. Common misconceptions prevail however regarding sample size estimation and the choice of the most appropriate patient population to analyse while other areas such as equivalence margin specification and covariate adjustment have been neglected. This paper challenges some of the regulatory advice and the interpretation that others have made of this guidance with the aim of stimulating further debate.     

Testing non-inferiority (and equivalence) between two diagnostic procedures in paired-sample ordinal data

Before adopting a new diagnostic procedure, which is more convenient and less expensive than the standard existing procedure, it is essentially important to assess whether the diagnostic accuracy of the new procedure is non-inferior (or equivalent) to that of the standard procedure. In this paper, we consider the situation where test responses are on an ordinal scale with more than two categories. We give two definitions of non-inferiority, one in terms of the probability of correctly identifying the case for a randomly selected pair of a case and a non-case over all possible cut-off points, and the other in terms of both the sensitivity and specificity directly. On the basis of large sample theory, we develop two simple test procedures for detecting non-inferiority. We further conduct Monte Carlo simulation to evaluate the finite sample performance of these test procedures. We note that the two asymptotic test procedures proposed here can actually perform reasonably well in a variety of situations even when the numbers of studied subjects from the diseased and non-diseased populations are not large. To illustrate the use of the proposed test procedures, we include an example of determining whether the diagnostic accuracy of using a digitized film is non-inferior to that of using a plain film for screening breast cancer. Finally, we note that the extension of these results to accommodate the case of detecting (two-sided) equivalence is simply straightforward.     

Estimating treatment effects in randomized clinical trials with non-compliance: the impact of maternal smoking on birthweight

This paper assesses the causal impact of late-term (8th month) maternal smoking on birthweight using data from a randomized clinical trial, in which some women were encouraged not to smoke, while others were not. The estimation of treatment effects in this case is made difficult as a result of the presence of non-compliers, women who would not change their smoking status, regardless of the receipt of encouragement. Because these women are not at risk of changing treatment status, treatment effect distributions may be difficult to construct for them. Consequently, the paper focuses on obtaining the distribution of treatment impacts for the sub-set of compliers found in the data. Because compliance status is not observed for all subjects in the sample, a Bayesian finite mixture model is estimated that recovers the treatment effect parameters of interest. The complier average treatment effect implies that smokers give birth to infants weighing 348 g less than those of non-smokers, on average, although the 95% posterior density interval contains zero. The treatment effect is stronger for women who were moderate smokers prior to pregnancy, implying a birthweight difference of 430 g. However, the model predicts that only about 22% of the women in the sample were at risk of changing their smoking behaviour in response to encouragement to quit.     

Non-parametric bounds on treatment effects with non-compliance by covariate adjustment

Consider a clinical trial where subjects are randomized to two treatment arms but compliance to the assignment is not perfect. Concerning this problem, this paper derives non-parametric bounds on treatment effects by making use of the observed covariates information. The new bounds are narrower and more informative than the existing ones. In addition, a new non-parametric point estimation approach is proposed based on stratified analysis. Furthermore, to examine the accuracy of estimating the proposed bounds, we provide variance estimators for the proposed approach. The results of this paper can yield credible information on treatment effects, which will be useful for medical research and public health policy analysis.