Some fundamental issues with non-inferiority testing in active controlled trials

In an active controlled non-inferiority trial without a placebo arm, it is often not entirely clear what the primary objective is. In many cases the considered goal is to demonstrate that the experimental treatment preserves at least some fraction of the effect of the active control. The active control effect is a parameter, the value of which is unknown. To test the hypothesis of effect preservation, the classical confidence interval approach requires specification of a non-inferiority margin which is a function of the unknown active control effect. When the margin is estimated, it is also not clear what is the relevant type I error of making a false assertion about preservation of the active control effect. The statistical uncertainty of the estimated margin arguably needs to be incorporated in evaluation of the type I error. In this paper we discuss these fundamental issues. We show that the classical confidence interval approach cannot attain the target type I error exactly since this error varies as the sample size or as the values of the nuisance parameters in the active controlled trial change. In contrast, the preservation tests, as proposed in literature, can attain the target type I error rate exactly, regardless of the sample size and the values of the nuisance parameters, but can do so only at the price of several strong assumptions holding that may not be directly verifiable. One assumption is the constancy condition holding whereby the effect of the active control in the historical trial populations is assumed to carry to the population of the active control trial. When this condition is violated, both the confidence interval approach and the preservation test method may be problematic.     

Likelihood ratio and a Bayesian approach were superior to standard noninferiority analysis when the noninferiority margin varied with the control event rate

OBJECTIVE: To present and compare three statistical approaches for analyzing a noninferiority trial when the noninferiority margin depends on the control event rate. STUDY DESIGN AND SETTING: In noninferiority trials with a binary outcome, the noninferiority margin is often defined as a fixed delta, the largest clinically acceptable difference in event rates between treatment groups. An alternative and more flexible approach is to allow delta to vary according to the true event rate in the control group. The appropriate statistical method for evaluating noninferiority with a variable noninferiority margin is not apparent. Three statistical approaches are proposed and compared: an observed event rate (OER) approach based on equating the true control rate to the observed rate, a Bayesian approach, and a likelihood ratio test. RESULTS AND CONCLUSIONS: Simulations studies indicate that the proportion of trials in which noninferiority was erroneously demonstrated was higher for the OER approach than with the Bayesian and likelihood ratio approaches. In some cases, the Type I error rate exceeded 10% for the OER approach. The OER approach is not recommended for the analysis of noninferiority trials with a variable margin of equivalence. The Bayesian and likelihood ratio methods yielded better operating characteristics.     

Controlling the type 1 error rate in non-inferiority trials

Two different approaches have been proposed for establishing the efficacy of an experimental therapy through a non-inferiority trial: The fixed-margin approach involves first defining a non-inferiority margin and then demonstrating that the experimental therapy is not worse than the control by more than this amount, and the synthesis approach involves combining the data from the non-inferiority trial with the data from historical trials evaluating the effect of the control. In this paper, we introduce a unified approach that has both these approaches as special cases and show how the parameters of this approach can be selected to control the unconditional type 1 error rate in the presence of departures from the assumptions of assay sensitivity and constancy. It is shown that the fixed-margin approach can be extremely inefficient and that it is always possible to achieve equivalent control of the unconditional type 1 error rate, with higher power, by using an appropriately chosen synthesis method.     

Choice of lambda-margin and dependency of non-inferiority trials

For a two-arm active control clinical trial designed to test for non-inferiority of the test treatment compared with the active control standard treatment, data of historical studies are often utilized. For example, with a cross-trial comparison approach (also called synthetic approach or lambda-margin approach), the trial is conducted to test the hypothesis that the mean difference or the ratio between the current test product and the active control is no larger than a certain portion of the mean difference or the ratio of the active control and placebo obtained in the historical data when the positive response indicates treatment effectiveness. The regulatory agency usually requires that the clinical trials of two different test treatments are independent in most regular cases. It also requires, in general, two independent trials of the same test treatment in order to provide confirmatory evidence of the efficacy of the test product. In this article, we derived the relationship between the correlation of the test statistics of two trials with the choice of lambda (the percentage to preserve), the sample sizes and variances under the normality assumption. We showed that the smaller a lambda, the higher the correlation between the two non-inferiority tests. It is further shown that when an 80 per cent or larger lambda is used, the correlation can be controlled to be less than 10 per cent if the variances of the response variables in the current trial are not much smaller than those of the historical studies.     

Bio‐creep in non‐inferiority clinical trials

After a non‐inferiority clinical trial, a new therapy may be accepted as effective, even if its treatment effect is slightly smaller than the current standard. It is therefore possible that, after a series of trials where the new therapy is slightly worse than the preceding drugs, an ineffective or harmful therapy might be incorrectly declared efficacious; this is known as ‘bio‐creep’. Several factors may influence the rate at which bio‐creep occurs, including the distribution of the effects of the new agents being tested and how that changes over time, the choice of active comparator, the method used to account for the variability of the estimate of the effect of the active comparator, and changes in the effect of the active comparator from one trial to the next (violations of the constancy assumption). We performed a simulation study to examine which of these factors might lead to bio‐creep and found that bio‐creep was rare, except when the constancy assumption was violated. Copyright © 2010 John Wiley & Sons, Ltd.     

Non-inferiority trials: design concepts and issues - the encounters of academic consultants in statistics

Placebo-controlled trials are the ideal for evaluating medical treatment efficacy. They allow for control of the placebo effect and are most efficient, requiring the smallest numbers of patients to detect a treatment effect. A placebo control is ethically justified if no standard treatment exists, if the standard treatment has not been proven efficacious, there are no risks associated with delaying treatment or escape clauses are included in the protocol. Where possible and justified, they should be the first choice for medical treatment evaluation. Given the large number of proven effective treatments, placebo-controlled trials are often unethical. In these situations active-controlled trials are generally appropriate. The non-inferiority trial is appropriate for evaluation of the efficacy of an experimental treatment versus an active control when it is hypothesized that the experimental treatment may not be superior to a proven effective treatment, but is clinically and statistically not inferior in effectiveness. These trials are not easy to design. An active control must be selected. Good historical placebo-controlled trials documenting the efficacy of the active control must exist. From these historical trials statistical analysis must be performed and clinical judgement applied in order to determine the non-inferiority margin M and to assess assay sensitivity. The latter refers to establishing that the active drug would be superior to the placebo in the setting of the present non-inferiority trial (that is, the constancy assumption). Further, a putative placebo analysis of the new treatment versus the placebo using data from the non-inferiority trial and the historical active versus placebo-controlled trials is needed. Useable placebo-controlled historical trials for the active control are often not available, and determination of assay sensitivity and an appropriate M is difficult and debatable. Serious consideration to expansions of and alternatives to non-inferiority trials are needed.     

Assessing overall evidence from noninferiority trials with shared historical data

For regulatory approval of a new drug, the United States Code of Federal Regulations (CFR) requires ‘substantial evidence’ from ‘adequate and well‐controlled investigations’. This requirement is interpreted in the Food and Drug Administration guidance as the need of ‘at least two adequate and well‐controlled studies, each convincing on its own to establish effectiveness’. The guidance also emphasizes the need of ‘independent substantiation of experimental results from multiple studies’. However, several authors have noted the loss of independence between two noninferiority trials that use the same set of historical data to make inferences, raising questions about whether the CFR requirement is met in noninferiority trials through current practice. In this article, we first propose a statistical interpretation of the CFR requirement in terms of trial‐level and overall type I error rates, which captures the essence of the requirement and can be operationalized for noninferiority trials. We next examine four typical regulatory settings in which the proposed requirement may or may not be fulfilled by existing methods of analysis (fixed margin and synthesis). In situations where the criteria are not met, we then propose adjustments to the existing methods. As illustrated with several examples, our results and findings can be helpful in designing and analyzing noninferiority trials in a way that is both compliant with the regulatory interpretation of the CFR requirement and reasonably powerful. Copyright © 2012 John Wiley & Sons, Ltd.     

A framework for considering the risk‐benefit trade‐off in designing noninferiority trials using composite outcome approaches

When a new treatment regimen is expected to have comparable or slightly worse efficacy to that of the control regimen but has benefits in other domains such as safety and tolerability, a noninferiority (NI) trial may be appropriate but is fraught with difficulty in justifying an acceptable NI margin that is based on both clinical and statistical input. To overcome this, we propose to utilize composite risk‐benefit outcomes that combine elements from domains of importance (eg, efficacy, safety, and tolerability). The composite outcome itself may be analyzed using a superiority framework, or it can be used as a tool at the design stage of a NI trial for selecting an NI margin for efficacy that balances changes in risks and benefits. In the latter case, the choice of NI margin may be based on a novel quantity called the maximum allowable decrease in efficacy (MADE), defined as the marginal difference in efficacy between arms that would yield a null treatment effect for the composite outcome given an assumed distribution for the composite outcome. We observe that MADE: (1) is larger when the safety improvement for the experimental arm is larger, (2) depends on the association between the efficacy and safety outcomes, and (3) depends on the control arm efficacy rate. We use a numerical example for power comparisons between a superiority test for the composite outcome vs a noninferiority test for efficacy using the MADE as the NI margin, and apply the methods to a TB treatment trial.     

Using historical control information for the design and analysis of clinical trials with overdispersed count data

Results from clinical trials are never interpreted in isolation. Previous studies in a similar setting provide valuable information for designing a new trial. For the analysis, however, the use of trial‐external information is challenging and therefore controversial, although it seems attractive from an ethical or efficiency perspective. Here, we consider the formal use of historical control data on lesion counts in a multiple sclerosis trial. The approach to incorporating historical data is Bayesian, in that historical information is captured in a prior that accounts for between‐trial variability and hence leads to discounting of historical data. We extend the meta‐analytic‐predictive approach, a random‐effects meta‐analysis of historical data combined with the prediction of the parameter in the new trial, from normal to overdispersed count data of individual‐patient or aggregate‐trial format. We discuss the prior derivation for the lesion mean count in the control group of the new trial for two populations. For the general population (without baseline enrichment), with 1936 control patients from nine historical trials, between‐trial variability was moderate to substantial, leading to a prior effective sample size of about 45 control patients. For the more homogenous population (with enrichment), with 412 control patients from five historical trials, the prior effective sample size was approximately 63 patients. Although these numbers are small relative to the historical data, they are fairly typical in settings where between‐trial heterogeneity is moderate. For phase II, reducing the number of control patients by 45 or by 63 may be an attractive option in many multiple sclerosis trials. Copyright © 2013 John Wiley & Sons, Ltd.     

Model misspecification sensitivity analysis in estimating causal effects of interventions with non-compliance

Randomized trials often face complications in assessing the effect of treatment because of study participants' non-compliance. If compliance type is observed in both the treatment and control conditions, the causal effect of treatment can be estimated for a targeted subpopulation of interest based on compliance type. However, in practice, compliance type is not observed completely. Given this missing compliance information, the complier average causal effect (CACE) estimation approach provides a way to estimate differential effects of treatments by imposing the exclusion restriction for non-compliers. Under the exclusion restriction, the CACE approach estimates the effect of treatment assignment for compliers, but disallows the effect of treatment assignment for non-compliers. The exclusion restriction plays a key role in separating outcome distributions based on compliance type. However, the CACE estimate can be substantially biased if the assumption is violated. This study examines the bias mechanism in the estimation of CACE when the assumption of the exclusion restriction is violated. How covariate information affects the sensitivity of the CACE estimate to violation of the exclusion restriction assumption is also examined.     

Estimating restricted mean treatment effects with stacked survival models

The difference in restricted mean survival times between two groups is a clinically relevant summary measure. With observational data, there may be imbalances in confounding variables between the two groups. One approach to account for such imbalances is estimating a covariate‐adjusted restricted mean difference by modeling the covariate‐adjusted survival distribution and then marginalizing over the covariate distribution. Because the estimator for the restricted mean difference is defined by the estimator for the covariate‐adjusted survival distribution, it is natural to expect that a better estimator of the covariate‐adjusted survival distribution is associated with a better estimator of the restricted mean difference. We therefore propose estimating restricted mean differences with stacked survival models. Stacked survival models estimate a weighted average of several survival models by minimizing predicted error. By including a range of parametric, semi‐parametric, and non‐parametric models, stacked survival models can robustly estimate a covariate‐adjusted survival distribution and, therefore, the restricted mean treatment effect in a wide range of scenarios. We demonstrate through a simulation study that better performance of the covariate‐adjusted survival distribution often leads to better mean squared error of the restricted mean difference although there are notable exceptions. In addition, we demonstrate that the proposed estimator can perform nearly as well as Cox regression when the proportional hazards assumption is satisfied and significantly better when proportional hazards is violated. Finally, the proposed estimator is illustrated with data from the United Network for Organ Sharing to evaluate post‐lung transplant survival between large‐volume and small‐volume centers. Copyright © 2016 John Wiley & Sons, Ltd.     

Noninferiority hypotheses and choice of noninferiority margin

Ng (Drug Inf. J. 1993; 27:705-719; Drug Inf. J. 2001; 35:1517-1527) proposed that the noninferiority (NI) margin should be a small fraction of the therapeutic effect of the active control as compared with placebo in the setting of testing the NI hypothesis of the mean difference with a continuous outcome. For testing the NI hypothesis of the mean ratio with a continuous outcome, a similar NI margin on a log scale is proposed. This approach may also be applied in the setting of testing the NI hypotheses for survival data based on hazard ratios. Some pitfalls of testing the NI hypotheses with binary endpoints based on the difference or the ratio of proportions will be discussed. Testing the NI hypothesis with binary endpoints based on the odds ratio is proposed.     

Variance reduction in randomised trials by inverse probability weighting using the propensity score

In individually randomised controlled trials, adjustment for baseline characteristics is often undertaken to increase precision of the treatment effect estimate. This is usually performed using covariate adjustment in outcome regression models. An alternative method of adjustment is to use inverse probability‐of‐treatment weighting (IPTW), on the basis of estimated propensity scores. We calculate the large‐sample marginal variance of IPTW estimators of the mean difference for continuous outcomes, and risk difference, risk ratio or odds ratio for binary outcomes. We show that IPTW adjustment always increases the precision of the treatment effect estimate. For continuous outcomes, we demonstrate that the IPTW estimator has the same large‐sample marginal variance as the standard analysis of covariance estimator. However, ignoring the estimation of the propensity score in the calculation of the variance leads to the erroneous conclusion that the IPTW treatment effect estimator has the same variance as an unadjusted estimator; thus, it is important to use a variance estimator that correctly takes into account the estimation of the propensity score. The IPTW approach has particular advantages when estimating risk differences or risk ratios. In this case, non‐convergence of covariate‐adjusted outcome regression models frequently occurs. Such problems can be circumvented by using the IPTW adjustment approach. © 2013 The authors. Statistics in Medicine published by John Wiley & Sons, Ltd.     

Bayesian approach for assessing non‐inferiority in a three‐arm trial with pre‐specified margin

Non‐inferiority trials are becoming increasingly popular for comparative effectiveness research. However, inclusion of the placebo arm, whenever possible, gives rise to a three‐arm trial which has lesser burdensome assumptions than a standard two‐arm non‐inferiority trial. Most of the past developments in a three‐arm trial consider defining a pre‐specified fraction of unknown effect size of reference drug, that is, without directly specifying a fixed non‐inferiority margin. However, in some recent developments, a more direct approach is being considered with pre‐specified fixed margin albeit in the frequentist setup. Bayesian paradigm provides a natural path to integrate historical and current trials' information via sequential learning. In this paper, we propose a Bayesian approach for simultaneous testing of non‐inferiority and assay sensitivity in a three‐arm trial with normal responses. For the experimental arm, in absence of historical information, non‐informative priors are assumed under two situations, namely when (i) variance is known and (ii) variance is unknown. A Bayesian decision criteria is derived and compared with the frequentist method using simulation studies. Finally, several published clinical trial examples are reanalyzed to demonstrate the benefit of the proposed procedure. Copyright © 2015 John Wiley & Sons, Ltd.     

TACT method for non-inferiority testing in active controlled trials

In active controlled trials without a placebo arm, non-inferiority testing is often considered but has different objectives. For the objective of demonstrating the efficacy of an experimental treatment or retention of a fraction of the control effect by the treatment, there are two types of statistical methods for testing - the synthesis method and the confidence interval method. According to the study of Wang, Hung and Tsong, the former is efficient under the so-called constancy condition but may have the alpha error rate inflate rapidly if the condition does not hold. In contrast, the latter method with careful selection of the non-inferiority margin tends to be conservative if the condition holds and may still have a valid alpha error otherwise unless the effect of the active control is less to a large extent in the active controlled trial than in the historical trials. We developed the TACT method, Two-stage Active Control Testing, as a viable compromise between the two methods. Through the TACT method, the uninterpretable non-inferiority testing may be avoided prior to the end of the trial. The TACT method carefully constructed can have a valid alpha error rate and the power close to the synthesis method if the constancy condition holds. In addition, the TACT method is more powerful than the confidence interval method for testing for the efficacy of the new treatment relative to the putative placebo and for showing that the new treatment is not inferior to the active control comparator.     

Reporting of noninferiority trials was incomplete in trial registries

Objective

To examine the registration of noninferiority trials, with a focus on the reporting of study design and noninferiority margins.

Study Design and Setting

Cross-sectional study of registry records of noninferiority trials published from 2005 to 2009 and records of noninferiority trials in the International Standard Randomized Controlled Trial Number (ISRCTN) or ClinicalTrials.gov trial registries. The main outcome was the proportion of records that reported the noninferiority design and margin.

Results

We analyzed 87 registry records of published noninferiority trials and 149 registry records describing noninferiority trials. Thirty-five (40%) of 87 records from published trials described the trial as a noninferiority trial; only two (2%) reported the noninferiority margin. Reporting of the noninferiority design was more frequent in the ISRCTN registry (13 of 18 records, 72%) compared with ClinicalTrials.gov (22 of 69 records, 32%; P = 0.002). Among the 149 records identified in the registries, 13 (9%) reported the noninferiority margin. Only one of the industry-sponsored trial compared with 11 of the publicly funded trials reported the margin (P = 0.001).

Conclusion

Most registry records of noninferiority trials do not mention the noninferiority design and do not include the noninferiority margin. The registration of noninferiority trials is unsatisfactory and must be improved.     

Indirect assessment of confounding: graphic description and limits on effect of adjusting for covariates

Confounding is recognized as a mixing of effects that can lead to spurious conclusions about the association between disease and a putative risk factor. Confounding occurs if an extraneous factor causes disease and is associated with the exposure of interest. Since information on potential confounders may be missing, the investigator may assess confounding indirectly by specifying values for three types of parameters: the prevalence of the covariate in the population, the association between exposure and the covariate, and the effect of the covariate on disease. Qualitative and quantitative arguments suggest that adjustment for a potential confounder may have small effects on the risk ratio, even if the confounder is a strong risk factor. In this report we illustrate graphically the effect that adjustment for a confounder will have on the risk ratio and derive limits for the magnitude of that effect. Our approach allows the investigator to calculate limits for the maximum effect of covariate adjustment, even if only one or two of the relevant parameters can be specified.     

Bayesian sequential meta‐analysis design in evaluating cardiovascular risk in a new antidiabetic drug development program

Recently, the Center for Drug Evaluation and Research at the Food and Drug Administration released a guidance that makes recommendations about how to demonstrate that a new antidiabetic therapy to treat type 2 diabetes is not associated with an unacceptable increase in cardiovascular risk. One of the recommendations from the guidance is that phases II and III trials should be appropriately designed and conducted so that a meta‐analysis can be performed. In addition, the guidance implies that a sequential meta‐analysis strategy could be adopted. That is, the initial meta‐analysis could aim at demonstrating the upper bound of a 95% confidence interval (CI) for the estimated hazard ratio to be < 1.8 for the purpose of enabling a new drug application or a biologics license application. Subsequently after the marketing authorization, a final meta‐analysis would need to show the upper bound to be < 1.3. In this context, we develop a new Bayesian sequential meta‐analysis approach using survival regression models to assess whether the size of a clinical development program is adequate to evaluate a particular safety endpoint. We propose a Bayesian sample size determination methodology for sequential meta‐analysis clinical trial design with a focus on controlling the familywise type I error rate and power. We use the partial borrowing power prior to incorporate the historical survival meta‐data into the Bayesian design. We examine various properties of the proposed methodology, and simulation‐based computational algorithms are developed to generate predictive data at various interim analyses, sample from the posterior distributions, and compute various quantities such as the power and the type I error in the Bayesian sequential meta‐analysis trial design. We apply the proposed methodology to the design of a hypothetical antidiabetic drug development program for evaluating cardiovascular risk. Copyright © 2013 John Wiley & Sons, Ltd.     

The impact of covariate adjustment at randomization and analysis for binary outcomes: understanding differences between superiority and noninferiority trials

The question of when to adjust for important prognostic covariates often arises in the design of clinical trials, and there remain various opinions on whether to adjust during both randomization and analysis, at randomization alone, or at analysis alone. Furthermore, little is known about the impact of covariate adjustment in the context of noninferiority (NI) designs. The current simulation‐based research explores this issue in the NI setting, as compared with the typical superiority setting, by assessing the differential impact on power, type I error, and bias in the treatment estimate as well as its standard error, in the context of logistic regression under both simple and covariate adjusted permuted block randomization algorithms. In both the superiority and NI settings, failure to adjust for covariates that influence outcome in the analysis phase, regardless of prior adjustment at randomization, results in treatment estimates that are biased toward zero, with standard errors that are deflated. However, as no treatment difference is approached under the null hypothesis in superiority and under the alternative in NI, this results in decreased power and nominal or conservative (deflated) type I error in the context of superiority but inflated power and type I error under NI. Results from the simulation study suggest that, regardless of the use of the covariate in randomization, it is appropriate to adjust for important prognostic covariates in analysis, as this yields nearly unbiased estimates of treatment as well as nominal type I error. Copyright © 2015 John Wiley & Sons, Ltd.     

A note on the conservativeness of the confidence interval approach for the selection of non-inferiority margin in the two-arm active-control trial

Compared with placebo-control clinical trials, the interpretation of efficacy results from active-control trials requires more caution. This is because efficacy results from such trials cannot be reliably interpreted without a thorough understanding of the efficacy evidence that formed the basis for the approval of the active control, especially when such drug efficacy is to be established on the basis of clinical evidence from the traditional two-arm active-control clinical equivalence studies as opposed to the multi-arm active control. This is because in addition to over-reliance on the quantification of a clinically irrelevant acceptable margin of inferiority from historical data, such interpretation also depends on cross-trial inference for demonstration of experimental drug effect. We provide a brief overview of some design issues with the traditional two-arm active-control clinical trial and discuss regulators' concern regarding Type I error rate control (with the two most popular methods for the quantification of the non-inferiority margin) in cross-trial demonstration of experimental drug effect. Simulation results are presented to show that the point estimate method provides adequate control of the Type I error rate with > or =75 per cent retention of known active-control effect and that the confidence interval approach is uniformly ultra-conservative. We also report (via a numerical example from real clinical trial data) a couple of potentially less stringent alternative approaches for establishing the non-inferiority of a test drug over a control, which have been used in the past to provide additional efficacy evidence in NDA submission.