Some fundamental issues with non-inferiority testing in active controlled trials

In an active controlled non-inferiority trial without a placebo arm, it is often not entirely clear what the primary objective is. In many cases the considered goal is to demonstrate that the experimental treatment preserves at least some fraction of the effect of the active control. The active control effect is a parameter, the value of which is unknown. To test the hypothesis of effect preservation, the classical confidence interval approach requires specification of a non-inferiority margin which is a function of the unknown active control effect. When the margin is estimated, it is also not clear what is the relevant type I error of making a false assertion about preservation of the active control effect. The statistical uncertainty of the estimated margin arguably needs to be incorporated in evaluation of the type I error. In this paper we discuss these fundamental issues. We show that the classical confidence interval approach cannot attain the target type I error exactly since this error varies as the sample size or as the values of the nuisance parameters in the active controlled trial change. In contrast, the preservation tests, as proposed in literature, can attain the target type I error rate exactly, regardless of the sample size and the values of the nuisance parameters, but can do so only at the price of several strong assumptions holding that may not be directly verifiable. One assumption is the constancy condition holding whereby the effect of the active control in the historical trial populations is assumed to carry to the population of the active control trial. When this condition is violated, both the confidence interval approach and the preservation test method may be problematic.     

Strength of evidence of non‐inferiority trials—The adjustment of the type I error rate in non‐inferiority trials with the synthesis method

In non‐inferiority trials that employ the synthesis method several types of dependencies among test statistics occur due to sharing of the same information from the historical trial. The conditions under which the dependencies appear may be divided into three categories. The first case is when a new drug is approved with single non‐inferiority trial. The second case is when a new drug is approved if two independent non‐inferiority trials show positive results. The third case is when two new different drugs are approved with the same active control. The problem of the dependencies is that they can make the type I error rate deviate from the nominal level. In order to study such deviations, we introduce the unconditional and conditional across‐trial type I error rates when the non‐inferiority margin is estimated from the historical trial, and investigate how the dependencies affect the type I error rates. We show that the unconditional across‐trial type I error rate increases dramatically as does the correlation between two non‐inferiority tests when a new drug is approved based on the positive results of two non‐inferiority trials. We conclude that the conditional across‐trial type I error rate involves the unknown treatment effect in the historical trial. The formulae of the conditional across‐trial type I error rates provide us with a way of investigating the conditional across‐trial type I error rates for various assumed values of the treatment effect in the historical trial. Copyright © 2010 John Wiley & Sons, Ltd.     

Group sequential test strategies for superiority and non-inferiority hypotheses in active controlled clinical trials.

In a group sequential active controlled clinical trial, the study hypothesis may be a superiority hypothesis that an experimental treatment is more effective than the active control therapy or a non-inferiority hypothesis that the treatment is no worse than the active control within some non-inferiority range. When it is necessary to plan for testing the superiority and the non-inferiority hypotheses, we propose an adaptive group sequential closed test strategy by which the sample size is planned for testing superiority and is to be increased for showing non-inferiority given that it is deemed more plausible than superiority based on the observed sample path during the course of the trial. The proposed adaptive test strategy is valid in terms of having the type I error probability maintained at the targeted alpha level for both superiority and non-inferiority. It has power advantage or sample size saving over the traditional group sequential test designed for testing either superiority only or non-inferiority only.     

A note on the conservativeness of the confidence interval approach for the selection of non-inferiority margin in the two-arm active-control trial

Compared with placebo-control clinical trials, the interpretation of efficacy results from active-control trials requires more caution. This is because efficacy results from such trials cannot be reliably interpreted without a thorough understanding of the efficacy evidence that formed the basis for the approval of the active control, especially when such drug efficacy is to be established on the basis of clinical evidence from the traditional two-arm active-control clinical equivalence studies as opposed to the multi-arm active control. This is because in addition to over-reliance on the quantification of a clinically irrelevant acceptable margin of inferiority from historical data, such interpretation also depends on cross-trial inference for demonstration of experimental drug effect. We provide a brief overview of some design issues with the traditional two-arm active-control clinical trial and discuss regulators' concern regarding Type I error rate control (with the two most popular methods for the quantification of the non-inferiority margin) in cross-trial demonstration of experimental drug effect. Simulation results are presented to show that the point estimate method provides adequate control of the Type I error rate with > or =75 per cent retention of known active-control effect and that the confidence interval approach is uniformly ultra-conservative. We also report (via a numerical example from real clinical trial data) a couple of potentially less stringent alternative approaches for establishing the non-inferiority of a test drug over a control, which have been used in the past to provide additional efficacy evidence in NDA submission.     

Modified Haybittle-Peto group sequential designs for testing superiority and non-inferiority hypotheses in clinical trials

In designing an active controlled clinical trial, one sometimes has to choose between a superiority objective (to demonstrate that a new treatment is more effective than an active control therapy) and a non-inferiority objective (to demonstrate that it is no worse than the active control within some pre-specified non-inferiority margin). It is often difficult to decide which study objective should be undertaken at the planning stage when one does not have actual data on the comparative advantage of the new treatment. By making use of recent advances in the theory of efficient group sequential tests, we show how this difficulty can be resolved by a flexible group sequential design that can adaptively choose between the superiority and non-inferiority objectives during interim analyses. While maintaining the type I error probability at a pre-specified level, the proposed test is shown to have power advantage and/or sample size saving over fixed sample size tests for either only superiority or non-inferiority, and over other group sequential designs in the literature.     

TACT method for non-inferiority testing in active controlled trials

In active controlled trials without a placebo arm, non-inferiority testing is often considered but has different objectives. For the objective of demonstrating the efficacy of an experimental treatment or retention of a fraction of the control effect by the treatment, there are two types of statistical methods for testing - the synthesis method and the confidence interval method. According to the study of Wang, Hung and Tsong, the former is efficient under the so-called constancy condition but may have the alpha error rate inflate rapidly if the condition does not hold. In contrast, the latter method with careful selection of the non-inferiority margin tends to be conservative if the condition holds and may still have a valid alpha error otherwise unless the effect of the active control is less to a large extent in the active controlled trial than in the historical trials. We developed the TACT method, Two-stage Active Control Testing, as a viable compromise between the two methods. Through the TACT method, the uninterpretable non-inferiority testing may be avoided prior to the end of the trial. The TACT method carefully constructed can have a valid alpha error rate and the power close to the synthesis method if the constancy condition holds. In addition, the TACT method is more powerful than the confidence interval method for testing for the efficacy of the new treatment relative to the putative placebo and for showing that the new treatment is not inferior to the active control comparator.     

One- and two-stage designs for stratified phase II clinical trials

A global one-sample test for response rates for stratified phase II clinical trials is proposed. Such a test is analogous to that of a stratified log-rank test for time-to-event data. Both one- and two-stage tests are developed, and conditional and unconditional approaches are introduced in each case, where the conditional approach involves conditioning on the observed samples sizes within the strata. The methodology generates samples sizes and stopping boundaries that provide designs with the desired power and type I error probability. These methods are useful for designing stratified phase II clinical trials. An application to a Children's Oncology Group phase II clinical trial in relapsed neuroblastoma patients is presented.     

Two-stage adaptive strategy for superiority and non-inferiority hypotheses in active controlled clinical trials

In active controlled trials without a placebo arm, there are usually two study objectives: to test a superiority hypothesis that the experimental treatment is more effective than the active control therapy, and to test a non-inferiority hypothesis that the experimental treatment is therapeutically no worse than the active control within a defined margin. For a two-stage adaptive design, it is not necessary to give a fixed sample size calculation at the planning stage of the study when treatment effect information is often insufficient. Instead, decision and estimation of the design specifications can be made more reliably after the first stage when interim results are available. We propose the use of conditional power approach to determine the sample size and critical values for testing the superiority and non-inferiority hypotheses for the second stage based on the observed result of the first stage. The proposed adaptive procedure preserves the overall type I error rate for both superiority and non-inferiority, and has the flexibility of early termination of the study (for futility or efficacy) or extending the study by appropriate sample size.     

Statistical considerations of FDA and CPMP rules for the investigation of new anti-bacterial products

In this paper I investigate statistical properties of some guidance given by the FDA and by the CPMP on the planning, conduct and analysis of clinical trials with new anti-bacterial substances using an active control design. It is demonstrated that the non-inferiority margin proposed by the FDA has some undesirable features, and that the CPMP guidance may need further interpretation with respect to a statement that the non-inferiority margin may be smaller than 10 per cent for response rates >90 per cent. A new margin is proposed that combines the desirable properties from both the FDA and the CPMP guidance. It is also shown that the approximate unconditional tests that are in use in such trials are quite unreliable with respect to preserving the nominal type I error. Unconditional exact tests are presented as a remedy for this issue.     

A framework for two-stage adaptive procedures to simultaneously test non-inferiority and superiority

In clinical trials it is often desirable to test for non-inferiority and for superiority simultaneously. For such a situation a two-stage adaptive procedure may be advantageous to a conventional single-stage procedure because a two-stage adaptive procedure allows the design of stage II, including the main study objective and sample size, to depend on the outcome of stage I. We propose a framework for designing two-stage adaptive procedures with a possible switch of the primary study objectives at the end of stage I between non-inferiority and superiority. The framework permits control of the type I error rate and specification of the unconditional powers and maximum sample size for each of non-inferiority and superiority objectives. The actions at the end of stage I are predetermined as functions of the stage I observations, thus making specification of the unconditional powers possible. Based on the results at the end of stage I, the primary objective for stage II is chosen, and sample sizes and critical values for stage II are determined.     

An approximate unconditional test of non-inferiority between two proportions

This paper investigates an approximate unconditional test for non-inferiority between two independent binomial proportions. The P-value of the approximate unconditional test is evaluated using the maximum likelihood estimate of the nuisance parameter. In this paper, we clarify some differences in defining the rejection regions between the approximate unconditional and conventional conditional or unconditional exact test. We compare the approximate unconditional test with the asymptotic test and unconditional exact test by Chan (Statistics in Medicine, 17, 1403-1413, 1998) with respect to the type I error and power. In general, the type I errors and powers are in the decreasing order of the asymptotic, approximate unconditional and unconditional exact tests. In many cases, the type I errors are above the nominal level from the asymptotic test, and are below the nominal level from the unconditional exact test. In summary, when the non-inferiority test is formulated in terms of the difference between two proportions, the approximate unconditional test is the most desirable, because it is easier to implement and generally more powerful than the unconditional exact test and its size rarely exceeds the nominal size. However, when a test between two proportions is formulated in terms of the ratio of two proportions, such as a test of efficacy, more caution should be made in selecting a test procedure. The performance of the tests depends on the sample size and the range of plausible values of the nuisance parameter. Published in 2000 by John Wiley & Sons, Ltd.     

A two-stage sample size recalculation procedure for placebo- and active-controlled non-inferiority trials

Many non-inferiority trials of a test treatment versus an active control may also, if ethical, incorporate a placebo arm. Inclusion of a placebo arm enables a direct assessment of assay sensitivity. It also allows construction of a non-inferiority test that avoids the problematic specification of an absolute non-inferiority margin, and instead evaluates whether the test treatment preserves a pre-specified proportion of the effect of the active control over placebo. We describe a two-stage procedure for sample size recalculation in such a setting that maintains the desired power more closely than a fixed sample approach when the magnitude of the effect of the active control differs from that anticipated. We derive an allocation rule for randomization under which the procedure preserves the type I error rate, and show that this coincides with that previously presented for optimal allocation of the sample size among the three treatment arms.     

A frequentist design for basket trials using adaptive lasso

A basket trial aims to expedite the drug development process by evaluating a new therapy in multiple populations within the same clinical trial. Each population, referred to as a “basket”, can be defined by disease type, biomarkers, or other patient characteristics. The objective of a basket trial is to identify the subset of baskets for which the new therapy shows promise. The conventional approach would be to analyze each of the baskets independently. Alternatively, several Bayesian dynamic borrowing methods have been proposed that share data across baskets when responses appear similar. These methods can achieve higher power than independent testing in exchange for a risk of some inflation in the type 1 error rate. In this paper we propose a frequentist approach to dynamic borrowing for basket trials using adaptive lasso. Through simulation studies we demonstrate adaptive lasso can achieve similar power and type 1 error to the existing Bayesian methods. The proposed approach has the benefit of being easier to implement and faster than existing methods. In addition, the adaptive lasso approach is very flexible: it can be extended to basket trials with any number of treatment arms and any type of endpoint.     

Design and analysis of non-inferiority mortality trials in oncology

The recent revision of the Declaration of Helsinki and the existence of many new therapies that affect survival or serious morbidity, and that therefore cannot be denied patients, have generated increased interest in active-control trials, particularly those intended to show equivalence or non-inferiority to the active-control. A non-inferiority hypothesis has historically been formulated in terms of a fixed margin. This margin was historically designed to exclude a 'clinically meaningful difference', but has become recognized that the margin must also be no larger than the assured effect of the control in the new study. Depending on how this 'assured effect' is determined or estimated, the selected margin may be very small, leading to very large sample sizes, especially when there is an added requirement that a loss of some specified fraction of the assured effect must be ruled out. In cases where it is appropriate, this paper proposes non-inferiority analyses that do not involve a fixed margin, but can be described as a two confidence interval procedure that compares the 95 per cent two-sided CI for the difference between the treatment and the control to a confidence interval for the control effect (based on a meta-analysis of historical data comparing the control to placebo) that is chosen to preserve a study-wide type I error rate of about 0.025 (similar to the usual standard for a superiority trial) for testing for retention of a prespecified fraction of the control effect. The approach assumes that the estimate of the historical active-control effect size is applicable in the current study. If there is reason to believe that this effect size is diminished (for example, improved concomitant therapies) the estimate of this historical effect could be reduced appropriately. The statistical methodology for testing this non-inferiority hypothesis is developed for a hazard ratio (rather than an absolute difference between treatments, because a hazard ratio seems likely to be less population dependent than the absolute difference). In the case of oncology, the hazard ratio is the usual way of comparing treatments with respect to time to event (time to progression or survival) endpoints. The proportional hazards assumption is regarded as reasonable (approximately holding). The testing procedures proposed are conditionally equivalent to two confidence interval procedures that relax the conservatism of two 95 per cent confidence interval testing procedures and preserve the type I error rate at a one-sided 0.025 level. An application of this methodology to Xeloda, a recently approved drug for the treatment of metastatic colorectal cancers, is illustrated. Other methodologies are also described and assessed - including a point estimate procedure, a Bayesian procedure and two delta-method confidence interval procedures. Published in 2003 by John Wiley & Sons, Ltd.     

A proof of concept phase II non-inferiority criterion

Traditional phase III non-inferiority trials require compelling evidence that the treatment vs control effect bfθ is better than a pre-specified non-inferiority margin θ(NI) . The standard approach compares this margin to the 95 per cent confidence interval of the effect parameter. In the phase II setting, in order to declare Proof of Concept (PoC) for non-inferiority and proceed in the development of the drug, different criteria that are specifically tailored toward company internal decision making may be more appropriate. For example, less evidence may be needed as long as the effect estimate is reasonably convincing. We propose a non-inferiority design that addresses the specifics of the phase II setting. The requirements are that (1) the effect estimate be better than a critical threshold θ(C), and (2) the type I error with regard to θ(NI) is controlled at a pre-specified level. This design is compared with the traditional design from a frequentist as well as a Bayesian perspective, where the latter relies on the Level of Proof (LoP) metric, i.e. the probability that the true effect is better than effect values of interest. Clinical input is required to establish the value θ(C), which makes the design transparent and improves interactions within clinical teams. The proposed design is illustrated for a non-inferiority trial for a time-to-event endpoint in oncology.     

Planning and analysis of three-arm non-inferiority trials with binary endpoints

Three-arm trials including an experimental treatment, an active control and a placebo group are frequently preferred for the assessment of non-inferiority. In contrast to two-arm non-inferiority studies, these designs allow a direct proof of efficacy of a new treatment by comparison with placebo. As a further advantage, the test problem for establishing non-inferiority can be formulated in such a way that rejection of the null hypothesis assures that a pre-defined portion of the (unknown) effect the reference shows versus placebo is preserved by the treatment under investigation. We present statistical methods for this study design and the situation of a binary outcome variable. Asymptotic test procedures are given and their actual type I error rates are calculated. Approximate sample size formulae are derived and their accuracy is discussed. Furthermore, the question of optimal allocation of the total sample size is considered. Power properties of the testing strategy including a pre-test for assay sensitivity are presented. The derived methods are illustrated by application to a clinical trial in depression.     

Tests of equivalence and non-inferiority for diagnostic accuracy based on the paired areas under ROC curves

Assessment of equivalence or non-inferiority in accuracy between two diagnostic procedures often involves comparisons of paired areas under the receiver operating characteristic (ROC) curves. With some pre-specified clinically meaningful limits, the current approach to evaluating equivalence is to perform the two one-sided tests (TOST) based on the difference in paired areas under ROC curves estimated by the non-parametric method. We propose to use the standardized difference for assessing equivalence or non-inferiority in diagnostic accuracy based on paired areas under ROC curves between two diagnostic procedures. The bootstrap technique is also suggested for both non-parametric method and the standardized difference approach. A simulation study was conducted empirically to investigate the size and power of the four methods for various combinations of distributions, data types, sample sizes, and different correlations. Simulation results demonstrate that the bootstrap procedure of the standardized difference approach not only can adequately control the type I error rate at the nominal level but also provides equivalent power under both symmetrical and skewed distributions. A numerical example using published data illustrates the proposed methods.     

An efficient and exact approach for detecting trends with binary endpoints

Lloyd (Aust. Nz. J. Stat., 50, 329-345, 2008) developed an exact testing approach to control for nuisance parameters, which was shown to be advantageous in testing for differences between two population proportions. We utilized this approach to obtain unconditional tests for trends in 2 × K contingency tables. We compare the unconditional procedure with other unconditional and conditional approaches based on the well-known Cochran-Armitage test statistic. We give an example to illustrate the approach, and provide a comparison between the methods with regards to type I error and power. The proposed procedure is preferable because it is less conservative and has superior power properties.     

A boundary‐optimized rejection region test for the two‐sample binomial problem

Testing the equality of 2 proportions for a control group versus a treatment group is a well‐researched statistical problem. In some settings, there may be strong historical data that allow one to reliably expect that the control proportion is one, or nearly so. While one‐sample tests or comparisons to historical controls could be used, neither can rigorously control the type I error rate in the event the true control rate changes. In this work, we propose an unconditional exact test that exploits the historical information while controlling the type I error rate. We sequentially construct a rejection region by first maximizing the rejection region in the space where all controls have an event, subject to the constraint that our type I error rate does not exceed α for any true event rate; then with any remaining α we maximize the additional rejection region in the space where one control avoids the event, and so on. When the true control event rate is one, our test is the most powerful nonrandomized test for all points in the alternative space. When the true control event rate is nearly one, we demonstrate that our test has equal or higher mean power, averaging over the alternative space, than a variety of well‐known tests. For the comparison of 4 controls and 4 treated subjects, our proposed test has higher power than all comparator tests. We demonstrate the properties of our proposed test by simulation and use our method to design a malaria vaccine trial.     

Increasing the sample size when the unblinded interim result is promising

Increasing the sample size based on unblinded interim result may inflate the type I error rate and appropriate statistical adjustments may be needed to control the type I error rate at the nominal level. We briefly review the existing approaches which allow early stopping due to futility, or change the test statistic by using different weights, or adjust the critical value for final test, or enforce rules for sample size recalculation. The implication of early stopping due to futility and a simple modification to the weighted Z-statistic approach are discussed. In this paper, we show that increasing the sample size when the unblinded interim result is promising will not inflate the type I error rate and therefore no statistical adjustment is necessary. The unblinded interim result is considered promising if the conditional power is greater than 50 per cent or equivalently, the sample size increment needed to achieve a desired power does not exceed an upper bound. The actual sample size increment may be determined by important factors such as budget, size of the eligible patient population and competition in the market. The 50 per cent-conditional-power approach is extended to a group sequential trial with one interim analysis where a decision may be made at the interim analysis to stop the trial early due to a convincing treatment benefit, or to increase the sample size if the interim result is not as good as expected. The type I error rate will not be inflated if the sample size may be increased only when the conditional power is greater than 50 per cent. If there are two or more interim analyses in a group sequential trial, our simulation study shows that the type I error rate is also well controlled.