Graphical approaches using a Bonferroni mixture of weighted Simes tests

Graphical approaches to multiple testing procedures are very flexible and easy to communicate with non‐statisticians. The availability of the R package gMCP further propelled the application of graphical approaches in randomized clinical trials. Bretz et al. (Biometrical Journal 2011; 53:894–913) introduced a class of nonparametric testing procedures based on a Bonferroni mixture of weighted Simes tests for intersection hypotheses. Such approaches are extremely useful when the conditions for the Simes test are known to hold for hypotheses within certain subsets but may not hold for hypotheses across subsets. We describe the calculation of adjusted p‐values for such approaches, which is currently not available in the gMCP package. We also optimize the generation of the weights for each intersection hypothesis in the closure of a graph‐based multiple testing procedure, which can dramatically reduce the computing time for simulation‐based power calculations. We show the validity of the Simes test for comparing several treatments with a control, performing noninferiority and superiority tests, or testing the treatment effect in an overall and a subpopulation for the normal, binary, count, and time‐to‐event data. The proposed method is illustrated using an example for designing a confirmatory clinical trial. Copyright © 2016 John Wiley & Sons, Ltd.     

Simultaneous inference for factorial multireader diagnostic trials

We study inference methods for the analysis of multireader diagnostic trials. In these studies, data are usually collected in terms of a factorial design involving the factors Modality and Reader. Furthermore, repeated measures appear in a natural way since the same patient is observed under different modalities by several readers and the repeated measures may have a quite involved dependency structure. The hypotheses are formulated in terms of the areas under the ROC curves. Currently, only global testing procedures exist for the analysis of such data. We derive rank‐based multiple contrast test procedures and simultaneous confidence intervals which take the correlation between the test statistics into account. The procedures allow for testing arbitrary multiple hypotheses. Extensive simulation studies show that the new approaches control the nominal type 1 error rate very satisfactorily. A real data set illustrates the application of the proposed methods.     

Compatible simultaneous lower confidence bounds for the Holm procedure and other Bonferroni-based closed tests

We consider the problem of simultaneously testing multiple one-sided null hypotheses. Single-step procedures, such as the Bonferroni test, are characterized by the fact that the rejection or non-rejection of a null hypothesis does not take the decision for any other hypothesis into account. For stepwise test procedures, such as the Holm procedure, the rejection or non-rejection of a null hypothesis may depend on the decision of other hypotheses. It is well known that stepwise test procedures are by construction more powerful than their single-step counterparts. This power advantage, however, comes only at the cost of increased difficulties in constructing compatible simultaneous confidence intervals for the parameters of interest. For example, such simultaneous confidence intervals are easily obtained for the Bonferroni method, but surprisingly hard to derive for the Holm procedure. In this paper, we discuss the inherent problems and show that ad hoc solutions used in practice typically do not control the pre-specified simultaneous confidence level. Instead, we derive simultaneous confidence intervals that are compatible with a certain class of closed test procedures using weighted Bonferroni tests for each intersection hypothesis. The class of multiple test procedures covered in this paper includes gatekeeping procedures based on Bonferroni adjustments, fixed sequence procedures, the simple weighted or unweighted Bonferroni procedure by Holm and the fallback procedure. We illustrate the results with a numerical example.     

Fallback tests for co‐primary endpoints

When efficacy of a treatment is measured by co‐primary endpoints, efficacy is claimed only if for each endpoint an individual statistical test is significant at level α. While such a strategy controls the family‐wise type I error rate (FWER), it is often strictly conservative and allows for no inference if not all null hypotheses can be rejected. In this paper, we investigate fallback tests, which are defined as uniform improvements of the classical test for co‐primary endpoints. They reject whenever the classical test rejects but allow for inference also in settings where only a subset of endpoints show a significant effect. Similarly to the fallback tests for hierarchical testing procedures, these fallback tests for co‐primary endpoints allow one to continue testing even if the primary objective of the trial was not met. We propose examples of fallback tests for two and three co‐primary endpoints that control the FWER in the strong sense under the assumption of multivariate normal test statistics with arbitrary correlation matrix and investigate their power in a simulation study. The fallback procedures for co‐primary endpoints are illustrated with a clinical trial in a rare disease and a diagnostic trial. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.     

Multiplicity adjustments in testing for bioequivalence

Bioequivalence of two drugs is usually demonstrated by rejecting two one‐sided null hypotheses using the two one‐sided tests for pharmacokinetic parameters: area under the concentration‐time curve (AUC) and maximum concentration (Cmax). By virtue of the intersection–union test, there is no need for multiplicity adjustment in testing the two one‐sided null hypotheses within each parameter. However, the decision rule for bioequivalence often requires equivalence to be achieved simultaneously on both parameters that contain four one‐sided null hypotheses together; without adjusting for multiplicity, the family wise error rate (FWER) could fail to be controlled at the nominal type‐I error rate α. The multiplicity issue for bioequivalence in this regard is scarcely discussed in the literature. To address this issue, we propose two approaches including a closed test procedure that controls FWER for the simultaneous AUC and Cmax bioequivalence and requires no adjustment of the type‐I error, and an alpha‐adaptive sequential testing (AAST) that controls FWER by pre‐specifying the significance level on AUC (α₁) and obtaining it for Cmax (α₂) adaptively after testing of AUC. While both methods control FWER, the closed test requires testing of eight intersection null hypotheses each at α, and AAST is at times accomplished through a slight deduction in α₁ and no deduction in α₂ relative to α. The latter considers equivalence reached in AUC a higher importance than that in Cmax. Illustrated with published data, the two approaches, although operate differently, can lead to the same substantive conclusion and are better than a traditional method like Bonferroni adjustment. Copyright © 2014 John Wiley & Sons, Ltd.     

Advanced multiplicity adjustment methods in clinical trials

During the last decade, many novel approaches for addressing multiplicity problems arising in clinical trials have been introduced in the literature. These approaches provide great flexibility in addressing given clinical trial objectives and yet maintain strong control of the familywise error rate. In this tutorial article, we review multiple testing strategies that are related to the following: (a) recycling local significance levels to test hierarchically ordered hypotheses; (b) adapting the significance level for testing a hypothesis to the findings of testing previous hypotheses within a given test sequence, also in view of certain consistency requirements; (c) grouping hypotheses into hierarchical families of hypotheses along with recycling the significance level between those families; and (d) graphical methods that permit repeated recycling of the significance level. These four different methodologies are related to each other, and we point out some connections as we describe and illustrate them. By contrasting the main features of these approaches, our objective is to help practicing statisticians to select an appropriate method for their applications. In this regard, we discuss how to apply some of these strategies to clinical trial settings and provide algorithms to calculate critical values and adjusted p‐values for their use in practice. The methods are illustrated with several numerical examples. Copyright © 2013 John Wiley & Sons, Ltd.     

A Powerful Method for Combining P‐Values in Genomic Studies

After genetic regions have been identified in genomewide association studies (GWAS), investigators often follow up with more targeted investigations of specific regions. These investigations typically are based on single nucleotide polymorphisms (SNPs) with dense coverage of a region. Methods are thus needed to test the hypothesis of any association in given genetic regions. Several approaches for combining P‐values obtained from testing individual SNP hypothesis tests are available. We recently proposed a sequential procedure for testing the global null hypothesis of no association in a region. When this global null hypothesis is rejected, this method provides a list of significant hypotheses and has weak control of the family‐wise error rate. In this paper, we devise a permutation‐based version of the test that accounts for correlations of tests based on SNPs in the same genetic region. Based on simulated data, the method has correct control of the type I error rate and higher or comparable power to other tests.     

On generalized fixed sequence procedures for controlling the FWER

Testing a sequence of pre‐ordered hypotheses to decide which of these can be rejected or accepted while controlling the familywise error rate (FWER) is of importance in many scientific studies such as clinical trials. In this paper, we first introduce a generalized fixed sequence procedure whose critical values are defined by using a function of the numbers of rejections and acceptances, and which allows follow‐up hypotheses to be tested even if some earlier hypotheses are not rejected. We then construct the least favorable configuration for this generalized fixed sequence procedure and present a sufficient condition for the FWER control under arbitrary dependence. Based on the condition, we develop three new generalized fixed sequence procedures controlling the FWER under arbitrary dependence. We also prove that each generalized fixed sequence procedure can be described as a specific closed testing procedure. Through simulation studies and a clinical trial example, we compare the power performance of these proposed procedures with those of the existing FWER controlling procedures. Finally, when the pairwise joint distributions of the true null p‐values are known, we further improve these procedures by incorporating pairwise correlation information while maintaining the control of the FWER. Copyright © 2015 John Wiley & Sons, Ltd.     

Consonant closed likelihood ratio test procedures with application to dose–response study

Coherence and Consonance are two important concepts in multiple testing procedures (MTP). Closed test is a well‐known coherent hypothesis testing procedure, which is not necessarily consonant. In this paper, we propose two consonant closed likelihood ratio tests which are compared with the step‐down Dunnett test (SD) and Dunnett–Tamhane step‐up test (SU) in several aspects. Simulation and dose–response study examples show that the new procedures have certain advantages over the SD and SU test. For example, the rejection region of our new procedures is larger than that of the SD test particularly when most of the null hypotheses are false. The new procedures control the family‐wise error rate (FWER) strongly without the equal correlation assumption, which is a necessary condition for the SU test. In terms of computing effort, the new procedures require similar moderate computation for critical constants as the SU test and they follow the same steps as any closed test procedures. We also provide guideline in applying consonance adjustment in multivariate analysis with mixed model and repeated measurements. Copyright © 2010 John Wiley & Sons, Ltd.     

Understanding MCP‐MOD dose finding as a method based on linear regression

MCP‐MOD is a testing and model selection approach for clinical dose finding studies. During testing, contrasts of dose group means are derived from candidate dose response models. A multiple‐comparison procedure is applied that controls the alpha level for the family of null hypotheses associated with the contrasts. Provided at least one contrast is significant, a corresponding set of “good” candidate models is identified. The model generating the most significant contrast is typically selected. There have been numerous publications on the method. It was endorsed by the European Medicines Agency. The MCP‐MOD procedure can be alternatively represented as a method based on simple linear regression, where “simple” refers to the inclusion of an intercept and a single predictor variable, which is a transformation of dose. It is shown that the contrasts are equal to least squares linear regression slope estimates after a rescaling of the predictor variables. The test for each contrast is the usual t statistic for a null slope parameter, except that a variance estimate with fewer degrees of freedom is used in the standard error. Selecting the model corresponding to the most significant contrast P value is equivalent to selecting the predictor variable yielding the smallest residual sum of squares. This criteria orders the models like a common goodness‐of‐fit test, but it does not assure a good fit. Common inferential methods applied to the selected model are subject to distortions that are often present following data‐based model selection.     

Type‐II generalized family‐wise error rate formulas with application to sample size determination

Multiple endpoints are increasingly used in clinical trials. The significance of some of these clinical trials is established if at least r null hypotheses are rejected among m that are simultaneously tested. The usual approach in multiple hypothesis testing is to control the family‐wise error rate, which is defined as the probability that at least one type‐I error is made. More recently, the q‐generalized family‐wise error rate has been introduced to control the probability of making at least q false rejections. For procedures controlling this global type‐I error rate, we define a type‐II r‐generalized family‐wise error rate, which is directly related to the r‐power defined as the probability of rejecting at least r false null hypotheses. We obtain very general power formulas that can be used to compute the sample size for single‐step and step‐wise procedures. These are implemented in our R package rPowerSampleSize available on the CRAN, making them directly available to end users. Complexities of the formulas are presented to gain insight into computation time issues. Comparison with Monte Carlo strategy is also presented. We compute sample sizes for two clinical trials involving multiple endpoints: one designed to investigate the effectiveness of a drug against acute heart failure and the other for the immunogenicity of a vaccine strategy against pneumococcus. Copyright © 2016 John Wiley & Sons, Ltd.     

Exact approaches for testing hypotheses based on the intra‐class kappa coefficient

Testing involving the intra‐class kappa coefficient is commonly performed in order to assess agreement involving categorical ratings. A number of procedures have been proposed, which make use of the limiting null distribution as the sample size goes to infinity in order to compute the observed significance. As with many tests based on asymptotic null distributions, these tests are associated with problematic type I error control for selected sample sizes and points in the parameter space. We propose and study a collection of exact testing approaches for both the one‐sample and K‐sample scenarios. For the one‐sample case, p‐values are obtained using the exact distribution of the test statistic conditional on a sufficient statistic. In addition, unconditional approaches are considered on the basis of maximization across the nuisance parameter space. Numerical evaluation reveals advantages with the exact unconditional procedures. Copyright © 2014 John Wiley & Sons, Ltd.     

Symmetric graphs for equally weighted tests,with application to the Hochberg procedure

The graphical approach to multiple testing provides a convenient tool for designing, visualizing, and performing multiplicity adjustments in confirmatory clinical trials while controlling the familywise error rate. It assigns a set of weights to each intersection null hypothesis within the closed test framework. These weights form the basis for intersection tests using weighted individual p-values, such as the weighted Bonferroni test. In this paper, we extend the graphical approach to intersection tests that assume equal weights for the elementary null hypotheses associated with any intersection hypothesis, including the Hochberg procedure as well as omnibus tests such as Fisher's combination, O'Brien's, and F tests. More specifically, we introduce symmetric graphs that generate sets of equal weights so that the aforementioned tests can be applied with the graphical approach. In addition, we visualize the Hochberg and the truncated Hochberg procedures in serial and parallel gatekeeping settings using symmetric component graphs. We illustrate the method with two clinical trial examples.     

Multi‐arm group sequential designs with a simultaneous stopping rule

Multi‐arm group sequential clinical trials are efficient designs to compare multiple treatments to a control. They allow one to test for treatment effects already in interim analyses and can have a lower average sample number than fixed sample designs. Their operating characteristics depend on the stopping rule: We consider simultaneous stopping, where the whole trial is stopped as soon as for any of the arms the null hypothesis of no treatment effect can be rejected, and separate stopping, where only recruitment to arms for which a significant treatment effect could be demonstrated is stopped, but the other arms are continued. For both stopping rules, the family‐wise error rate can be controlled by the closed testing procedure applied to group sequential tests of intersection and elementary hypotheses. The group sequential boundaries for the separate stopping rule also control the family‐wise error rate if the simultaneous stopping rule is applied. However, we show that for the simultaneous stopping rule, one can apply improved, less conservative stopping boundaries for local tests of elementary hypotheses. We derive corresponding improved Pocock and O'Brien type boundaries as well as optimized boundaries to maximize the power or average sample number and investigate the operating characteristics and small sample properties of the resulting designs. To control the power to reject at least one null hypothesis, the simultaneous stopping rule requires a lower average sample number than the separate stopping rule. This comes at the cost of a lower power to reject all null hypotheses. Some of this loss in power can be regained by applying the improved stopping boundaries for the simultaneous stopping rule. The procedures are illustrated with clinical trials in systemic sclerosis and narcolepsy. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

False discovery rate and permutation test: An evaluation in ERP data analysis

Current analysis of event‐related potentials (ERP) data is usually based on the a priori selection of channels and time windows of interest for studying the differences between experimental conditions in the spatio‐temporal domain. In this work we put forward a new strategy designed for situations when there is not a priori information about ‘when’ and ‘where’ these differences appear in the spatio‐temporal domain, simultaneously testing numerous hypotheses, which increase the risk of false positives. This issue is known as the problem of multiple comparisons and has been managed with methods that control the false discovery rate (FDR), such as permutation test and FDR methods. Although the former has been previously applied, to our knowledge, the FDR methods have not been introduced in the ERP data analysis. Here we compare the performance (on simulated and real data) of permutation test and two FDR methods (Benjamini and Hochberg (BH) and local‐fdr, by Efron). All these methods have been shown to be valid for dealing with the problem of multiple comparisons in the ERP analysis, avoiding the ad hoc selection of channels and/or time windows. FDR methods are a good alternative to the common and computationally more expensive permutation test. The BH method for independent tests gave the best overall performance regarding the balance between type I and type II errors. The local‐fdr method is preferable for high dimensional (multichannel) problems where most of the tests conform to the empirical null hypothesis. Differences among the methods according to assumptions, null distributions and dimensionality of the problem are also discussed. Copyright © 2009 John Wiley & Sons, Ltd.     

A graphical approach to sequentially rejective multiple test procedures

For clinical trials with multiple treatment arms or endpoints a variety of sequentially rejective, weighted Bonferroni‐type tests have been proposed, such as gatekeeping procedures, fixed sequence tests, and fallback procedures. They allow to map the difference in importance as well as the relationship between the various research questions onto an adequate multiple test procedure. Since these procedures rely on the closed test principle, they usually require the explicit specification of a large number of intersection hypotheses tests. The underlying test strategy may therefore be difficult to communicate. We propose a simple iterative graphical approach to construct and perform such Bonferroni‐type tests. The resulting multiple test procedures are represented by directed, weighted graphs, where each node corresponds to an elementary hypothesis, together with a simple algorithm to generate such graphs while sequentially testing the individual hypotheses. The approach is illustrated with the visualization of several common gatekeeping strategies. A case study is used to illustrate how the methods from this article can be used to tailor a multiple test procedure to given study objectives. Copyright © 2008 John Wiley & Sons, Ltd.     

Three‐sided hypothesis testing: Simultaneous testing of superiority,equivalence and inferiority

We propose three‐sided testing, a testing framework for simultaneous testing of inferiority, equivalence and superiority in clinical trials, controlling for multiple testing using the partitioning principle. Like the usual two‐sided testing approach, this approach is completely symmetric in the two treatments compared. Still, because the hypotheses of inferiority and superiority are tested with one‐sided tests, the proposed approach has more power than the two‐sided approach to infer non‐inferiority or non‐superiority. Applied to the classical point null hypothesis of equivalence, the three‐sided testing approach shows that it is sometimes possible to make an inference on the sign of the parameter of interest, even when the null hypothesis itself could not be rejected. Relationships with confidence intervals are explored, and the effectiveness of the three‐sided testing approach is demonstrated in a number of recent clinical trials. Copyright © 2010 John Wiley & Sons, Ltd.     

Regional inference with averaged P values increases the power to detect linkage

Controversy exists with respect to the choice of an appropriate critical value when testing for linkage in a genomic screen. A number of critical values have been proposed for single‐locus and multi‐locus linkage analyses. In this study, criteria based on multiple single‐locus analyses (i.e., regional test criteria) are evaluated using simulation methods for three different map densities. Tests based on single loci, multiple consecutive single loci, and moving averages of consecutive single loci are considered. Appropriate critical values are determined based on results from simulations under the null hypothesis of no linkage. The power of each "regional test " was compared to the power of a single‐locus test. Results suggest that the best power was found when averaging P values over an interval size of 9–15 cM, and that testing the average of P values from two consecutive loci is superior to testing each single locus separately. The increase in power ranged from 7– 29% over the simulations considered. Genet. Epidemiol. 17:157–164, 1999. © 1999 Wiley‐Liss, Inc.     

Constructing multiple test procedures for partially ordered hypothesis sets

A popular method to control multiplicity in confirmatory clinical trials is to use a so-called hierarchical, or fixed sequence, test procedure. This requires that the null hypotheses are ordered a priori, for example, in order of clinical importance. The procedure tests the hypotheses in this order using alpha-level tests until one is not rejected. It then stops, so that no subsequent hypotheses are eligible for rejection. This procedure strongly controls the familywise error rate (FWE), that is to say, the probability that any true hypotheses are rejected. This paper describes a simple generalization of this approach in which the null hypotheses are partially ordered. It is convenient to display the partial ordering in a directed acyclic graph (DAG). We consider sequentially rejective procedures based on the partial ordering, in which a hypothesis is tested only when all preceding hypotheses have been tested and rejected. In general such procedures do not control the FWE, but it is shown that when certain intersection hypotheses are added, strong control of the FWE is obtained. The purpose of the method is to construct inference strategies for confirmatory clinical trials that better reflect the trial objectives.     

On estimation of the variance in Cochran-Armitage trend tests for genetic association using case-control studies

The Cochran-Armitage trend test has been used in case-control studies for testing genetic association. As the variance of the test statistic is a function of unknown parameters, e.g. disease prevalence and allele frequency, it must be estimated. The usual estimator combining data for cases and controls assumes they follow the same distribution under the null hypothesis. Under the alternative hypothesis, however, the cases and controls follow different distributions. Thus, the power of the trend tests may be affected by the variance estimator used. In particular, the usual method combining both cases and controls is not an asymptotically unbiased estimator of the null variance when the alternative is true. Two different estimates of the null variance are available which are consistent under both the null and alternative hypotheses. In this paper, we examine sample size and small sample power performance of trend tests, which are optimal for three common genetic models as well as a robust trend test based on the three estimates of the variance and provide guidelines for choosing an appropriate test.