Stratified false discovery control for large-scale hypothesis testing with application to genome-wide association studies

The multiplicity problem has become increasingly important in genetic studies as the capacity for high-throughput genotyping has increased. The control of False Discovery Rate (FDR) (Benjamini and Hochberg. [1995] J. R. Stat. Soc. Ser. B 57:289-300) has been adopted to address the problems of false positive control and low power inherent in high-volume genome-wide linkage and association studies. In many genetic studies, there is often a natural stratification of the m hypotheses to be tested. Given the FDR framework and the presence of such stratification, we investigate the performance of a stratified false discovery control approach (i.e. control or estimate FDR separately for each stratum) and compare it to the aggregated method (i.e. consider all hypotheses in a single stratum). Under the fixed rejection region framework (i.e. reject all hypotheses with unadjusted p-values less than a pre-specified level and then estimate FDR), we demonstrate that the aggregated FDR is a weighted average of the stratum-specific FDRs. Under the fixed FDR framework (i.e. reject as many hypotheses as possible and meanwhile control FDR at a pre-specified level), we specify a condition necessary for the expected total number of true positives under the stratified FDR method to be equal to or greater than that obtained from the aggregated FDR method. Application to a recent Genome-Wide Association (GWA) study by Maraganore et al. ([2005] Am. J. Hum. Genet. 77:685-693) illustrates the potential advantages of control or estimation of FDR by stratum. Our analyses also show that controlling FDR at a low rate, e.g. 5% or 10%, may not be feasible for some GWA studies.     

Graphical approaches for the control of generalized error rates

When simultaneously testing multiple hypotheses, the usual approach in the context of confirmatory clinical trials is to control the familywise error rate (FWER), which bounds the probability of making at least one false rejection. In many trial settings, these hypotheses will additionally have a hierarchical structure that reflects the relative importance and links between different clinical objectives. The graphical approach of Bretz et al (2009) is a flexible and easily communicable way of controlling the FWER while respecting complex trial objectives and multiple structured hypotheses. However, the FWER can be a very stringent criterion that leads to procedures with low power, and may not be appropriate in exploratory trial settings. This motivates controlling generalized error rates, particularly when the number of hypotheses tested is no longer small. We consider the generalized familywise error rate (k-FWER), which is the probability of making k or more false rejections, as well as the tail probability of the false discovery proportion (FDP), which is the probability that the proportion of false rejections is greater than some threshold. We also consider asymptotic control of the false discovery rate, which is the expectation of the FDP. In this article, we show how to control these generalized error rates when using the graphical approach and its extensions. We demonstrate the utility of the resulting graphical procedures on three clinical trial case studies.     

Global test for high-dimensional mediation: Testing groups of potential mediators

We address the problem of testing whether a possibly high-dimensional vector may act as a mediator between some exposure variable and the outcome of interest. We propose a global test for mediation, which combines a global test with the intersection-union principle. We discuss theoretical properties of our approach and conduct simulation studies that demonstrate that it performs equally well or better than its competitor. We also propose a multiple testing procedure, ScreenMin, that provides asymptotic control of either familywise error rate or false discovery rate when multiple groups of potential mediators are tested simultaneously. We apply our approach to data from a large Norwegian cohort study, where we look at the hypothesis that smoking increases the risk of lung cancer by modifying the level of DNA methylation.     

Rank truncated product of P-values, with application to genomewide association scans

Large exploratory studies are often characterized by a preponderance of true null hypotheses, with a small though multiple number of false hypotheses. Traditional multiple-test adjustments consider either each hypothesis separately, or all hypotheses simultaneously, but it may be more desirable to consider the combined evidence for subsets of hypotheses, in order to reduce the number of hypotheses to a manageable size. Previously, Zaykin et al. ([2002] Genet. Epidemiol. 22:170-185) proposed forming the product of all P-values at less than a preset threshold, in order to combine evidence from all significant tests. Here we consider a complementary strategy: form the product of the K most significant P-values. This has certain advantages for genomewide association scans: K can be chosen on the basis of a hypothesised disease model, and is independent of sample size. Furthermore, the alternative hypothesis corresponds more closely to the experimental situation where all loci have fixed effects. We give the distribution of the rank truncated product and suggest some methods to account for correlated tests in genomewide scans. We show that, under realistic scenarios, it provides increased power to detect genomewide association, while identifying a candidate set of good quality and fixed size for follow-up studies.     

Sequential testing for efficacy in clinical trials with non-transient effects

This paper describes a new type of sequential testing for clinical trials. The sequential nature of the data is not from additional patients, but rather from longer follow-up times. At each analysis, the null hypothesis that all treatments are equivalent in effect on the outcome after that amount of time is tested. The trial might still have staggered entry or not, but the key feature is that a different statistical hypothesis is tested at each analysis. It is assumed that any effect of treatment is non-transient, allowing a conclusion to be drawn in favour of one treatment or the other based on a difference at a single follow-up time. It is shown that a general method based on the Bonferroni inequality can be used to obtain critical cutpoints for sequential testing, that controls the chance of a type I error for the clinical decision. This method is applicable regardless of the test used at each analysis. In the case of a two-armed trial with a Gaussian outcome variable, it is shown how simulation can be used to obtain critical cutpoints that maintain the chance of a type I error for the clinical decision. The methods are compared by Monte-Carlo simulation, and it is seen that in most practical cases the Bonferroni method is not very conservative. The Bonferroni procedure is illustrated on the results of a real clinical trial of Pirfenidone on pulmonary fibrosis in Hermansky-Pudlak Syndrome.     

An investigation of two multivariate permutation methods for controlling the false discovery proportion

Identifying genes that are differentially expressed between classes of samples is an important objective of many microarray experiments. Because of the thousands of genes typically considered, there is a tension between identifying as many of the truly differentially expressed genes as possible, but not too many genes that are not really differentially expressed (false discoveries). Controlling the proportion of identified genes that are false discoveries, the false discovery proportion (FDP), is a goal of interest. In this paper, two multivariate permutation methods are investigated for controlling the FDP. One is based on a multivariate permutation testing (MPT) method that probabilistically controls the number of false discoveries, and the other is based on the Significance Analysis of Microarrays (SAM) procedure that provides an estimate of the FDP. Both methods account for the correlations among the genes. We find the ability of the methods to control the proportion of false discoveries varies substantially depending on the implementation characteristics. For example, for both methods one can proceed from the most significant gene to the least significant gene until the estimated FDP is just above the targeted level ('top-down' approach), or from the least significant gene to the most significant gene until the estimated FDP is just below the targeted level ('bottom-up' approach). We find that the top-down MPT-based method probabilistically controls the FDP, whereas our implementation of the top-down SAM-based method does not. Bottom-up MPT-based or SAM-based methods can result in poor control of the FDP.     

Multiple testing for a combination drug with two study endpoints

A combination drug product with two or more active compounds may be superior to each of its components with higher dose levels and, therefore, is preferred in terms of efficacy, cost, and safety. To study a combination drug, researchers often conduct trials by using a factorial design with combinations of dose levels of each drug component. By applying some bootstrap methods, we construct multiple testing procedures to simultaneously identify combinations superior to each drug component with any dose level. These multiple testing procedures are more powerful than Holm's step-down procedure that is known to be very conservative. When there is only one study endpoint, applying the bootstrap is straightforward. In many studies, however, there are two or more study endpoints and it is not simple to apply the bootstrap. We apply one version of the bootstrap and then use an upper bound to control the familywise error defined as the probability of rejecting at least one true null hypothesis. Properties of the bootstrap multiple testing procedures are discussed and examined in some simulation studies.     

An improved closed procedure for testing multiple hypotheses

Clinical trials routinely involve multiple hypothesis testing. The closed testing procedure (CTP) is a fundamental principle in testing multiple hypotheses. This article presents an improved CTP in which intersection hypotheses can be tested at a level greater than $\alpha$ such that the control of the familywise error rate at level $\alpha$ remains. Consequently, our method uniformly improves the power of discovering false hypotheses over the original CTP. We illustrate that an improvement by our method exists for many commonly used tests. An empirical study on the effectiveness of a glucose-lowering drug is provided.     

A Bayesian determination of threshold for identifying differentially expressed genes in microarray experiments

The original definitions of false discovery rate (FDR) and false non-discovery rate (FNR) can be understood as the frequentist risks of false rejections and false non-rejections, respectively, conditional on the unknown parameter, while the Bayesian posterior FDR and posterior FNR are conditioned on the data. From a Bayesian point of view, it seems natural to take into account the uncertainties in both the parameter and the data. In this spirit, we propose averaging out the frequentist risks of false rejections and false non-rejections with respect to some prior distribution of the parameters to obtain the average FDR (AFDR) and average FNR (AFNR), respectively. A linear combination of the AFDR and AFNR, called the average Bayes error rate (ABER), is considered as an overall risk. Some useful formulas for the AFDR, AFNR and ABER are developed for normal samples with hierarchical mixture priors. The idea of finding threshold values by minimizing the ABER or controlling the AFDR is illustrated using a gene expression data set. Simulation studies show that the proposed approaches are more powerful and robust than the widely used FDR method.     

A primer on strong vs weak control of familywise error rate

Multiple comparison adjustments have a long history, yet confusion remains about which procedures control type 1 error rate in a strong sense and how to show this. Part of the confusion stems from a powerful technique called the closed testing principle, whose statement is deceptively simple, but is sometimes misinterpreted. This primer presents a straightforward way to think about multiplicity adjustment.     

The optimal discovery procedure in multiple significance testing: an empirical Bayes approach

Multiple testing has been widely adopted for genome-wide studies such as microarray experiments. To improve the power of multiple testing, Storey (J. Royal Statist. Soc. B 2007; 69: 347-368) recently developed the optimal discovery procedure (ODP) which maximizes the number of expected true positives for each fixed number of expected false positives. However, in applying the ODP, we must estimate the true status of each significance test (null or alternative) and the true probability distribution corresponding to each test. In this article, we derive the ODP under hierarchical, random effects models and develop an empirical Bayes estimation method for the derived ODP. Our methods can effectively circumvent the estimation problems in applying the ODP presented by Storey. Simulations and applications to clinical studies of leukemia and breast cancer demonstrated that our empirical Bayes method achieved theoretical optimality and performed well in comparison with existing multiple testing procedures.     

False Discovery Rates for Rare Variants From Sequenced Data

The detection of rare deleterious variants is the preeminent current technical challenge in statistical genetics. Sorting the deleterious from neutral variants at a disease locus is challenging because of the sparseness of the evidence for each individual variant. Hierarchical modeling and Bayesian model uncertainty are two techniques that have been shown to be promising in pinpointing individual rare variants that may be driving the association. Interpreting the results from these techniques from the perspective of multiple testing is a challenge and the goal of this article is to better understand their false discovery properties. Using simulations, we conclude that accurate false discovery control cannot be achieved in this framework unless the magnitude of the variants' risk is large and the hierarchical characteristics have high accuracy in distinguishing deleterious from neutral variants.     

Sample size determination in step‐up testing procedures for multiple comparisons with a control

Step‐up procedures have been shown to be powerful testing methods in clinical trials for comparisons of several treatments with a control. In this paper, a determination of the optimal sample size for a step‐up procedure that allows a pre‐specified power level to be attained is discussed. Various definitions of power, such as all‐pairs power, any‐pair power, per‐pair power and average power, in one‐ and two‐sided tests are considered. An extensive numerical study confirms that square root allocation of sample size among treatments provides a better approximation of the optimal sample size relative to equal allocation. Based on square root allocation, tables are constructed, and users can conveniently obtain the approximate required sample size for the selected configurations of parameters and power. For clinical studies with difficulties in recruiting patients or when additional subjects lead to a significant increase in cost, a more precise computation of the required sample size is recommended. In such circumstances, our proposed procedure may be adopted to obtain the optimal sample size. It is also found that, contrary to conventional belief, the optimal allocation may considerably reduce the total sample size requirement in certain cases. The determination of the required sample sizes using both allocation rules are illustrated with two examples in clinical studies. Copyright © 2010 John Wiley & Sons, Ltd.     

A new class of powerful and informative simultaneous confidence intervals

Step‐down tests uniformly improve single‐step tests with regard to power and the average number of rejected hypotheses. However, when extended to simultaneous confidence intervals (SCIs), the resulting SCIs often provide no additional information to the sheer hypothesis test. We speak, in this case, of a non‐informative rejection. Non‐informative rejections are particularly problematic in clinical trials with multiple treatments, where an informative rejection is required to obtain useful estimates of the treatment effects. The extension of single‐step tests to confidence intervals does not have this deficiency. As a consequence, step‐down tests, when extended to SCIs, do not uniformly improve single‐step tests with regard to informative rejections. To overcome this deficiency, we suggest the construction of a new class of simultaneous confidence intervals that uniformly improve the Bonferroni and Holm SCIs with regard to informative rejections. This can be achieved using a dual family of weighted Bonferroni tests, with the weights depending continuously on the parameter values. We provide a simple algorithm for these computations and show that the resulting lower confidence bounds have an attractive shrinkage property. The method is extended to union‐intersection tests, such as the Dunnett procedure, and is investigated in a comparative simulation study. We further illustrate the utility of the method with an example from a real clinical trial in which two experimental treatments are compared with an active comparator with respect to non‐inferiority and superiority. Copyright © 2014 John Wiley & Sons, Ltd.     

微阵列数据的多重比较

目的 介绍阳性结果错误率（FDR）及相关控制方法在微阵列数据多重比较中的应用。方法 用BH、BL、BY和ALSU四种FDR控制程序比较了3226个基因在两组乳腺癌患者中的表达差异。结果 四个程序在各自实用的范围内均将FDR控制在0．05以下，检验效能由大到小的顺序为：ALSU〉BH〉BY〉BL。ALSU程序因引入m0的估计,更为合理。不仅提高了检验效能，同时又较好地控制了假阳性错误。结论 在微阵列数据的比较中必须考虑FDR的控制，同时又要考虑提高检验效能。多重比较中，控制FDR比控制总Ⅰ型错误率（FWER）检验效能高，且更为实用。     

Constructing multiple test procedures for partially ordered hypothesis sets

A popular method to control multiplicity in confirmatory clinical trials is to use a so-called hierarchical, or fixed sequence, test procedure. This requires that the null hypotheses are ordered a priori, for example, in order of clinical importance. The procedure tests the hypotheses in this order using alpha-level tests until one is not rejected. It then stops, so that no subsequent hypotheses are eligible for rejection. This procedure strongly controls the familywise error rate (FWE), that is to say, the probability that any true hypotheses are rejected. This paper describes a simple generalization of this approach in which the null hypotheses are partially ordered. It is convenient to display the partial ordering in a directed acyclic graph (DAG). We consider sequentially rejective procedures based on the partial ordering, in which a hypothesis is tested only when all preceding hypotheses have been tested and rejected. In general such procedures do not control the FWE, but it is shown that when certain intersection hypotheses are added, strong control of the FWE is obtained. The purpose of the method is to construct inference strategies for confirmatory clinical trials that better reflect the trial objectives.     

Step‐up testing procedure for multiple comparisons with a control for a latent variable model with ordered categorical responses

In clinical studies, multiple comparisons of several treatments to a control with ordered categorical responses are often encountered. A popular statistical approach to analyzing the data is to use the logistic regression model with the proportional odds assumption. As discussed in several recent research papers, if the proportional odds assumption fails to hold, the undesirable consequence of an inflated familywise type I error rate may affect the validity of the clinical findings. To remedy the problem, a more flexible approach that uses the latent normal model with single‐step and stepwise testing procedures has been recently proposed. In this paper, we introduce a step‐up procedure that uses the correlation structure of test statistics under the latent normal model. A simulation study demonstrates the superiority of the proposed procedure to all existing testing procedures. Based on the proposed step‐up procedure, we derive an algorithm that enables the determination of the total sample size and the sample size allocation scheme with a pre‐determined level of test power before the onset of a clinical trial. A clinical example is presented to illustrate our proposed method. Copyright © 2014 John Wiley & Sons, Ltd.     

Multiple comparisons between two groups on multiple Bernoulli outcomes while accounting for covariates

The problem of adjusting for multiplicity when one has multiple outcome variables can be handled quite nicely by step-down permutation tests. More difficult is the problem when one wants an analysis of each outcome variable to be adjusted for some covariates and the outcome variables are Bernoulli. Special permutations can be used where the outcome vectors are permuted within each strata of the data defined by the levels of the (made discrete) covariates. This method is described and shown to control the familywise error rate at any prespecified level. The method is compared through simulation to a vector bootstrap approach, also using a step-down testing procedure. It is seen that the method using permutations within strata is superior to the vector bootstrap in terms of error control and power. The method is illustrated on a data set of 55 minor malformations of babies of diabetic and non-diabetic mothers.     

Significance testing for small microarray experiments

Which significance test is carried out when the number of repeats is small in microarray experiments can dramatically influence the results. When in two sample comparisons both conditions have fewer than, say, five repeats traditional test statistics require extreme results, before a gene is considered statistically significant differentially expressed after a multiple comparisons correction. In the literature many approaches to circumvent this problem have been proposed. Some of these proposals use (empirical) Bayes arguments to moderate the variance estimates for individual genes. Other proposals try to stabilize these variance estimate by combining groups of genes or similar experiments. In this paper we compare several of these approaches, both on data sets where both experimental conditions are the same, and thus few statistically significant differentially expressed genes should be identified, and on experiments where both conditions do differ. This allows us to identify which approaches are most powerful without identifying many false positives. We conclude that after balancing the numbers of false positives and true positives an empirical Bayes approach and an approach which combines experiments perform best. Standard t-tests are inferior and offer almost no power when the sample size is small.