Multiple hypothesis testing in genomics

This paper presents an overview of the current state of the art in multiple testing in genomics data from a user's perspective. We describe methods for familywise error control, false discovery rate control and false discovery proportion estimation and confidence, both conceptually and practically, and explain when to use which type of error rate. We elaborate on the assumptions underlying the methods and discuss pitfalls in the interpretation of results. In our discussion, we take into account the exploratory nature of genomics experiments, looking at selection of genes before or after testing, and at the role of validation experiments. Copyright © 2014 John Wiley & Sons, Ltd.     

Global test for high-dimensional mediation: Testing groups of potential mediators

We address the problem of testing whether a possibly high-dimensional vector may act as a mediator between some exposure variable and the outcome of interest. We propose a global test for mediation, which combines a global test with the intersection-union principle. We discuss theoretical properties of our approach and conduct simulation studies that demonstrate that it performs equally well or better than its competitor. We also propose a multiple testing procedure, ScreenMin, that provides asymptotic control of either familywise error rate or false discovery rate when multiple groups of potential mediators are tested simultaneously. We apply our approach to data from a large Norwegian cohort study, where we look at the hypothesis that smoking increases the risk of lung cancer by modifying the level of DNA methylation.     

Many Phenotypes Without Many False Discoveries: Error Controlling Strategies for Multitrait Association Studies

The genetic basis of multiple phenotypes such as gene expression, metabolite levels, or imaging features is often investigated by testing a large collection of hypotheses, probing the existence of association between each of the traits and hundreds of thousands of genotyped variants. Appropriate multiplicity adjustment is crucial to guarantee replicability of findings, and the false discovery rate (FDR) is frequently adopted as a measure of global error. In the interest of interpretability, results are often summarized so that reporting focuses on variants discovered to be associated to some phenotypes. We show that applying FDR‐controlling procedures on the entire collection of hypotheses fails to control the rate of false discovery of associated variants as well as the expected value of the average proportion of false discovery of phenotypes influenced by such variants. We propose a simple hierarchical testing procedure that allows control of both these error rates and provides a more reliable basis for the identification of variants with functional effects. We demonstrate the utility of this approach through simulation studies comparing various error rates and measures of power for genetic association studies of multiple traits. Finally, we apply the proposed method to identify genetic variants that impact flowering phenotypes in Arabidopsis thaliana, expanding the set of discoveries.     

An improved closed procedure for testing multiple hypotheses

Clinical trials routinely involve multiple hypothesis testing. The closed testing procedure (CTP) is a fundamental principle in testing multiple hypotheses. This article presents an improved CTP in which intersection hypotheses can be tested at a level greater than $\alpha$ such that the control of the familywise error rate at level $\alpha$ remains. Consequently, our method uniformly improves the power of discovering false hypotheses over the original CTP. We illustrate that an improvement by our method exists for many commonly used tests. An empirical study on the effectiveness of a glucose-lowering drug is provided.     

On generalized fixed sequence procedures for controlling the FWER

Testing a sequence of pre‐ordered hypotheses to decide which of these can be rejected or accepted while controlling the familywise error rate (FWER) is of importance in many scientific studies such as clinical trials. In this paper, we first introduce a generalized fixed sequence procedure whose critical values are defined by using a function of the numbers of rejections and acceptances, and which allows follow‐up hypotheses to be tested even if some earlier hypotheses are not rejected. We then construct the least favorable configuration for this generalized fixed sequence procedure and present a sufficient condition for the FWER control under arbitrary dependence. Based on the condition, we develop three new generalized fixed sequence procedures controlling the FWER under arbitrary dependence. We also prove that each generalized fixed sequence procedure can be described as a specific closed testing procedure. Through simulation studies and a clinical trial example, we compare the power performance of these proposed procedures with those of the existing FWER controlling procedures. Finally, when the pairwise joint distributions of the true null p‐values are known, we further improve these procedures by incorporating pairwise correlation information while maintaining the control of the FWER. Copyright © 2015 John Wiley & Sons, Ltd.     

A primer on strong vs weak control of familywise error rate

Multiple comparison adjustments have a long history, yet confusion remains about which procedures control type 1 error rate in a strong sense and how to show this. Part of the confusion stems from a powerful technique called the closed testing principle, whose statement is deceptively simple, but is sometimes misinterpreted. This primer presents a straightforward way to think about multiplicity adjustment.     

Automatic detection of significant areas for functional data with directional error control

In this paper, we propose a large-scale multiple testing procedure to find the significant sub-areas between two samples of curves automatically. The procedure is optimal in that it controls the directional false discovery rate at any specified level on a continuum asymptotically. By introducing a nonparametric Gaussian process regression model for the two-sided multiple test, the procedure is computationally inexpensive. It can cope with problems with multidimensional covariates and accommodate different sampling designs across the samples. We further propose the significant curve/surface, giving an insight on dynamic significant differences between two curves. Simulation studies demonstrate that the proposed procedure enjoys superior performance with strong power and good directional error control. The procedure is also illustrated with the application to two executive function studies in hemiplegia.     

Nonparametric estimation of the rediscovery rate

Validation studies have been used to increase the reliability of the statistical conclusions for scientific discoveries; such studies improve the reproducibility of the findings and reduce the possibility of false positives. Here, one of the important roles of statistics is to quantify reproducibility rigorously. Two concepts were recently defined for this purpose: (i) rediscovery rate (RDR), which is the expected proportion of statistically significant findings in a study that can be replicated in the validation study and (ii) false discovery rate in the validation study (vFDR). In this paper, we aim to develop a nonparametric approach to estimate the RDR and vFDR and show an explicit link between the RDR and the FDR. Among other things, the link explains why reproducing statistically significant results even with low FDR level may be difficult. Two metabolomics datasets are considered to illustrate the application of the RDR and vFDR concepts in high‐throughput data analysis. Copyright © 2016 John Wiley & Sons, Ltd.     

Extension of adaptive alpha allocation methods for strong control of the family‐wise error rate

With recent advancements in clinical trial design and the availability of rigorous statistical methods that provide strong control of the family‐wise type I error rate for multiple testing of hypotheses, it is now common for sponsors to design clinical trials with prospectively specified multiple testing of hypotheses of both primary and secondary endpoints and with the intent to obtain labeling claims for secondary endpoints. One of these recent advancements in multiple testing techniques is the adaptive alpha allocation approach (4A) proposed by Li and Mehrotra (Statistics in Medicine 2008; 27:5377–5391), which groups the hypotheses into two families on the basis of perceived trial power and allows the significance level for the second family to be set adaptively on the basis of the largest observed p‐value in the first family. We introduce a class of flexible functions that generalize the 4A procedure and can lead to relatively more powerful test statistics. In the case when the test statistics are correlated, we introduce well‐defined functions to calculate the significance level for the second family. The numerical computation for our methods is straightforward, making application in practice easy. Copyright © 2012 John Wiley & Sons, Ltd.     

Type‐II generalized family‐wise error rate formulas with application to sample size determination

Multiple endpoints are increasingly used in clinical trials. The significance of some of these clinical trials is established if at least r null hypotheses are rejected among m that are simultaneously tested. The usual approach in multiple hypothesis testing is to control the family‐wise error rate, which is defined as the probability that at least one type‐I error is made. More recently, the q‐generalized family‐wise error rate has been introduced to control the probability of making at least q false rejections. For procedures controlling this global type‐I error rate, we define a type‐II r‐generalized family‐wise error rate, which is directly related to the r‐power defined as the probability of rejecting at least r false null hypotheses. We obtain very general power formulas that can be used to compute the sample size for single‐step and step‐wise procedures. These are implemented in our R package rPowerSampleSize available on the CRAN, making them directly available to end users. Complexities of the formulas are presented to gain insight into computation time issues. Comparison with Monte Carlo strategy is also presented. We compute sample sizes for two clinical trials involving multiple endpoints: one designed to investigate the effectiveness of a drug against acute heart failure and the other for the immunogenicity of a vaccine strategy against pneumococcus. Copyright © 2016 John Wiley & Sons, Ltd.     

Impaired performance of FDR-based strategies in whole-genome association studies when SNPs are excluded prior to the analysis

With recent advances in genomewide microarray technologies, whole-genome association (WGA) studies have aimed at identifying susceptibility genes for complex human diseases using hundreds of thousands of single nucleotide polymorphisms (SNPs) genotyped at the same time. In this context and to take into account multiple testing, false discovery rate (FDR)-based strategies are now used frequently. However, a critical aspect of these strAtegies is that they are applied to a collection or a family of hypotheses and, thus, critically depend on these precise hypotheses. We investigated how modifying the family of hypotheses to be tested affected the performance of FDR-based procedures in WGA studies. We showed that FDR-based procedures performed more poorly when excluding SNPs with high prior probability of being associated. Results of simulation studies mimicking WGA studies according to three scenarios are reported, and show the extent to which SNPs elimination (family contraction) prior to the analysis impairs the performance of FDR-based procedures. To illustrate this situation, we used the data from a recent WGA study on type-1 diabetes (Clayton et al. [2005] Nat. Genet. 37:1243-1246) and report the results obtained when excluding or not SNPs located inside the human leukocyte antigen region. Based on our findings, excluding markers with high prior probability of being associated cannot be recommended for the analysis of WGA data with FDR-based strategies.     

Designing multi‐arm multi‐stage clinical trials using a risk–benefit criterion for treatment selection

Multi‐arm clinical trials that compare several active treatments to a common control have been proposed as an efficient means of making an informed decision about which of several treatments should be evaluated further in a confirmatory study. Additional efficiency is gained by incorporating interim analyses and, in particular, seamless Phase II/III designs have been the focus of recent research. Common to much of this work is the constraint that selection and formal testing should be based on a single efficacy endpoint, despite the fact that in practice, safety considerations will often play a central role in determining selection decisions. Here, we develop a multi‐arm multi‐stage design for a trial with an efficacy and safety endpoint. The safety endpoint is explicitly considered in the formulation of the problem, selection of experimental arm and hypothesis testing. The design extends group‐sequential ideas and considers the scenario where a minimal safety requirement is to be fulfilled and the treatment yielding the best combined safety and efficacy trade‐off satisfying this constraint is selected for further testing. The treatment with the best trade‐off is selected at the first interim analysis, while the whole trial is allowed to compose of J analyses. We show that the design controls the familywise error rate in the strong sense and illustrate the method through an example and simulation. We find that the design is robust to misspecification of the correlation between the endpoints and requires similar numbers of subjects to a trial based on efficacy alone for moderately correlated endpoints. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

A group sequential Holm procedure with multiple primary endpoints

We propose a group sequential Holm procedure when there are multiple primary endpoints. This method addresses multiplicities arising from multiple primary endpoints and from multiple analyses in a group sequential design. It has been shown to be a closed testing procedure and preserves the familywise error rate in the strong sense. When multiple endpoints are the only concern without an interim analysis, the method simplifies to the weighted Holm procedure. The proposed method is more powerful than the parallel group sequential method and avoids the need to anticipate the testing order as in the fixed sequence testing scheme. Copyright © 2012 John Wiley & Sons, Ltd.     

Adaptive extensions of a two-stage group sequential procedure for testing primary and secondary endpoints (II): sample size re-estimation

In this part II of the paper on adaptive extensions of a two‐stage group sequential procedure (GSP) for testing primary and secondary endpoints, we focus on the second stage sample size re‐estimation based on the first stage data. First, we show that if we use the Cui–Huang–Wang statistics at the second stage, then we can use the same primary and secondary boundaries as for the original procedure (without sample size re‐estimation) and still control the type I familywise error rate. This extends their result for the single endpoint case. We further show that the secondary boundary can be sharpened in this case by taking the unknown correlation coefficient ρ between the primary and secondary endpoints into account through the use of the confidence limit method proposed in part I of this paper. If we use the sufficient statistics instead of the CHW statistics, then we need to modify both the primary and secondary boundaries; otherwise, the error rate can get inflated. We show how to modify the boundaries of the original group sequential procedure to control the familywise error rate. We provide power comparisons between competing procedures. We illustrate the procedures with a clinical trial example. Copyright © 2012 John Wiley & Sons, Ltd.     

Symmetric graphs for equally weighted tests,with application to the Hochberg procedure

The graphical approach to multiple testing provides a convenient tool for designing, visualizing, and performing multiplicity adjustments in confirmatory clinical trials while controlling the familywise error rate. It assigns a set of weights to each intersection null hypothesis within the closed test framework. These weights form the basis for intersection tests using weighted individual p-values, such as the weighted Bonferroni test. In this paper, we extend the graphical approach to intersection tests that assume equal weights for the elementary null hypotheses associated with any intersection hypothesis, including the Hochberg procedure as well as omnibus tests such as Fisher's combination, O'Brien's, and F tests. More specifically, we introduce symmetric graphs that generate sets of equal weights so that the aforementioned tests can be applied with the graphical approach. In addition, we visualize the Hochberg and the truncated Hochberg procedures in serial and parallel gatekeeping settings using symmetric component graphs. We illustrate the method with two clinical trial examples.     

The optimal discovery procedure in multiple significance testing: an empirical Bayes approach

Multiple testing has been widely adopted for genome-wide studies such as microarray experiments. To improve the power of multiple testing, Storey (J. Royal Statist. Soc. B 2007; 69: 347-368) recently developed the optimal discovery procedure (ODP) which maximizes the number of expected true positives for each fixed number of expected false positives. However, in applying the ODP, we must estimate the true status of each significance test (null or alternative) and the true probability distribution corresponding to each test. In this article, we derive the ODP under hierarchical, random effects models and develop an empirical Bayes estimation method for the derived ODP. Our methods can effectively circumvent the estimation problems in applying the ODP presented by Storey. Simulations and applications to clinical studies of leukemia and breast cancer demonstrated that our empirical Bayes method achieved theoretical optimality and performed well in comparison with existing multiple testing procedures.     

Power and sample size for DNA microarray studies

A microarray study aims at having a high probability of declaring genes to be differentially expressed if they are truly expressed, while keeping the probability of making false declarations of expression acceptably low. Thus, in formal terms, well-designed microarray studies will have high power while controlling type I error risk. Achieving this objective is the purpose of this paper. Here, we discuss conceptual issues and present computational methods for statistical power and sample size in microarray studies, taking account of the multiple testing that is generic to these studies. The discussion encompasses choices of experimental design and replication for a study. Practical examples are used to demonstrate the methods. The examples show forcefully that replication of a microarray experiment can yield large increases in statistical power. The paper refers to cDNA arrays in the discussion and illustrations but the proposed methodology is equally applicable to expression data from oligonucleotide arrays.     

Stratified false discovery control for large-scale hypothesis testing with application to genome-wide association studies

The multiplicity problem has become increasingly important in genetic studies as the capacity for high-throughput genotyping has increased. The control of False Discovery Rate (FDR) (Benjamini and Hochberg. [1995] J. R. Stat. Soc. Ser. B 57:289-300) has been adopted to address the problems of false positive control and low power inherent in high-volume genome-wide linkage and association studies. In many genetic studies, there is often a natural stratification of the m hypotheses to be tested. Given the FDR framework and the presence of such stratification, we investigate the performance of a stratified false discovery control approach (i.e. control or estimate FDR separately for each stratum) and compare it to the aggregated method (i.e. consider all hypotheses in a single stratum). Under the fixed rejection region framework (i.e. reject all hypotheses with unadjusted p-values less than a pre-specified level and then estimate FDR), we demonstrate that the aggregated FDR is a weighted average of the stratum-specific FDRs. Under the fixed FDR framework (i.e. reject as many hypotheses as possible and meanwhile control FDR at a pre-specified level), we specify a condition necessary for the expected total number of true positives under the stratified FDR method to be equal to or greater than that obtained from the aggregated FDR method. Application to a recent Genome-Wide Association (GWA) study by Maraganore et al. ([2005] Am. J. Hum. Genet. 77:685-693) illustrates the potential advantages of control or estimation of FDR by stratum. Our analyses also show that controlling FDR at a low rate, e.g. 5% or 10%, may not be feasible for some GWA studies.     

Sample size determination in step‐up testing procedures for multiple comparisons with a control

Step‐up procedures have been shown to be powerful testing methods in clinical trials for comparisons of several treatments with a control. In this paper, a determination of the optimal sample size for a step‐up procedure that allows a pre‐specified power level to be attained is discussed. Various definitions of power, such as all‐pairs power, any‐pair power, per‐pair power and average power, in one‐ and two‐sided tests are considered. An extensive numerical study confirms that square root allocation of sample size among treatments provides a better approximation of the optimal sample size relative to equal allocation. Based on square root allocation, tables are constructed, and users can conveniently obtain the approximate required sample size for the selected configurations of parameters and power. For clinical studies with difficulties in recruiting patients or when additional subjects lead to a significant increase in cost, a more precise computation of the required sample size is recommended. In such circumstances, our proposed procedure may be adopted to obtain the optimal sample size. It is also found that, contrary to conventional belief, the optimal allocation may considerably reduce the total sample size requirement in certain cases. The determination of the required sample sizes using both allocation rules are illustrated with two examples in clinical studies. Copyright © 2010 John Wiley & Sons, Ltd.