The use of permutation tests for the analysis of parallel and stepped‐wedge cluster‐randomized trials

We investigate the use of permutation tests for the analysis of parallel and stepped‐wedge cluster‐randomized trials. Permutation tests for parallel designs with exponential family endpoints have been extensively studied. The optimal permutation tests developed for exponential family alternatives require information on intraclass correlation, a quantity not yet defined for time‐to‐event endpoints. Therefore, it is unclear how efficient permutation tests can be constructed for cluster‐randomized trials with such endpoints. We consider a class of test statistics formed by a weighted average of pair‐specific treatment effect estimates and offer practical guidance on the choice of weights to improve efficiency. We apply the permutation tests to a cluster‐randomized trial evaluating the effect of an intervention to reduce the incidence of hospital‐acquired infection. In some settings, outcomes from different clusters may be correlated, and we evaluate the validity and efficiency of permutation test in such settings. Lastly, we propose a permutation test for stepped‐wedge designs and compare its performance with mixed‐effect modeling and illustrate its superiority when sample sizes are small, the underlying distribution is skewed, or there is correlation across clusters. Copyright © 2017 John Wiley & Sons, Ltd.     

Statistical efficiency and optimal design for stepped cluster studies under linear mixed effects models

In stepped cluster designs the intervention is introduced into some (or all) clusters at different times and persists until the end of the study. Instances include traditional parallel cluster designs and the more recent stepped‐wedge designs. We consider the precision offered by such designs under mixed‐effects models with fixed time and random subject and cluster effects (including interactions with time), and explore the optimal choice of uptake times. The results apply both to cross‐sectional studies where new subjects are observed at each time‐point, and longitudinal studies with repeat observations on the same subjects. The efficiency of the design is expressed in terms of a ‘cluster‐mean correlation’ which carries information about the dependency‐structure of the data, and two design coefficients which reflect the pattern of uptake‐times. In cross‐sectional studies the cluster‐mean correlation combines information about the cluster‐size and the intra‐cluster correlation coefficient. A formula is given for the ‘design effect’ in both cross‐sectional and longitudinal studies. An algorithm for optimising the choice of uptake times is described and specific results obtained for the best balanced stepped designs. In large studies we show that the best design is a hybrid mixture of parallel and stepped‐wedge components, with the proportion of stepped wedge clusters equal to the cluster‐mean correlation. The impact of prior uncertainty in the cluster‐mean correlation is considered by simulation. Some specific hybrid designs are proposed for consideration when the cluster‐mean correlation cannot be reliably estimated, using a minimax principle to ensure acceptable performance across the whole range of unknown values. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.     

Bias and inference from misspecified mixed‐effect models in stepped wedge trial analysis

Many stepped wedge trials (SWTs) are analysed by using a mixed‐effect model with a random intercept and fixed effects for the intervention and time periods (referred to here as the standard model). However, it is not known whether this model is robust to misspecification. We simulated SWTs with three groups of clusters and two time periods; one group received the intervention during the first period and two groups in the second period. We simulated period and intervention effects that were either common‐to‐all or varied‐between clusters. Data were analysed with the standard model or with additional random effects for period effect or intervention effect. In a second simulation study, we explored the weight given to within‐cluster comparisons by simulating a larger intervention effect in the group of the trial that experienced both the control and intervention conditions and applying the three analysis models described previously. Across 500 simulations, we computed bias and confidence interval coverage of the estimated intervention effect. We found up to 50% bias in intervention effect estimates when period or intervention effects varied between clusters and were treated as fixed effects in the analysis. All misspecified models showed undercoverage of 95% confidence intervals, particularly the standard model. A large weight was given to within‐cluster comparisons in the standard model. In the SWTs simulated here, mixed‐effect models were highly sensitive to departures from the model assumptions, which can be explained by the high dependence on within‐cluster comparisons. Trialists should consider including a random effect for time period in their SWT analysis model. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.     

Design and analysis considerations for cohort stepped wedge cluster randomized trials with a decay correlation structure

A stepped wedge cluster randomized trial is a type of longitudinal cluster design that sequentially switches clusters to intervention over time until all clusters are treated. While the traditional posttest-only parallel design requires adjustment for a single intraclass correlation coefficient, the stepped wedge design allows multiple outcome measurements from the same cluster and so additional correlation parameters are necessary to characterize the within-cluster correlation structure. Although a number of studies have differentiated between the concepts of within-period and between-period correlations, only a few studies have allowed the between-period correlation to decay over time. In this article, we consider the proportional decay correlation structure for a cohort stepped wedge design, and provide a matrix-adjusted quasi-least squares approach to accurately estimate the correlation parameters along with the marginal intervention effect. We further develop the sample size and power procedures accounting for the correlation decay, and investigate the accuracy of the power procedure with continuous outcomes in a simulation study. We show that the empirical power agrees well with the prediction even with as few as nine clusters, when data are analyzed with matrix-adjusted quasi-least squares concurrently with a suitable bias-corrected sandwich variance. Two trial examples are provided to illustrate the new sample size procedure.     

Improving the efficiency of individually randomized clinical trials by staggering the introduction of the intervention

In cluster randomized trials, the introduction of the intervention can be staggered in different clusters, leading to a stepped wedge design. This strategy can lead to gains in efficiency, which might also translate to the context of individually randomized trials, though this has been relatively unexplored. Here, we present one illustrative example. We consider trials in which participants start in a control condition such as routine care and can cross over at any stage to the active intervention. We make the assumption that the effect of the intervention is the same however long the delay before a participant crosses over to the intervention condition. We consider designs for a trial with three repeated assessments, including a baseline, and show that a three-sequence design with staggered introduction of the intervention in two of the sequences estimates the treatment effect after one period more efficiently than a parallel groups design.     

Sample size calculation for stepped wedge and other longitudinal cluster randomised trials

The sample size required for a cluster randomised trial is inflated compared with an individually randomised trial because outcomes of participants from the same cluster are correlated. Sample size calculations for longitudinal cluster randomised trials (including stepped wedge trials) need to take account of at least two levels of clustering: the clusters themselves and times within clusters. We derive formulae for sample size for repeated cross‐section and closed cohort cluster randomised trials with normally distributed outcome measures, under a multilevel model allowing for variation between clusters and between times within clusters. Our formulae agree with those previously described for special cases such as crossover and analysis of covariance designs, although simulation suggests that the formulae could underestimate required sample size when the number of clusters is small. Whether using a formula or simulation, a sample size calculation requires estimates of nuisance parameters, which in our model include the intracluster correlation, cluster autocorrelation, and individual autocorrelation. A cluster autocorrelation less than 1 reflects a situation where individuals sampled from the same cluster at different times have less correlated outcomes than individuals sampled from the same cluster at the same time. Nuisance parameters could be estimated from time series obtained in similarly clustered settings with the same outcome measure, using analysis of variance to estimate variance components. Copyright © 2016 John Wiley & Sons, Ltd.     

Stepped wedge designs could reduce the required sample size in cluster randomized trials

ObjectiveThe stepped wedge design is increasingly being used in cluster randomized trials (CRTs). However, there is not much information available about the design and analysis strategies for these kinds of trials. Approaches to sample size and power calculations have been provided, but a simple sample size formula is lacking. Therefore, our aim is to provide a sample size formula for cluster randomized stepped wedge designs.Study Design and SettingWe derived a design effect (sample size correction factor) that can be used to estimate the required sample size for stepped wedge designs. Furthermore, we compared the required sample size for the stepped wedge design with a parallel group and analysis of covariance (ANCOVA) design.ResultsOur formula corrects for clustering as well as for the design. Apart from the cluster size and intracluster correlation, the design effect depends on choices of the number of steps, the number of baseline measurements, and the number of measurements between steps. The stepped wedge design requires a substantial smaller sample size than a parallel group and ANCOVA design.ConclusionFor CRTs, the stepped wedge design is far more efficient than the parallel group and ANCOVA design in terms of sample size.     

Sample size calculations for stepped wedge and cluster randomised trials: a unified approach

ObjectivesTo clarify and illustrate sample size calculations for the cross-sectional stepped wedge cluster randomized trial (SW-CRT) and to present a simple approach for comparing the efficiencies of competing designs within a unified framework.Study Design and SettingWe summarize design effects for the SW-CRT, the parallel cluster randomized trial (CRT), and the parallel cluster randomized trial with before and after observations (CRT-BA), assuming cross-sectional samples are selected over time. We present new formulas that enable trialists to determine the required cluster size for a given number of clusters. We illustrate by example how to implement the presented design effects and give practical guidance on the design of stepped wedge studies.ResultsFor a fixed total cluster size, the choice of study design that provides the greatest power depends on the intracluster correlation coefficient (ICC) and the cluster size. When the ICC is small, the CRT tends to be more efficient; when the ICC is large, the SW-CRT tends to be more efficient and can serve as an alternative design when the CRT is an infeasible design.ConclusionOur unified approach allows trialists to easily compare the efficiencies of three competing designs to inform the decision about the most efficient design in a given scenario.     

Robust analysis of stepped wedge trials using cluster‐level summaries within periods

In stepped‐wedge trials (SWTs), the intervention is rolled out in a random order over more than 1 time‐period. SWTs are often analysed using mixed‐effects models that require strong assumptions and may be inappropriate when the number of clusters is small. We propose a non‐parametric within‐period method to analyse SWTs. This method estimates the intervention effect by comparing intervention and control conditions in a given period using cluster‐level data corresponding to exposure. The within‐period intervention effects are combined with an inverse‐variance‐weighted average, and permutation tests are used. We present an example and, using simulated data, compared the method to (1) a parametric cluster‐level within‐period method, (2) the most commonly used mixed‐effects model, and (3) a more flexible mixed‐effects model. We simulated scenarios where period effects were common to all clusters, and when they varied according to a distribution informed by routinely collected health data. The non‐parametric within‐period method provided unbiased intervention effect estimates with correct confidence‐interval coverage for all scenarios. The parametric within‐period method produced confidence intervals with low coverage for most scenarios. The mixed‐effects models' confidence intervals had low coverage when period effects varied between clusters but had greater power than the non‐parametric within‐period method when period effects were common to all clusters. The non‐parametric within‐period method is a robust method for analysing SWT. The method could be used by trial statisticians who want to emphasise that the SWT is a randomised trial, in the common position of being uncertain about whether data will meet the assumptions necessary for mixed‐effect models.     

Maintaining the validity of inference in small-sample stepped wedge cluster randomized trials with binary outcomes when using generalized estimating equations

Stepped wedge cluster trials are an increasingly popular alternative to traditional parallel cluster randomized trials. Such trials often utilize a small number of clusters and numerous time intervals, and these components must be considered when choosing an analysis method. A generalized linear mixed model containing a random intercept and fixed time and intervention covariates is the most common analysis approach. However, the sole use of a random intercept applies a constant intraclass correlation coefficient structure, which is an assumption that is likely to be violated given stepped wedge trials (SWTs) have multiple time intervals. Alternatively, generalized estimating equations (GEE) are robust to the misspecification of the working correlation structure, although it has been shown that small-sample adjustments to standard error estimates and the use of appropriate degrees of freedom are required to maintain the validity of inference when the number of clusters is small. In this article, we show, using an extensive simulation study based on a motivating example and a more general design, the use of GEE can maintain the validity of inference in small-sample SWTs with binary outcomes. Furthermore, we show which combinations of bias corrections to standard error estimates and degrees of freedom work best in terms of attaining nominal type I error rates.     

Stepped‐wedge cluster randomised controlled trials: a generic framework including parallel and multiple‐level designs

Stepped‐wedge cluster randomised trials (SW‐CRTs) are being used with increasing frequency in health service evaluation. Conventionally, these studies are cross‐sectional in design with equally spaced steps, with an equal number of clusters randomised at each step and data collected at each and every step. Here we introduce several variations on this design and consider implications for power. One modification we consider is the incomplete cross‐sectional SW‐CRT, where the number of clusters varies at each step or where at some steps, for example, implementation or transition periods, data are not collected. We show that the parallel CRT with staggered but balanced randomisation can be considered a special case of the incomplete SW‐CRT. As too can the parallel CRT with baseline measures. And we extend these designs to allow for multiple layers of clustering, for example, wards within a hospital. Building on results for complete designs, power and detectable difference are derived using a Wald test and obtaining the variance–covariance matrix of the treatment effect assuming a generalised linear mixed model. These variations are illustrated by several real examples. We recommend that whilst the impact of transition periods on power is likely to be small, where they are a feature of the design they should be incorporated. We also show examples in which the power of a SW‐CRT increases as the intra‐cluster correlation (ICC) increases and demonstrate that the impact of the ICC is likely to be smaller in a SW‐CRT compared with a parallel CRT, especially where there are multiple levels of clustering. Finally, through this unified framework, the efficiency of the SW‐CRT and the parallel CRT can be compared. © 2014 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

Meta‐analysis of absolute mean differences from randomised trials with treatment‐related clustering associated with care providers

Nesting of patients within care providers in trials of physical and talking therapies creates an additional level within the design. The statistical implications of this are analogous to those of cluster randomised trials, except that the clustering effect may interact with treatment and can be restricted to one or more of the arms. The statistical model that is recommended at the trial level includes a random effect for the care provider but allows the provider and patient level variances to differ across arms. Evidence suggests that, while potentially important, such within‐trial clustering effects have rarely been taken into account in trials and do not appear to have been considered in meta‐analyses of these trials. This paper describes summary measures and individual‐patient‐data methods for meta‐analysing absolute mean differences from randomised trials with two‐level nested clustering effects, contrasting fixed and random effects meta‐analysis models. It extends methods for incorporating trials with unequal variances and homogeneous clustering to allow for between‐arm and between‐trial heterogeneity in intra‐class correlation coefficient estimates. The work is motivated by a meta‐analysis of trials of counselling in primary care, where the control is no counselling and the outcome is the Beck Depression Inventory. Assuming equal counsellor intra‐class correlation coefficients across trials, the recommended random‐effects heteroscedastic model gave a pooled absolute mean difference of ?2.53 (95% CI ?5.33 to 0.27) using summary measures and ?2.51 (95% CI ?5.35 to 0.33) with the individual‐patient‐data. Pooled estimates were consistently below a minimally important clinical difference of four to five points on the Beck Depression Inventory. Copyright © 2014 John Wiley & Sons, Ltd.     

Balance in cluster randomized trials

This paper explores the role of balancing covariates between treatment groups in the design of cluster randomized trials. General expressions are obtained for two criteria to evaluate designs for parallel group studies with two treatments. The first is the variance of the estimated treatment effect and the second is the extent to which the estimated treatment effect is changed by adjusting for covariates. It is argued that the second of these is more important for cluster randomized trials. Methods of obtaining balanced designs from covariates which are available at the start of a study are proposed. An imbalance measure is used to compare the extent to which designs balance important covariates between the arms of a trial. Several approaches to selecting a well balanced design are possible. A method that randomly selects one member from the class of designs with acceptable bias will allow randomization inference as well as model-based inference. The methods are illustrated with data from a trial of school-based sex education.     

Accounting for a decaying correlation structure in cluster randomized trials with continuous recruitment

A requirement for calculating sample sizes for cluster randomized trials (CRTs) conducted over multiple periods of time is the specification of a form for the correlation between outcomes of subjects within the same cluster, encoded via the within-cluster correlation structure. Previously proposed within-cluster correlation structures have made strong assumptions; for example, the usual assumption is that correlations between the outcomes of all pairs of subjects are identical (“uniform correlation”). More recently, structures that allow for a decay in correlation between pairs of outcomes measured in different periods have been suggested. However, these structures are overly simple in settings with continuous recruitment and measurement. We propose a more realistic “continuous-time correlation decay” structure whereby correlations between subjects' outcomes decay as the time between these subjects' measurement times increases. We investigate the use of this structure on trial planning in the context of a primary care diabetes trial, where there is evidence of decaying correlation between pairs of patients' outcomes over time. In particular, for a range of different trial designs, we derive the variance of the treatment effect estimator under continuous-time correlation decay and compare this to the variance obtained under uniform correlation. For stepped wedge and cluster randomized crossover designs, incorrectly assuming uniform correlation will underestimate the required sample size under most trial configurations likely to occur in practice. Planning of CRTs requires consideration of the most appropriate within-cluster correlation structure to obtain a suitable sample size.     

Baseline adjustments for binary data in repeated cross-sectional cluster randomized trials

Analysis of covariance models, which adjust for a baseline covariate, are often used to compare treatment groups in a controlled trial in which individuals are randomized. Such analysis adjusts for any baseline imbalance and usually increases the precision of the treatment effect estimate. We assess the value of such adjustments in the context of a cluster randomized trial with repeated cross-sectional design and a binary outcome. In such a design, a new sample of individuals is taken from the clusters at each measurement occasion, so that baseline adjustment has to be at the cluster level. Logistic regression models are used to analyse the data, with cluster level random effects to allow for different outcome probabilities in each cluster. We compare the estimated treatment effect and its precision in models that incorporate a covariate measuring the cluster level probabilities at baseline and those that do not. In two data sets, taken from a cluster randomized trial in the treatment of menorrhagia, the value of baseline adjustment is only evident when the number of subjects per cluster is large. We assess the generalizability of these findings by undertaking a simulation study, and find that increased precision of the treatment effect requires both large cluster sizes and substantial heterogeneity between clusters at baseline, but baseline imbalance arising by chance in a randomized study can always be effectively adjusted for.     

The analysis of terminal endpoint events in stepped wedge designs

The stepped wedge design is a unique clinical trial design that allows for a sequential introduction of an intervention. However, the statistical analysis is unclear when this design is applied in survival data. The time‐dependent introduction of the intervention in combination with terminal endpoints and interval censoring makes the analysis more complicated. In this paper, a time‐on‐study scale discrete survival model was constructed. Simulations were conducted primarily to study the performance of our model for different settings of the stepped wedge design. Secondary, we compared our approach to continuous Cox proportional hazard model. The results show that the discrete survival model estimates the intervention effects unbiasedly. If the length of the censoring interval is increased, the precision of the estimates is decreased. Without left truncation and late entry, the number of steps improves the precision of the estimates, whereas in combination of left truncation and late entry, the number of steps decreases the precision. Given the same number of participants and clusters, a parallel group design has higher precision than a stepped wedge design. Copyright © 2016 John Wiley & Sons, Ltd.     

Information content of stepped-wedge designs when treatment effect heterogeneity and/or implementation periods are present

Stepped-wedge cluster randomized trials, which randomize clusters of subjects to treatment sequences in which clusters switch from control to intervention conditions, are being conducted with increasing frequency. Due to the real-world nature of this design, methodological and implementation challenges are ubiquitous. To account for such challenges, more complex statistical models to plan studies and analyze data are required. In this paper, we consider stepped-wedge trials that accommodate treatment effect heterogeneity across clusters, implementation periods during which no data are collected, or both treatment effect heterogeneity and implementation periods. Previous work has shown that the sequence-period cells of a stepped-wedge design contribute unequal amounts of information to the estimation of the treatment effect. In this paper, we extend that work by considering the amount of information available for the estimation of the treatment effect in each sequence-period cell, sequence, and period of stepped-wedge trials with more complex designs and outcome models. When either treatment effect heterogeneity and/or implementation periods are present, the pattern of information content of sequence-period cells tends to be clustered around the times of the switch from control to intervention condition, similarly to when these complexities are absent. However, the presence and degree of treatment effect heterogeneity and the number of implementation periods can influence the information content of periods and sequences markedly.     

Efficient design of cluster randomized trials with treatment‐dependent costs and treatment‐dependent unknown variances

Cluster randomized trials evaluate the effect of a treatment on persons nested within clusters, where treatment is randomly assigned to clusters. Current equations for the optimal sample size at the cluster and person level assume that the outcome variances and/or the study costs are known and homogeneous between treatment arms. This paper presents efficient yet robust designs for cluster randomized trials with treatment‐dependent costs and treatment‐dependent unknown variances, and compares these with 2 practical designs. First, the maximin design (MMD) is derived, which maximizes the minimum efficiency (minimizes the maximum sampling variance) of the treatment effect estimator over a range of treatment‐to‐control variance ratios. The MMD is then compared with the optimal design for homogeneous variances and costs (balanced design), and with that for homogeneous variances and treatment‐dependent costs (cost‐considered design). The results show that the balanced design is the MMD if the treatment‐to control cost ratio is the same at both design levels (cluster, person) and within the range for the treatment‐to‐control variance ratio. It still is highly efficient and better than the cost‐considered design if the cost ratio is within the range for the squared variance ratio. Outside that range, the cost‐considered design is better and highly efficient, but it is not the MMD. An example shows sample size calculation for the MMD, and the computer code (SPSS and R) is provided as supplementary material. The MMD is recommended for trial planning if the study costs are treatment‐dependent and homogeneity of variances cannot be assumed.     

The stepped wedge design

Not all questions concerning therapeutic effects in medical research can be answered satisfactorily by standard trials. The stepped wedge cluster design is a special form of randomised study in which an intervention at group level is implemented in stages. The design can be used in situations where randomization at the patient level is inappropriate or impossible and where a stepped implementation of the experimental intervention is important for ethical, logistic or financial reasons. The time before the experimental intervention is introduced in a cluster is used as control period. The statistical analysis of cluster trials should account for the fact that individuals within clusters may show many similarities. Ethical aspects of stepped wedge design should be considered carefully, especially when obtaining individual informed consent is not possible.     

A marginal estimate for the overall treatment effect on a survival outcome within the joint modeling framework

Joint models for longitudinal and survival data are increasingly used and enjoy a wide range of application areas. In this article, we focus on the application of joint models on clinical trial data with special interest in the treatment effect on the survival outcome. Within a joint model, the estimated treatment effect on the survival outcome is an aggregate comprising the indirect treatment effect through the longitudinal outcome and the direct treatment effect on the survival outcome. This overall treatment effect is, however, conditional on random effects, and therefore has a subject-specific interpretation. The conditional interpretation arises from the shared random effects between the longitudinal and survival process in combination with the nonlinear link function of the survival model. The overall treatment effect is, therefore, not valid for population-based inference, which is the goal for most clinical trials. We propose a method to obtain a marginal estimate of the overall treatment effect on the survival outcome in a joint model. Additionally, we extend our proposal to allow for different parameterizations for the association between the longitudinal and survival outcome. The proposed method is demonstrated on data of a clinical study on the effect of synbiotic on the gut microbiota of cesarean delivered infants, where we estimate the marginal overall treatment effect on the risk of eczema or atopic dermatitis using longitudinal information on fecal bifidobacteria.