Robust analysis of stepped wedge trials using cluster‐level summaries within periods

In stepped‐wedge trials (SWTs), the intervention is rolled out in a random order over more than 1 time‐period. SWTs are often analysed using mixed‐effects models that require strong assumptions and may be inappropriate when the number of clusters is small. We propose a non‐parametric within‐period method to analyse SWTs. This method estimates the intervention effect by comparing intervention and control conditions in a given period using cluster‐level data corresponding to exposure. The within‐period intervention effects are combined with an inverse‐variance‐weighted average, and permutation tests are used. We present an example and, using simulated data, compared the method to (1) a parametric cluster‐level within‐period method, (2) the most commonly used mixed‐effects model, and (3) a more flexible mixed‐effects model. We simulated scenarios where period effects were common to all clusters, and when they varied according to a distribution informed by routinely collected health data. The non‐parametric within‐period method provided unbiased intervention effect estimates with correct confidence‐interval coverage for all scenarios. The parametric within‐period method produced confidence intervals with low coverage for most scenarios. The mixed‐effects models' confidence intervals had low coverage when period effects varied between clusters but had greater power than the non‐parametric within‐period method when period effects were common to all clusters. The non‐parametric within‐period method is a robust method for analysing SWT. The method could be used by trial statisticians who want to emphasise that the SWT is a randomised trial, in the common position of being uncertain about whether data will meet the assumptions necessary for mixed‐effect models.     

Maintaining the validity of inference in small-sample stepped wedge cluster randomized trials with binary outcomes when using generalized estimating equations

Stepped wedge cluster trials are an increasingly popular alternative to traditional parallel cluster randomized trials. Such trials often utilize a small number of clusters and numerous time intervals, and these components must be considered when choosing an analysis method. A generalized linear mixed model containing a random intercept and fixed time and intervention covariates is the most common analysis approach. However, the sole use of a random intercept applies a constant intraclass correlation coefficient structure, which is an assumption that is likely to be violated given stepped wedge trials (SWTs) have multiple time intervals. Alternatively, generalized estimating equations (GEE) are robust to the misspecification of the working correlation structure, although it has been shown that small-sample adjustments to standard error estimates and the use of appropriate degrees of freedom are required to maintain the validity of inference when the number of clusters is small. In this article, we show, using an extensive simulation study based on a motivating example and a more general design, the use of GEE can maintain the validity of inference in small-sample SWTs with binary outcomes. Furthermore, we show which combinations of bias corrections to standard error estimates and degrees of freedom work best in terms of attaining nominal type I error rates.     

A simulation study of odds ratio estimation for binary outcomes from cluster randomized trials

We used simulation to compare accuracy of estimation and confidence interval coverage of several methods for analysing binary outcomes from cluster randomized trials. The following methods were used to estimate the population-averaged intervention effect on the log-odds scale: marginal logistic regression models using generalized estimating equations with information sandwich estimates of standard error (GEE); unweighted cluster-level mean difference (CL/U); weighted cluster-level mean difference (CL/W) and cluster-level random effects linear regression (CL/RE). Methods were compared across trials simulated with different numbers of clusters per trial arm, numbers of subjects per cluster, intraclass correlation coefficients (rho), and intervention versus control arm proportions. Two thousand data sets were generated for each combination of design parameter values. The results showed that the GEE method has generally acceptable properties, including close to nominal levels of confidence interval coverage, when a simple adjustment is made for data with relatively few clusters. CL/U and CL/W have good properties for trials where the number of subjects per cluster is sufficiently large and rho is sufficiently small. CL/RE also has good properties in this situation provided a t-distribution multiplier is used for confidence interval calculation in studies with small numbers of clusters. For studies where the number of subjects per cluster is small and rho is large all cluster-level methods may perform poorly for studies with between 10 and 50 clusters per trial arm.     

Multiple imputation methods for bivariate outcomes in cluster randomised trials

Missing observations are common in cluster randomised trials. The problem is exacerbated when modelling bivariate outcomes jointly, as the proportion of complete cases is often considerably smaller than the proportion having either of the outcomes fully observed. Approaches taken to handling such missing data include the following: complete case analysis, single‐level multiple imputation that ignores the clustering, multiple imputation with a fixed effect for each cluster and multilevel multiple imputation. We contrasted the alternative approaches to handling missing data in a cost‐effectiveness analysis that uses data from a cluster randomised trial to evaluate an exercise intervention for care home residents. We then conducted a simulation study to assess the performance of these approaches on bivariate continuous outcomes, in terms of confidence interval coverage and empirical bias in the estimated treatment effects. Missing‐at‐random clustered data scenarios were simulated following a full‐factorial design. Across all the missing data mechanisms considered, the multiple imputation methods provided estimators with negligible bias, while complete case analysis resulted in biased treatment effect estimates in scenarios where the randomised treatment arm was associated with missingness. Confidence interval coverage was generally in excess of nominal levels (up to 99.8%) following fixed‐effects multiple imputation and too low following single‐level multiple imputation. Multilevel multiple imputation led to coverage levels of approximately 95% throughout. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

A multilevel model framework for meta-analysis of clinical trials with binary outcomes

In this paper we explore the potential of multilevel models for meta-analysis of trials with binary outcomes for both summary data, such as log-odds ratios, and individual patient data. Conventional fixed effect and random effects models are put into a multilevel model framework, which provides maximum likelihood or restricted maximum likelihood estimation. To exemplify the methods, we use the results from 22 trials to prevent respiratory tract infections; we also make comparisons with a second example data set comprising fewer trials. Within summary data methods, confidence intervals for the overall treatment effect and for the between-trial variance may be derived from likelihood based methods or a parametric bootstrap as well as from Wald methods; the bootstrap intervals are preferred because they relax the assumptions required by the other two methods. When modelling individual patient data, a bias corrected bootstrap may be used to provide unbiased estimation and correctly located confidence intervals; this method is particularly valuable for the between-trial variance. The trial effects may be modelled as either fixed or random within individual data models, and we discuss the corresponding assumptions and implications. If random trial effects are used, the covariance between these and the random treatment effects should be included; the resulting model is equivalent to a bivariate approach to meta-analysis. Having implemented these techniques, the flexibility of multilevel modelling may be exploited in facilitating extensions to standard meta-analysis methods.     

Propensity score matching with clustered data. An application to the estimation of the impact of caesarean section on the Apgar score

This article focuses on the implementation of propensity score matching for clustered data. Different approaches to reduce bias due to cluster‐level confounders are considered and compared using Monte Carlo simulations. We investigated methods that exploit the clustered structure of the data in two ways: in the estimation of the propensity score model (through the inclusion of fixed or random effects) or in the implementation of the matching algorithm. In addition to a pure within‐cluster matching, we also assessed the performance of a new approach, ‘preferential’ within‐cluster matching. This approach first searches for control units to be matched to treated units within the same cluster. If matching is not possible within‐cluster, then the algorithm searches in other clusters. All considered approaches successfully reduced the bias due to the omission of a cluster‐level confounder. The preferential within‐cluster matching approach, combining the advantages of within‐cluster and between‐cluster matching, showed a relatively good performance both in the presence of big and small clusters, and it was often the best method. An important advantage of this approach is that it reduces the number of unmatched units as compared with a pure within‐cluster matching. We applied these methods to the estimation of the effect of caesarean section on the Apgar score using birth register data. Copyright © 2016 John Wiley & Sons, Ltd.     

Analysis of a cluster randomized trial with binary outcome data using a multi-level model

The use of multi-level logistic regression models was explored for the analysis of data from a cluster randomized trial investigating whether a training programme for general practitioners' reception staff could improve women's attendance at breast screening. Twenty-six general practices were randomized with women nested within them, requiring a two-level model which allowed for between-practice variability. Comparisons were made with fixed effect (FE) and random effects (RE) cluster summary statistic methods, ordinary logistic regression and a marginal model based on generalized estimating equations with robust variance estimates. An FE summary statistic method and ordinary logistic regression considerably understated the variance of the intervention effect, thus overstating its statistical significance. The marginal model produced a higher statistical significance for the intervention effect compared to that obtained from the RE summary statistic method and the multi-level model. Because there was only a moderate number of practices and these had unbalanced cluster sizes, reliable asymptotic properties for the robust standard errors used in the marginal model may not have been achieved. While the RE summary statistic method cannot handle multiple covariates easily, marginal and multi-level models can do so. In contrast to multi-level models however, marginal models do not provide direct estimates of variance components, but treat these as nuisance parameters. Estimates of the variance components were of particular interest in this example. Additionally, parametric bootstrap methods within the multi-level model framework provide confidence intervals for these variance components, as well as a confidence interval for the effect of intervention which allows for the imprecision in the estimated variance components. The assumption of normality of the random effects can be checked, and the models extended to investigate multiple sources of variability.     

Model‐based standardization to adjust for unmeasured cluster‐level confounders with complex survey data

Model‐based standardization uses a statistical model to estimate a standardized, or unconfounded, population‐averaged effect. With it, one can compare groups had the distribution of confounders been identical in both groups to that of the standard population. We develop two methods for model‐based standardization with complex survey data that accommodate a categorical confounder that clusters the individual observations into a very large number of subgroups. The first method combines a random‐intercept generalized linear mixed model with a conditional pseudo‐likelihood estimator of the fixed effects. The second method combines a between–within generalized linear mixed model with census data on the cluster‐level means of the individual‐level covariates. We conduct simulation studies to compare the two approaches. We apply the two methods to the 2008 Florida Behavioral Risk Factor Surveillance System survey data to estimate standardized proportions of people who drink alcohol, within age groups, adjusting for measured individual‐level and unmeasured cluster‐level confounders. Copyright © 2015 John Wiley & Sons, Ltd.     

Sample size and robust marginal methods for cluster‐randomized trials with censored event times

In cluster‐randomized trials, intervention effects are often formulated by specifying marginal models, fitting them under a working independence assumption, and using robust variance estimates to address the association in the responses within clusters. We develop sample size criteria within this framework, with analyses based on semiparametric Cox regression models fitted with event times subject to right censoring. At the design stage, copula models are specified to enable derivation of the asymptotic variance of estimators from a marginal Cox regression model and to compute the number of clusters necessary to satisfy power requirements. Simulation studies demonstrate the validity of the sample size formula in finite samples for a range of cluster sizes, censoring rates, and degrees of within‐cluster association among event times. The power and relative efficiency implications of copula misspecification is studied, as well as the effect of within‐cluster dependence in the censoring times. Sample size criteria and other design issues are also addressed for the setting where the event status is only ascertained at periodic assessments and times are interval censored. Copyright © 2014 JohnWiley & Sons, Ltd.     

Transforming the Model T: random effects meta‐analysis with stable weights

Standard meta‐analytic theory assumes that study outcomes are normally distributed with known variances. However, methods derived from this theory are often applied to effect sizes having skewed distributions with estimated variances. Both shortcomings can be largely overcome by first applying a variance stabilizing transformation. Here we concentrate on study outcomes with Student t‐distributions and show that we can better estimate parameters of fixed or random effects models with confidence intervals using stable weights or with profile approximate likelihood intervals following stabilization. We achieve even better coverage with a finite sample bias correction. Further, a simple t‐interval provides very good coverage of an overall effect size without estimation of the inter‐study variance. We illustrate the methodology on two meta‐analytic studies from the medical literature, the effect of salt reduction on systolic blood pressure and the effect of opioids for the relief of breathlessness. Substantial simulation studies compare traditional methods with those newly proposed. We can apply the theoretical results to other study outcomes for which an effective variance stabilizer is available. Copyright © 2012 John Wiley & Sons, Ltd.     

A comparison of the statistical power of different methods for the analysis of cluster randomization trials with binary outcomes

Cluster randomization trials are randomized controlled trials (RCTs) in which intact clusters of subjects are randomized to either the intervention or to the control. Cluster randomization trials require different statistical methods of analysis than do conventional randomized controlled trials due to the potential presence of within-cluster homogeneity in responses. A variety of statistical methods have been proposed in the literature for the analysis of cluster randomization trials with binary outcomes. However, little is known about the relative statistical power of these methods to detect a statistically significant intervention effect. We conducted a series of Monte Carlo simulations to examine the statistical power of three methods that compare cluster-specific response rates between arms of the trial: the t-test, the Wilcoxon rank sum test, and the permutation test; and three methods that compare subject-level response rates: an adjusted chi-square test, a logistic-normal random effects model, and a generalized estimating equations (GEE) method. In our simulations we allowed the number of clusters, the number of subjects per cluster, the intraclass correlation coefficient and the magnitude of the intervention effect to vary. We demonstrated that the GEE approach tended to have the highest power for detecting a statistically significant intervention effect. However, in most of the 240 scenarios examined, the differences between the competing statistical methods were negligible. The largest mean difference in power between any two different statistical methods across the 240 scenarios was 0.02. The largest observed difference in power between two different statistical methods across the 240 scenarios and 15 pair-wise comparisons of methods was 0.14.     

Neither fixed nor random: weighted least squares meta‐analysis

This study challenges two core conventional meta‐analysis methods: fixed effect and random effects. We show how and explain why an unrestricted weighted least squares estimator is superior to conventional random‐effects meta‐analysis when there is publication (or small‐sample) bias and better than a fixed‐effect weighted average if there is heterogeneity. Statistical theory and simulations of effect sizes, log odds ratios and regression coefficients demonstrate that this unrestricted weighted least squares estimator provides satisfactory estimates and confidence intervals that are comparable to random effects when there is no publication (or small‐sample) bias and identical to fixed‐effect meta‐analysis when there is no heterogeneity. When there is publication selection bias, the unrestricted weighted least squares approach dominates random effects; when there is excess heterogeneity, it is clearly superior to fixed‐effect meta‐analysis. In practical applications, an unrestricted weighted least squares weighted average will often provide superior estimates to both conventional fixed and random effects. Copyright © 2015 John Wiley & Sons, Ltd.     

Analysis of cluster randomized cross-over trial data: a comparison of methods

In a cluster randomized cross-over trial, all participating clusters receive both intervention and control treatments consecutively, in separate time periods. Patients recruited by each cluster within the same time period receive the same intervention, and randomization determines order of treatment within a cluster. Such a design has been used on a number of occasions. For analysis of the trial data, the approach of analysing cluster-level summary measures is appealing on the grounds of simplicity, while hierarchical modelling allows for the correlation of patients within periods within clusters and offers flexibility in the model assumptions. We consider several cluster-level approaches and hierarchical models and make comparison in terms of empirical precision, coverage, and practical considerations. The motivation for a cluster randomized trial to employ cross-over of trial arms is particularly strong when the number of clusters available is small, so we examine performance of the methods under small, medium and large (6, 18, 30) numbers of clusters. One hierarchical model and two cluster-level methods were found to perform consistently well across the designs considered. These three methods are efficient, provide appropriate standard errors and coverage, and continue to perform well when incorporating adjustment for an individual-level covariate. We conclude that choice between hierarchical models and cluster-level methods should be influenced by the extent of complexity in the planned analysis.     

Cluster randomized controlled trials in primary care: An introduction

Background: Cluster randomized trials occur when groups or clusters of individuals, rather than the individuals themselves, are randomized to intervention and control groups and outcomes are measured on individuals within those clusters. Within primary care, between 1997 and 2000, there has been a virtual doubling in the number of published cluster randomized trials. A recent systematic review, specifically within primary care, found study quality to be both generally lower than that reported elsewhere and not to have shown any recent quality improvement. Objective: To discuss the design, conduct and analysis of cluster randomized trials within primary care in terms of the appropriate expertise required, potential bias, ethical considerations and expense. Discussion: Compared with trials that involve the randomization of individual participants, cluster randomized trials are more complex to design and analyse and, for a given sample size, have decreased power and a broadening of confidence intervals. Cluster randomized trials are specifically prone to potential bias at two levels—the cluster and individual. Regarding the former, it is recommended that cluster allocation be undertaken by a party independent to the research team and careful consideration be given to ensure minimal cluster attrition. Bias at the individual level can be overcome by identifying trial participants before randomization and at this time obtaining consent for intervention, data collection or both. A unique ethical aspect to cluster randomized trials is that cluster leaders may consent to the trial on behalf of potential cluster members. Additional costs of cluster randomized trials include the increased number of patients required, the complexity in their design and conduct and, usually, the need to recruit clusters de novo.

Conclusion: Cluster randomized trials are a powerful and increasingly popular research tool. They are uniquely placed for the conduct of research within primary-care clusters where intracluster contamination can occur. Associated methodological issues are straightforward and surmountable and just need careful consideration and management.     

Analysis of multicentre trials with continuous outcomes: when and how should we account for centre effects?

In multicentre trials, randomisation is often carried out using permuted blocks stratified by centre. It has previously been shown that stratification variables used in the randomisation process should be adjusted for in the analysis to obtain correct inference. For continuous outcomes, the two primary methods of accounting for centres are fixed‐effects and random‐effects models. We discuss the differences in interpretation between these two models and the implications that each pose for analysis. We then perform a large simulation study comparing the performance of these analysis methods in a variety of situations. In total, we assessed 378 scenarios. We found that random centre effects performed as well or better than fixed‐effects models in all scenarios. Random centre effects models led to increases in power and precision when the number of patients per centre was small (e.g. 10 patients or less) and, in some scenarios, when there was an imbalance between treatments within centres, either due to the randomisation method or to the distribution of patients across centres. With small samples sizes, random‐effects models maintained nominal coverage rates when a degree‐of‐freedom (DF) correction was used. We assessed the robustness of random‐effects models when assumptions regarding the distribution of the centre effects were incorrect and found this had no impact on results. We conclude that random‐effects models offer many advantages over fixed‐effects models in certain situations and should be used more often in practice. Copyright © 2012 John Wiley & Sons, Ltd.     

Theory of general balance applied to step wedge designs

A standard idealized step-wedge design satisfies the requirements, in terms of the structure of the observation units, to be considered a balanced design and can be labeled as a criss-cross design (time crossed with cluster) with replication. As such, Nelder's theory of general balance can be used to decompose the analysis of variance into independent strata (grand mean, cluster, time, cluster:time, residuals). If time is considered as a fixed effect, then the treatment effect of interest is estimated solely within the cluster and time:cluster strata; the time effects are estimated solely within the time stratum. This separation leads directly to scalar, rather than matrix, algebraic manipulations to provide closed-form expressions for standard errors of the treatment effect estimate. We use the tools provided by the theory of general balance to obtain an expression for the standard error of the estimated treatment effect in a general case where the assumed covariance structure includes random-effects at the time and time:cluster levels. This provides insights that are helpful for experimental design regarding the assumed correlation within clusters over time, sample size in terms of numbers of clusters and replication within cluster, and components of the standard error for estimated treatment effect.     

Estimating overall exposure effects for the clustered and censored outcome using random effect Tobit regression models

The random effect Tobit model is a regression model that accommodates both left‐ and/or right‐censoring and within‐cluster dependence of the outcome variable. Regression coefficients of random effect Tobit models have conditional interpretations on a constructed latent dependent variable and do not provide inference of overall exposure effects on the original outcome scale. Marginalized random effects model (MREM) permits likelihood‐based estimation of marginal mean parameters for the clustered data. For random effect Tobit models, we extend the MREM to marginalize over both the random effects and the normal space and boundary components of the censored response to estimate overall exposure effects at population level. We also extend the ‘Average Predicted Value’ method to estimate the model‐predicted marginal means for each person under different exposure status in a designated reference group by integrating over the random effects and then use the calculated difference to assess the overall exposure effect. The maximum likelihood estimation is proposed utilizing a quasi‐Newton optimization algorithm with Gauss–Hermite quadrature to approximate the integration of the random effects. We use these methods to carefully analyze two real datasets. Copyright © 2016 John Wiley & Sons, Ltd.     

Modelling the hierarchical structure in datasets with very small clusters: a simulation study to explore the effect of the proportion of clusters when the outcome is continuous

In cluster‐randomised trials, the problem of non‐independence within clusters is well known, and appropriate statistical analysis documented. Clusters typically seen in cluster trials are large in size and few in number, whereas datasets of preterm infants incorporate clusters of size two (twins), size three (triplets) and so on, with the majority of infants being in ‘clusters’ of size one. In such situations, it is unclear whether adjustment for clustering is needed or even possible. In this paper, we compared analyses allowing for clustering (linear mixed model) with analyses ignoring clustering (linear regression). Through simulations based on two real datasets, we explored estimation bias in predictors of a continuous outcome in different size datasets typical of preterm samples, with varying percentages of twins. Overall, the biases for estimated coefficients were similar for linear regression and mixed models, but the standard errors were consistently much less well estimated when using a linear model. Non‐convergence was rare but was observed in approximately 5% of mixed models for samples below 200 and percentage of twins 2% or less. We conclude that in datasets with small clusters, mixed models should be the method of choice irrespective of the percentage of twins. If the mixed model does not converge, a linear regression can be fitted, but standard error will be underestimated, and so type I error may be inflated. Copyright © 2012 John Wiley & Sons, Ltd.     

Mixed models,linear dependency,and identification in age‐period‐cohort models

This paper examines the identification problem in age‐period‐cohort models that use either linear or categorically coded ages, periods, and cohorts or combinations of these parameterizations. These models are not identified using the traditional fixed effect regression model approach because of a linear dependency between the ages, periods, and cohorts. However, these models can be identified if the researcher introduces a single just identifying constraint on the model coefficients. The problem with such constraints is that the results can differ substantially depending on the constraint chosen. Somewhat surprisingly, age‐period‐cohort models that specify one or more of ages and/or periods and/or cohorts as random effects are identified. This is the case without introducing an additional constraint. I label this identification as statistical model identification and show how statistical model identification comes about in mixed models and why which effects are treated as fixed and which are treated as random can substantially change the estimates of the age, period, and cohort effects. Copyright © 2017 John Wiley & Sons, Ltd.     

Comparison of the risk difference, risk ratio and odds ratio scales for quantifying the unadjusted intervention effect in cluster randomized trials

This paper evaluates methods for unadjusted analyses of binary outcomes in cluster randomized trials (CRTs). Under the generalized estimating equations (GEE) method the identity, log and logit link functions may be specified to make inferences on the risk difference, risk ratio and odds ratio scales, respectively. An alternative, 'cluster-level', method applies the t-test to summary statistics calculated for each cluster, using proportions, log proportions and log odds, to make inferences on the respective scales. Simulation was used to estimate the bias of the unadjusted intervention effect estimates and confidence interval coverage, generating data sets with different combinations of number of clusters, number of participants per cluster, intra-cluster correlation coefficient rho and intervention effect. When the identity link was specified, GEE had little bias and good coverage, performing slightly better than the log and logit link functions. The cluster-level method provided unbiased point estimates when proportions were used to summarize the clusters. When the log proportion and log odds were used, however, the method often had markedly large bias for two reasons: (i) bias in the modified summary statistic used for cluster-level estimation when a cluster has zero cases with the outcome of interest (arising when the number of participants sampled per cluster is small and the outcome prevalence is low) and (ii) asymptotically, the method estimates the ratio of geometric means of the cluster proportions or odds, respectively, between the trial arms rather than the ratio of arithmetic means.