Intracluster correlation coefficients from the 2005 WHO Global Survey on Maternal and Perinatal Health: implications for implementation research

Cluster-based studies involving aggregate units such as hospitals or medical practices are increasingly being used in healthcare evaluation. An important characteristic of such studies is the presence of intracluster correlation, typically quantified by the intracluster correlation coefficient (ICC). Sample size calculations for cluster-based studies need to account for the ICC, or risk underestimating the sample size required to yield the desired levels of power and significance. In this article, we present values for ICCs that were obtained from data on 97,095 pregnancies and 98,072 births taking place in a representative sample of 120 hospitals in eight Latin American countries. We present ICCs for 86 variables measured on mothers and newborns from pregnancy to the time of hospital discharge, including 'process variables' representing actual medical care received for each mother and newborn. Process variables are of primary interest in the field of implementation research. We found that overall, ICCs ranged from a minimum of 0.0003 to a maximum of 0.563 (median 0.067). For maternal and newborn outcome variables, the median ICCs were 0.011 (interquartile range 0.007-0.037) and 0.054 (interquartile range 0.013-0.075) respectively; however, for process variables, the median was 0.161 (interquartile range 0.072-0.328). Thus, we confirm previous findings that process variables tend to have higher ICCs than outcome variables. We demonstrate that ICCs generally tend to increase with higher prevalences (close to 0.5). These results can help researchers calculate the required sample size for future research studies in maternal and perinatal health.     

Constructing intervals for the intracluster correlation coefficient using Bayesian modelling, and application in cluster randomized trials

Studies in health research are commonly carried out in clustered settings, where the individual response data are correlated within clusters. Estimation and modelling of the extent of between-cluster variation contributes to understanding of the current study and to design of future studies. It is common to express between-cluster variation as an intracluster correlation coefficient (ICC), since this measure is directly comparable across outcomes. ICCs are generally reported unaccompanied by confidence intervals. In this paper, we describe a Bayesian modelling approach to interval estimation of the ICC. The flexibility of this framework allows useful extensions which are not easily available in existing methods, for example assumptions other than Normality for continuous outcome data, adjustment for individual-level covariates and simultaneous interval estimation of several ICCs. There is also the opportunity to incorporate prior beliefs on likely values of the ICC. The methods are exemplified using data from a cluster randomized trial.     

Intraclass correlation coefficients from three cluster randomised controlled trials in primary and residential health care

OBJECTIVE: This paper provides intraclass correlation coefficients (ICCs) for estimation of sample size inflation required in future cluster randomised trials in primary or residential care. METHODS: Three cluster randomised trials were conducted among middle-aged and older adults in primary care and residential care in Australia and New Zealand between 1995 and 2002. Baseline means or proportions, mean change, and ICCs with their standard errors and 95% confidence intervals are reported for outcome variables used in the three studies. The ICCs were estimated from a one-way random effects model using the analysis of variance method. RESULTS: ICCs for quality of life and psychological variables in the primary care studies were low (below 0.018). ICCs for clinical and physical activity variables ranged from 0 to 0.08. ICCs for health and functional status in residential care for the elderly were high, ranging from 0.025 to 0.514. CONCLUSIONS: The magnitude of the intraclass correlation varies with the venue of the trial, the outcome variables used, and the expected effect of the intervention. However, the intraclass correlations provided will be useful for more appropriate planning of residential and primary care-based trials in the future.     

Calculating sample sizes for cluster randomized trials: We can keep it simple and efficient!

ObjectiveSimple guidelines for calculating efficient sample sizes in cluster randomized trials with unknown intraclass correlation (ICC) and varying cluster sizes.MethodsA simple equation is given for the optimal number of clusters and sample size per cluster. Here, optimal means maximizing power for a given budget or minimizing total cost for a given power. The problems of cluster size variation and specification of the ICC of the outcome are solved in a simple yet efficient way.ResultsThe optimal number of clusters goes up, and the optimal sample size per cluster goes down as the ICC goes up or as the cluster-to-person cost ratio goes down. The available budget, desired power, and effect size only affect the number of clusters and not the sample size per cluster, which is between 7 and 70 for a wide range of cost ratios and ICCs. Power loss because of cluster size variation is compensated by sampling 10% more clusters. The optimal design for the ICC halfway the range of realistic ICC values is a good choice for the first stage of a two-stage design. The second stage is needed only if the first stage shows the ICC to be higher than assumed.ConclusionEfficient sample sizes for cluster randomized trials are easily computed, provided the cost per cluster and cost per person are specified.     

The effect of cluster randomization on sample size in prevention research

BACKGROUND: This paper concerns the issue of cluster randomization in primary care practice intervention trials. We present information on the cluster effect of measuring the performance of various preventive maneuvers between groups of physicians based on a successful trial. We discuss the intracluster correlation coefficient of determining the required sample size and the implications for designing randomized controlled trials where groups of subjects (e.g., physicians in a group practice) are allocated at random. METHODS: We performed a cross-sectional study involving data from 46 participating practices with 106 physicians collected using self-administered questionnaires and a chart audit of 100 randomly selected charts per practice. The population was health service organizations (HSOs) located in Southern Ontario. We analyzed performance data for 13 preventive maneuvers determined by chart review and used analysis of variance to determine the intraclass correlation coefficient. An index of "up-to-datedness" was computed for each physician and practice as the number of a recommended preventive measure done divided by the number of eligible patients. An index called "inappropriateness" was computed in the same manner for the not-recommended measures. The intraclass correlation coefficients for 2 key study outcomes (up-to-datedness and inappropriateness) were also calculated and compared. RESULTS: The mean up-to-datedness score for the practices was 53.5% (95% confidence interval [CI], 51.0%-56.0%), and the mean inappropriateness score was 21.5% (95% CI, 18.1%-24.9%). The intraclass correlation for up-to-datedness was 0.0365 compared with inappropriateness at 0.1790. The intraclass correlation for preventive maneuvers ranged from 0.005 for blood pressure measurement to 0.66 for chest radiographs of smokers, and as a consequence required the sample size ranged from 20 to 42 physicians per group. CONCLUSIONS: Randomizing by practice clusters and analyzing at the level of the physician has important implications for sample size requirements. Larger intraclass correlations indicate interdependence among the physicians within a cluster; as a consequence, variability within clusters is reduced, and the required sample size increased. The key finding that many potential outcome measures perform differently in terms of the intracluster correlation reinforces the need for researchers to carefully consider the selection of outcome measures and adjust sample sizes accordingly when the unit of analysis and randomization are not the same.     

Intra-class correlation estimates for assessment of vitamin A intake in children

In many community-based surveys, multi-level sampling is inherent in the design. In the design of these studies, especially to calculate the appropriate sample size, investigators need good estimates of intra-class correlation coefficient (ICC), along with the cluster size, to adjust for variation inflation due to clustering at each level. The present study used data on the assessment of clinical vitamin A deficiency and intake of vitamin A-rich food in children in a district in India. For the survey, 16 households were sampled from 200 villages nested within eight randomly-selected blocks of the district. ICCs and components of variances were estimated from a three-level hierarchical random effects analysis of variance model. Estimates of ICCs and variance components were obtained at village and block levels. Between-cluster variation was evident at each level of clustering. In these estimates, ICCs were inversely related to cluster size, but the design effect could be substantial for large clusters. At the block level, most ICC estimates were below 0.07. At the village level, many ICC estimates ranged from 0.014 to 0.45. These estimates may provide useful information for the design of epidemiological studies in which the sampled (or allocated) units range in size from households to large administrative zones.     

Sample size considerations in observational health care quality studies

A common objective in health care quality studies involves measuring and comparing the quality of care delivered to cohorts of patients by different health care providers. The data used for inference involve observations on units grouped within clusters, such as patients treated within hospitals. Unlike cluster randomization trials where often clusters are randomized to interventions to learn about individuals, the target of inference in health quality studies is the cluster. Furthermore, randomization is often not performed and the resulting biases may invalidate standard tests. In this paper, we discuss approaches to sample size determination in the design of observational health quality studies when the outcome is binary. Methods for calculating sample size using marginal models are briefly reviewed, but the focus is on hierarchical binomial models. Sample size in unbalanced clusters and stratified designs are characterized. We draw upon the experiences that have arisen from a study funded by the Agency for Healthcare Research and Quality involving assessment of quality of care for patients with cardiovascular disease. If researchers are interested in comparing clusters, hierarchical models are preferred.     

Intraclass correlation for measures from a worksite health promotion study: estimates, correlates, and applications

Intervention trials that employ a group-randomized trial design require an adaptation of the usual analytic methods to account for the randomization of intact economic/social groups to study conditions and the positive ICC that is implied by such a design. In the absence of valid estimates of the ICCs for the outcomes of interest, investigators designing trials could only guess at how large a problem they faced and how much they would need to increase sample size to compensate. Aside from this paper, we are aware of only one other publication that provides such estimates, and that study provides estimates for only a handful of outcomes. Our purpose here has been to provide a replication and extension of those findings to a broader array of outcomes. The results presented here suggest that worksite-level ICCs for a variety of smoking and health-related outcomes are generally small and that these ICCs can generally be reduced by adjustment for individual-level characteristics. We have demonstrated how information about these ICCs can be incorporated in sample size calculations to avoid designing "underpowered" studies. Our results should assist investigators in planning studies to evaluate the effectiveness of worksite-based health promotion efforts.     

The design features and practicalities of conducting a pragmatic cluster randomized trial of obesity management in primary care

The aim of this paper is to describe the design features and practicalities of conducting a cluster randomized trial of obesity management in primary care. The aim of the trial is to assess the effectiveness of an obesity management educational intervention delivered to staff within primary care practices (unit of randomization) in terms of change in body weight of their patients (unit of analysis) at one year. The design features which merit particular attention in this cluster randomized trial include standardization of intervention, sample size considerations, recruitment of patients prior to randomization of practices, method of randomization to balance control and intervention practices with respect to practice and patient level characteristics, and blinding of outcome assessment. The practical problems (and our solutions) associated with implementing these design features, particularly those that result in a time delay between baseline data collection, randomization and intervention, are discussed.     

Comparison of tests for categorical data from a stratified cluster randomized trial

Two features commonly exhibited by randomized trials of health promotion interventions are cluster randomization and stratification. Ignoring correlations between individuals within clusters can lead to an inflated type I error rate and hence a P-value which overstates the significance of the result. This paper compares several methods for analysing categorical data from a stratified cluster randomized trial. We propose an extension of a method from survey sampling that uses the design effect to reduce the effective sample size. We compare this with three methods from Zhang and Boos that extend the standard Cochran-Mantel-Haenszel (CMH) statistic by using appropriate covariance matrices, and with a bootstrap method. The comparison is based on empirical type I error rates from a simulation study, in which the number of clusters randomized is small, as in most public health intervention studies. The method that performs consistently well is one of the Zhang and Boos extensions of the standard CMH statistic.     

Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

Cluster randomized trials (CRTs) refer to experiments with randomization carried out at the cluster or the group level. While numerous statistical methods have been developed for the design and analysis of CRTs, most of the existing methods focused on testing the overall treatment effect across the population characteristics, with few discussions on the differential treatment effect among subpopulations. In addition, the sample size and power requirements for detecting differential treatment effect in CRTs remain unclear, but are helpful for studies planned with such an objective. In this article, we develop a new sample size formula for detecting treatment effect heterogeneity in two-level CRTs for continuous outcomes, continuous or binary covariates measured at cluster or individual level. We also investigate the roles of two intraclass correlation coefficients (ICCs): the adjusted ICC for the outcome of interest and the marginal ICC for the covariate of interest. We further derive a closed-form design effect formula to facilitate the application of the proposed method, and provide extensions to accommodate multiple covariates. Extensive simulations are carried out to validate the proposed formula in finite samples. We find that the empirical power agrees well with the prediction across a range of parameter constellations, when data are analyzed by a linear mixed effects model with a treatment-by-covariate interaction. Finally, we use data from the HF-ACTION study to illustrate the proposed sample size procedure for detecting heterogeneous treatment effects.     

When Intraclass Correlation Coefficients Go Awry: A Case Study From a School-Based Smoking Prevention Study in South Africa

Objectives. We conducted a group randomized trial of 2 South African school-based smoking prevention programs and examined possible sources and implications of why our actual intraclass correlation coefficients (ICCs) were significantly higher than the ICC of 0.02 used to compute initial sample size requirements.Methods. Thirty-six South African high schools were randomly assigned to 1 of 3 experimental groups. On 3 occasions, students completed questionnaires on tobacco and drug use attitudes and behaviors. We used mixed-effects models to partition individual and school-level variance components, with and without covariate adjustment.Results. For 30-day smoking, unadjusted ICCs ranged from 0.12 to 0.17 across the 3 time points. For lifetime smoking, ICCs ranged from 0.18 to 0.22; for other drug use variables, 0.02 to 0.10; and for psychosocial variables, 0.09 to 0.23. Covariate adjustment substantially reduced most ICCs.Conclusions. The unadjusted ICCs we observed for smoking behaviors were considerably higher than those previously reported. This effectively reduced our sample size by a factor of 17. Future studies that anticipate significant cluster-level racial homogeneity may consider using higher-value ICCs in sample-size calculations to ensure adequate statistical power.It is common when one is conducting public health interventions to randomize and then intervene with intact social groups, such as schools, churches, or worksites, rather than individuals. Appropriate analysis of such group- or cluster-randomized trials must account for the statistical similarity of participants within these larger units.^1,2 Individuals within clusters may enter the study with greater similarity than individuals randomly selected from the general population and they may also respond to an intervention in a dependent manner. Failure to account for either source of similarity violates the basic premise of participant nonindependence assumed in traditional statistical approaches and can result in inflated type I error.^1,2 Such within-group similarity is typically captured by the intraclass correlation coefficient (ICC).The following metaphor may help conceptualize how an ICC can affect sample size. A zero ICC occurs when 2 completely unrelated individuals are randomly selected from the general population. In this case, each person contributes a full independent observation. If, however, siblings from the same household were selected, because of their shared environment and genetics, depending on the degree of similarity, this would result in slightly less than 2 independent observations. If nonidentical twins were selected, they would contribute an even lower degree of independent observation; identical twins, even less. In the extreme case, if Siamese twins were recruited, they would begin to approach the contribution of a single individual. In most cluster-randomized studies, the degree of nonindependence within a group is low, but its impact on overall statistical power can still be great because of the variance inflation formulae used to adjust for the ICC.If the ICC is known before conducting a trial, sample size requirements can be adjusted upward to account for the reduction in true sample size and statistical power, decreasing the chances of type II error. As noted by Murray et al.,³ however, it is often difficult to obtain a reliable ICC estimate that is entirely applicable to one''s proposed study. Archival ICC estimates are often derived from samples, measures, contexts, or study designs that differ from the one proposed, limiting their utility. Whereas overestimation of the ICC can lead to an overpowered study and, therefore, inefficient use of resources, underestimation of the ICC can lead to an underpowered study (i.e., type II error).We recently conducted a group randomized trial of 2 school-based smoking prevention programs designed for South African youths from 2004 to 2008. In computing our a priori sample-size requirements, we used ICC estimates drawn from published school-based smoking prevention studies, all of which were conducted in the United States. Prior ICCs were predominantly in the range of 0.02 to 0.04,⁴ and our sample size was computed accordingly. However, when the study was completed and we examined the actual ICC it turned out to be significantly higher than we had anticipated. As a result, many of our outcomes that would have been statistically significant with the ICC that was originally projected turned out to be null. We felt it would be useful to publish and explain these ICCs to assist other investigators.     

Individual allocation had an advantage over cluster randomization in statistical efficiency in some circumstances.

OBJECTIVE: To investigate the trade off between performing an individual randomized trial with a Complier Average Causal Effect analysis and accepting the fact that there will be some contamination, with a cluster randomized trial and the subsequent effect on the sample size and power of the trial. STUDY DESIGN AND SETTING: Monte Carlo simulations were undertaken to generate trial data where there was some contamination of control participants. The trials were simulated so the null hypothesis was false. Assessments were made of whether a type II error had been committed (i.e., whether the false null had not been rejected). RESULTS: As contamination increases, the power of the study to detect a true difference between the two groups declines. Using a Complier Average Causal Effect approach, unless anticipated contamination exceeds 30%, retains a sample size advantage over the cluster randomized design despite a relatively small cluster (i.e., 10) and reasonably small ICC (i.e., 0.04). CONCLUSION: If contamination can be measured precisely or estimated then in some circumstances individual allocation has an advantage over cluster randomization in statistical efficiency.     

Patterns of intra-cluster correlation from primary care research to inform study design and analysis

OBJECTIVE: To provide information concerning the magnitude of the intraclass correlation coefficient (ICC) for cluster-based studies set in primary care. STUDY DESIGN AND SETTING: Reanalysis of data from 31 cluster-based studies in primary care to estimate intraclass correlation coefficients from random effects models using maximum likelihood estimation. RESULTS: ICCs were estimated for 1,039 variables. The median ICC was 0.010 (interquartile range [IQR] 0 to 0.032, range 0 to 0.840). After adjusting for individual- and cluster-level characteristics, the median ICC was 0.005 (IQR 0 to 0.021). A given measure showed widely varying ICC estimates in different datasets. In six datasets, the ICCs for SF-36 physical functioning scale ranged from 0.001 to 0.055 and for SF-36 general health from 0 to 0.072. In four datasets, the ICC for systolic blood pressure ranged from 0 to 0.052 and for diastolic blood pressure from 0 to 0.108. CONCLUSION: The precise magnitude of between-cluster variation for a given measure can rarely be estimated in advance. Studies should be designed with reference to the overall distribution of ICCs and with attention to features that increase efficiency.     

Sample size calculations for cluster randomised trials. Changing Professional Practice in Europe Group (EU BIOMED II Concerted Action)

OBJECTIVES: Cluster randomised trials, in which groups of individuals are randomised, are increasingly being used in the health field. Adopting a clustered approach has implications for the design of such trials, and sample size calculations need to be inflated to accommodate for the clustering effect. Reliable estimates of intracluster correlation coefficients (ICCs) are required for robust sample size calculations to be made; however, little empirical evidence is available on their likely size, and on factors which influence their magnitude. The aim of this study was to generate empirical estimates of ICCs and to explore factors which may affect their magnitude. METHODS: Empirical estimates of ICCs were calculated for both process variables and patient outcomes from a number of datasets of primary and secondary care implementation studies. RESULTS: Estimates of ICCs varied according to setting and type of outcome. Estimates of ICCs for process variables were higher than those for patient outcomes, and estimates derived from secondary care were higher than those from primary care. ICCs for process variables in primary care were of the order of 0.05-0.15, whilst those in secondary care were of the order of 0.3. Estimates for patient outcomes in primary care were generally lower than 0.05. CONCLUSIONS: Adopting cluster randomisation has implications for the design, size and analysis of clinical trials. This study gives an insight into the potential size of ICCs in primary and secondary care, and provides a practical guide to researchers to aid the planning of future studies in this area.     

Sample size calculation for cluster randomization trials with a time-to-event endpoint

Cluster randomization trials randomize groups (called clusters) of subjects (called subunits) between intervention arms, and observations are collected from each subject. In this case, subunits within each cluster share common frailties, so that the observations from subunits of each cluster tend to be correlated. Oftentimes, the outcome of a cluster randomization trial is a time-to-event endpoint with censoring. In this article, we propose a closed form sample size formula for weighted rank tests to compare the marginal survival distributions between intervention arms under cluster randomization with possibly variable cluster sizes. Extensive simulation studies are conducted to evaluate the performance of our sample size formula under various design settings. Real study examples are taken to demonstrate our method.     

A comparison of methods to analyse continuous data from pseudo cluster randomized trials

A major methodological reason to use cluster randomization is to avoid the contamination that would arise in an individually randomized design. However, when patient recruitment cannot be completed before randomization of clusters, the non-blindedness of recruiters and patients may cause selection bias, while in the control clusters, it may slow recruitment due to patient or recruiter preferences for the intervention. As a compromise, pseudo cluster randomization has been proposed. Because no insight is available into the relative performance of methods to analyse data obtained from this design, we compared the type I and II error rates of mixed models, generalized estimating equations (GEE) and a paired t-test to those of the estimator originally proposed in this design. The bias in the point estimate and its standard error were also incorporated into this comparison. Furthermore, we evaluated the effect of the weighting scheme and the accuracy of the sample size formula that have been described previously. Power levels of the originally proposed estimator and the unweighted mixed models were in agreement with the sample size formula, but the power of paired t-test fell short. GEE produced too large type I errors, unless the number of clusters was large (>30-40 per arm). The use of the weighting scheme generally enhanced the power, but at the cost of increasing the type I error in mixed models and GEE. We recommend unweighted mixed models as the best compromise between feasibility and power to analyse data from a pseudo cluster randomized trial.     

Sample size estimation for stratified individual and cluster randomized trials with binary outcomes

Individual randomized trials (IRTs) and cluster randomized trials (CRTs) with binary outcomes arise in a variety of settings and are often analyzed by logistic regression (fitted using generalized estimating equations for CRTs). The effect of stratification on the required sample size is less well understood for trials with binary outcomes than for continuous outcomes. We propose easy-to-use methods for sample size estimation for stratified IRTs and CRTs and demonstrate the use of these methods for a tuberculosis prevention CRT currently being planned. For both IRTs and CRTs, we also identify the ratio of the sample size for a stratified trial vs a comparably powered unstratified trial, allowing investigators to evaluate how stratification will affect the required sample size when planning a trial. For CRTs, these can be used when the investigator has estimates of the within-stratum intracluster correlation coefficients (ICCs) or by assuming a common within-stratum ICC. Using these methods, we describe scenarios where stratification may have a practically important impact on the required sample size. We find that in the two-stratum case, for both IRTs and for CRTs with very small cluster sizes, there are unlikely to be plausible scenarios in which an important sample size reduction is achieved when the overall probability of a subject experiencing the event of interest is low. When the probability of events is not small, or when cluster sizes are large, however, there are scenarios where practically important reductions in sample size result from stratification.     

Sample size re-estimation in cluster randomization trials

Cluster randomization trials in which families are the unit of allocation are commonly adopted for the evaluation of disease prevention interventions. Sample size estimation for cluster randomization trials depends on parameters that quantify the variability within and between clusters and the variability in cluster size. Accurate advance estimates of these nuisance parameters may be difficult to obtain and misspecification may lead to an underpowered study. Since families are typically recruited over time, we propose using a portion of the data to estimate the nuisance parameters and to re-estimate sample size based on the estimates. This extends the standard internal pilot study methods to the setting of cluster randomization trials. The effect of this design on the power, significance level and sample size is analysed via simulation and is shown to provide a flexible and practical approach to cluster randomization trials.