Approaches to heterogeneity in meta-analysis.

This paper reviews publications from January 1999 to March 2001 on reproductive health topics that were self-identified as meta-analysis or were indexed as meta-analysis in MEDLINE. It sought to assess whether tests of statistical heterogeneity were done, whether the results were reported, and how a finding of significance for a test of statistical heterogeneity was handled and the results interpreted. The review identified some concerns. Tests of statistical heterogeneity were not done universally even though virtually all writers on the topic emphasize their importance. Even when done, results of these tests were not universally reported. Although the consensus appears to be that heterogeneity tests are conservative for meta-analysis of studies and a probability value of 0.10 is preferred, many meta-analyses used the conventional value of 0.05 without providing a reason. The rationale for the choice of a random or fixed effects model was not generally evident. The review also provided some positive models and some recommendations for assessing, reporting and exploring heterogeneity are made considering these models and the published recommendations of experts.     

Detecting and describing heterogeneity in meta-analysis

The investigation of heterogeneity is a crucial part of any meta-analysis. While it has been stated that the test for heterogeneity has low power, this has not been well quantified. Moreover the assumptions of normality implicit in the standard methods of meta-analysis are often not scrutinized in practice. Here we simulate how the power of the test for heterogeneity depends on the number of studies included, the total information (that is total weight or inverse variance) available and the distribution of weights among the different studies. We show that the power increases with the total information available rather than simply the number of studies, and that it is substantially lowered if, as is quite common in practice, one study comprises a large proportion of the total information. We also describe normal plots that are useful in assessing whether the data conform to a fixed effect or random effects model, together with appropriate tests, and give an application to the analysis of a multi-centre trial of blood pressure reduction. We conclude that the test of heterogeneity should not be the sole determinant of model choice in meta-analysis, and inspection of relevant normal plots, as well as clinical insight, may be more relevant to both the investigation and modelling of heterogeneity. © 1998 John Wiley & Sons, Ltd.     

Quantifying heterogeneity in a meta-analysis

The extent of heterogeneity in a meta-analysis partly determines the difficulty in drawing overall conclusions. This extent may be measured by estimating a between-study variance, but interpretation is then specific to a particular treatment effect metric. A test for the existence of heterogeneity exists, but depends on the number of studies in the meta-analysis. We develop measures of the impact of heterogeneity on a meta-analysis, from mathematical criteria, that are independent of the number of studies and the treatment effect metric. We derive and propose three suitable statistics: H is the square root of the chi2 heterogeneity statistic divided by its degrees of freedom; R is the ratio of the standard error of the underlying mean from a random effects meta-analysis to the standard error of a fixed effect meta-analytic estimate, and I2 is a transformation of (H) that describes the proportion of total variation in study estimates that is due to heterogeneity. We discuss interpretation, interval estimates and other properties of these measures and examine them in five example data sets showing different amounts of heterogeneity. We conclude that H and I2, which can usually be calculated for published meta-analyses, are particularly useful summaries of the impact of heterogeneity. One or both should be presented in published meta-analyses in preference to the test for heterogeneity.     

Explaining heterogeneity in meta-analysis: a comparison of methods.

Exploring the possible reasons for heterogeneity between studies is an important aspect of conducting a meta-analysis. This paper compares a number of methods which can be used to investigate whether a particular covariate, with a value defined for each study in the meta-analysis, explains any heterogeneity. The main example is from a meta-analysis of randomized trials of serum cholesterol reduction, in which the log-odds ratio for coronary events is related to the average extent of cholesterol reduction achieved in each trial. Different forms of weighted normal errors regression and random effects logistic regression are compared. These analyses quantify the extent to which heterogeneity is explained, as well as the effect of cholesterol reduction on the risk of coronary events. In a second example, the relationship between treatment effect estimates and their precision is examined, in order to assess the evidence for publication bias. We conclude that methods which allow for an additive component of residual heterogeneity should be used. In weighted regression, a restricted maximum likelihood estimator is appropriate, although a number of other estimators are also available. Methods which use the original form of the data explicitly, for example the binomial model for observed proportions rather than assuming normality of the log-odds ratios, are now computationally feasible. Although such methods are preferable in principle, they often give similar results in practice.     

Methods for exploring heterogeneity in meta-analysis

In meta-analysis, when the difference in results between studies is greater than would be expected by chance, one needs to investigate whether the observed variation in results across studies is associated with clinical and/or methodological differences between studies. This article reviews methods used in meta-analysis for exploring heterogeneity, including statistical tests for homogeneity, methods for visually displaying results of primary studies, methods for reducing heterogeneity, methods for investigating sources of heterogeneity, and identification of moderator variables or effect modifiers. The investigation of sources of heterogeneity in meta-analysis is by nature exploratory, and therefore its results should always be interpreted with caution. However, careful investigation of heterogeneity may provide an important second level of evidence that can be useful in suggesting direction of future research. Sometimes, it may provide clinically important results by indicating who might benefit more or less from a treatment or how an intervention should be applied.     

Confidence intervals for the amount of heterogeneity in meta-analysis

Effect size estimates to be combined in a systematic review are often found to be more variable than one would expect based on sampling differences alone. This is usually interpreted as evidence that the effect sizes are heterogeneous. A random-effects model is then often used to account for the heterogeneity in the effect sizes. A novel method for constructing confidence intervals for the amount of heterogeneity in the effect sizes is proposed that guarantees nominal coverage probabilities even in small samples when model assumptions are satisfied. A variety of existing approaches for constructing such confidence intervals are summarized and the various methods are applied to an example to illustrate their use. A simulation study reveals that the newly proposed method yields the most accurate coverage probabilities under conditions more analogous to practice, where assumptions about normally distributed effect size estimates and known sampling variances only hold asymptotically.     

Green tea and gastric cancer risk: meta-analysis of epidemiologic studies

OBJECTIVE: To evaluate the association between green tea consumption and the risk of gastric cancer. METHODS: Electronic search of the Cochrane Library, MEDLINE, EMBASE and Chinese Bio-medicine Database, which have articles published between (1966 and 2006), was conducted to select studies for this meta-analysis. RESULTS: This meta-analysis included 14 epidemiologic studies, with a total number of 6123 gastric cancer cases and 134006 controls. The combined results based on all studies showed that green tea consumption was not associated with the risk of gastric cancer [odds ratio (OR)=0.98, 95% confidence interval (CI)=0.77-1.24]. The summary OR from all population-based case-control studies showed a minor inverse association between green tea consumption and risk of gastric cancer (OR=0.68, 95% CI=0.49-0.92), while no associations were noted from hospital-based case-control studies (OR=1.12, 95% CI=0.70-1.77) and cohort studies (OR=1.56, 95% CI=0.93-2.60). No associations were noted both in males (OR=1.10, 95% CI=0.76-1.60) and females (OR=0.99, 95% CI=0.64-1.51). The summary OR from seven studies suggest that the highest consumption level of green tea was more than 5 cups per day and no associations were noted (OR=0.99, 95% CI=0.78-1.27). CONCLUSIONS: The results of this meta-analysis indicated that there is no clear epidemiological evidence to support the suggestion that green tea plays a role in the prevention of gastric cancer.     

Multivariate tests of means in independent groups designs. Effects of covariance heterogeneity and nonnormality

Health evaluation research often employs multivariate designs in which data on several outcome variables are obtained for independent groups of subjects. This article examines statistical procedures for testing hypotheses of multivariate mean equality in two-group designs. The conventional test for multivariate means, Hotelling's T2, rests on certain assumptions about the distribution of the data and the population variances and covariances. When these assumptions are violated, which is often the case in applied health research, T2 will result in invalid conclusions about the null hypothesis. This article describes parametric procedures that are robust, or insensitive, to assumption violations. A numeric example illustrates the statistical concepts that are presented and a computer program to implement these robust solutions is introduced.     

Cancer risk in asphalt workers and roofers: Review and meta-analysis of epidemiologic studies

Twenty epidemiologic studies have described cancer risk in asphalt workers and roofers in various countries. A current concern for these workers is the potential carcinogenicity posed by inhalation of bitumen fumes or dermal exposure to bitumens. Bitumens are chemically different from many carcinogenic coal-tar based materials. Both have been employed in road paving and waterproofing. We examined and combined the results of the epidemiologic studies conducted on asphalt workers and roofers. We examined the cancer risk separately in three broad job categories: 1) roofers (exposed to bitumen fumes and previously often to coal-tar fumes); 2) highway maintenance workers (HMWs) and road pavers (exposed to bitumen fumes as well as possibly coal-tar fumes previously); and 3) miscellaneous and unspecified bitumen/asphalt workers. In roofers, an increased risk was suggested for cancers of the lung (aggregated relative risk 1.8, 95% confidence interval 1.5–2.1), stomach (1.7, 1.1–2.5), nonmelanoma skin (4.0, 0.8–12), and leukemia (1.7, 0.9–2.9). Some of the excesses may be attributable to polycyclic aromatic hydrocarbons (PAH) from coal-tar products. The aggregated relative risks in road pavers and HMWs were consistently lower than in roofers for cancers of the lung (0.9, 0.8–1.0), stomach (1.1, 0.8–1.5), bladder (1.2, 0.7–1.8), skin (2.2, 1.2–3.7), and leukemias (1.3, 0.9–1.8). Their risk of skin cancer was significantly increased, based on one study. Miscellaneous and unspecified workers had a significant excess (1.5, 1.2–1.8) of lung cancer. The data were poorly focused to address the carcinogenicity of bitumen fumes, as contrasted with tar-derived exposures. For the prospect of shedding more light on the bitumen-cancer controversy, the feasibility of a powerful multicenter cohort is currently being studied by the International Agency for Research on Cancer (IARC).     

Incorporating external evidence on between-trial heterogeneity in network meta-analysis

In a network meta-analysis, between-study heterogeneity variances are often very imprecisely estimated because data are sparse, so standard errors of treatment differences can be highly unstable. External evidence can provide informative prior distributions for heterogeneity and, hence, improve inferences. We explore approaches for specifying informative priors for multiple heterogeneity variances in a network meta-analysis. First, we assume equal heterogeneity variances across all pairwise intervention comparisons (approach 1); incorporating an informative prior for the common variance is then straightforward. Models allowing unequal heterogeneity variances are more realistic; however, care must be taken to ensure implied variance-covariance matrices remain valid. We consider three strategies for specifying informative priors for multiple unequal heterogeneity variances. Initially, we choose different informative priors according to intervention comparison type and assume heterogeneity to be proportional across comparison types and equal within comparison type (approach 2). Next, we allow all heterogeneity variances in the network to differ, while specifying a common informative prior for each. We explore two different approaches to this: placing priors on variances and correlations separately (approach 3) or using an informative inverse Wishart distribution (approach 4). Our methods are exemplified through application to two network metaanalyses. Appropriate informative priors are obtained from previously published evidence-based distributions for heterogeneity. Relevant prior information on between-study heterogeneity can be incorporated into network meta-analyses, without needing to assume equal heterogeneity across treatment comparisons. The approaches proposed will be beneficial in sparse data sets and provide more appropriate intervals for treatment differences than those based on imprecise heterogeneity estimates.     

Investigating underlying risk as a source of heterogeneity in meta-analysis

In a meta-analysis of clinical trials, an important issue is whether the treatment benefit varies according to the underlying risk of the patients in the different trials. The usual naive analyses employed to investigate this question use either the observed risk of events in the control groups, or the average risk in the control and treatment groups, as a measure of underlying risk. These analyses are flawed and can produce seriously misleading results. We show how their biases depend on three components of variability, the within-trial and between-trial variances of the control group risks, and the between-trial variance of the treatment effects. We propose a Bayesian solution to the problem which can be carried out using the BUGS implementation of Gibbs sampling. The analysis is illustrated for a meta-analysis of bleeding and mortality data in trials of sclerotherapy for patients with cirrhosis, and the results contrasted with those from the naive approaches. Comparisons with other methods recently proposed for this problem are also made. We conclude that the Bayesian solution presented in this paper is not only more appropriate than other proposed methods, but is also sufficiently easy to implement that it can be used by applied researchers undertaking meta-analyses. © 1997 John Wiley & Sons, Ltd.