Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research

Biases in systematic reviews and meta-analyses may be examined in 'meta-epidemiological' studies, in which the influence of trial characteristics such as measures of study quality on treatment effect estimates is explored. Published studies to date have analysed data from collections of meta-analyses with binary outcomes, using logistic regression models that assume that there is no between- or within-meta-analysis heterogeneity. Using data from a study of publication bias (39 meta-analyses, 394 published and 88 unpublished trials) and language bias (29 meta-analyses, 297 English language trials and 52 non-English language trials), we compare results from logistic regression models, with and without robust standard errors to allow for clustering on meta-analysis, with results using a 'meta-meta-analytic' approach that can allow for between- and within-meta-analysis heterogeneity. We also consider how to allow for the confounding effects of different trial characteristics. We show that both within- and between meta-analysis heterogeneity may be of importance in the analysis of meta-epidemiological studies, and that confounding exists between the effects of publication status and trial quality.     

A comparison of three different models for estimating relative risk in meta-analysis of clinical trials under unobserved heterogeneity

We focus on the comparison of three statistical models used to estimate the treatment effect in meta-analysis when individually pooled data are available. The models are two conventional models, namely a multi-level and a model based upon an approximate likelihood, and a newly developed model, the profile likelihood model which might be viewed as an extension of the Mantel-Haenszel approach. To exemplify these methods, we use results from a meta-analysis of 22 trials to prevent respiratory tract infections. We show that by using the multi-level approach, in the case of baseline heterogeneity, the number of clusters or components is considerably over-estimated. The approximate and profile likelihood method showed nearly the same pattern for the treatment effect distribution. To provide more evidence two simulation studies are accomplished. The profile likelihood can be considered as a clear alternative to the approximate likelihood model. In the case of strong baseline heterogeneity, the profile likelihood method shows superior behaviour when compared with the multi-level model.     

Meta-analysis of continuous outcomes combining individual patient data and aggregate data

Meta-analysis of individual patient data (IPD) is the gold-standard for synthesizing evidence across clinical studies. However, for some studies IPD may not be available and only aggregate data (AD), such as a treatment effect estimate and its standard error, may be obtained. In this situation, methods for combining IPD and AD are important to utilize all the available evidence. In this paper, we develop and assess a range of statistical methods for combining IPD and AD in meta-analysis of continuous outcomes from randomized controlled trials.The methods take either a one-step or a two-step approach. The latter is simple, with IPD reduced to AD so that standard AD meta-analysis techniques can be employed. The one-step approach is more complex but offers a flexible framework to include both patient-level and trial-level parameters. It uses a dummy variable to distinguish IPD trials from AD trials and to constrain which parameters the AD trials estimate. We show that this is important when assessing how patient-level covariates modify treatment effect, as aggregate-level relationships across trials are subject to ecological bias and confounding. We thus develop models to separate within-trial and across-trials treatment-covariate interactions; this ensures that only IPD trials estimate the former, whilst both IPD and AD trials estimate the latter in addition to the pooled treatment effect and any between-study heterogeneity. Extension to multiple correlated outcomes is also considered. Ten IPD trials in hypertension, with blood pressure the continuous outcome of interest, are used to assess the models and identify the benefits of utilizing AD alongside IPD.     

A generalized weighting regression-derived meta-analysis estimator robust to small-study effects and heterogeneity

Heterogeneity and small-study effects are major concerns for the validity of meta-analysis. Although random effects meta-analysis provides a partial solution to heterogeneity, neither takes into account the presence of small-study effects, although they can rarely be ruled out with certainty. In this paper, we facilitate a better understanding of the properties of a recently described regression-based approach to deriving a meta-analysis estimator robust to small-study effects and unexplainable heterogeneity. The weightings of studies in the meta-analysis are derived algebraically for the regression model and compared with the weightings allocated to studies by fixed and random effects models. These weightings are compared in case studies with and without small-study effects. The presence of small-study effects causes pooled estimates from fixed and random effects meta-analyses to differ, potentially markedly, as a result of the different weights allocated to individual studies. Because random effects meta-analysis gives more weight to smaller studies, it becomes more vulnerable to the small-study effects. The regression approach gives heavier weight to the larger studies than either the fixed or random effects models, leading to its dominance in the estimated pooled effect. The weighting properties of the proposed regression-derived meta-analysis estimator are presented and compared with those of the standard meta-analytic estimators. We propose that there is much to recommend the routine use of this model as a reliable way to derive a pooled meta-analysis estimate that is robust to potential small-study effects, while still accommodating heterogeneity, even though uncertainty will often be considerably larger than for standard estimators.     

Meta-analyses in systematic reviews of randomized controlled trials in perinatal medicine: comparison of fixed and random effects models.

There is a need for empirical work comparing the random effects model with the fixed effects model in the calculation of a pooled relative risk in the meta-analysis in systematic reviews of randomized controlled trials. Such comparisons are particularly important when trial results are heterogeneous. We considered 84 independent meta-analyses in which each trial included a set of different women/newborns. These meta-analyses were included in systematic reviews published in the Cochrane Library's pregnancy and childbirth module. Twenty-one of these 84 meta-analyses demonstrated statistical heterogeneity at p<0.10. The random effects model estimates showed wider confidence intervals, particularly in those meta-analyses showing heterogeneity in the trial results. The summary relative risk for the random effects model tended to show a larger protective treatment effect than the fixed effects model in the heterogeneous meta-analyses. In this set of meta-analyses, statistical evaluation of publication bias cannot be shown to account for heterogeneity. Our empirical conclusion is that there may be opposing effects if the random effects model is used in the meta-analysis of clinical trials showing heterogeneity in the results: stronger treatment effects reflected in the summary relative risk, but wider confidence intervals about this summary measure.     

Investigating heterogeneity in an individual patient data meta-analysis of time to event outcomes

Differences across studies in terms of design features and methodology, clinical procedures, and patient characteristics, are factors that can contribute to variability in the treatment effect between studies in a meta-analysis (statistical heterogeneity). Regression modelling can be used to examine relationships between treatment effect and covariates with the aim of explaining the variability in terms of clinical, methodological, or other factors. Such an investigation can be undertaken using aggregate data or individual patient data. An aggregate data approach can be problematic as sufficient data are rarely available and translating aggregate effects to individual patients can often be misleading. An individual patient data approach, although usually more resource demanding, allows a more thorough investigation of potential sources of heterogeneity and enables a fuller analysis of time to event outcomes in meta-analysis. Hierarchical Cox regression models are used to identify and explore the evidence for heterogeneity in meta-analysis and examine the relationship between covariates and censored failure time data in this context. Alternative formulations of the model are possible and illustrated using individual patient data from a meta-analysis of five randomized controlled trials which compare two drugs for the treatment of epilepsy. The models are further applied to simulated data examples in which the degree of heterogeneity and magnitude of treatment effect are varied. The behaviour of each model in each situation is explored and compared.     

Quantifying heterogeneity in a meta-analysis

The extent of heterogeneity in a meta-analysis partly determines the difficulty in drawing overall conclusions. This extent may be measured by estimating a between-study variance, but interpretation is then specific to a particular treatment effect metric. A test for the existence of heterogeneity exists, but depends on the number of studies in the meta-analysis. We develop measures of the impact of heterogeneity on a meta-analysis, from mathematical criteria, that are independent of the number of studies and the treatment effect metric. We derive and propose three suitable statistics: H is the square root of the chi2 heterogeneity statistic divided by its degrees of freedom; R is the ratio of the standard error of the underlying mean from a random effects meta-analysis to the standard error of a fixed effect meta-analytic estimate, and I2 is a transformation of (H) that describes the proportion of total variation in study estimates that is due to heterogeneity. We discuss interpretation, interval estimates and other properties of these measures and examine them in five example data sets showing different amounts of heterogeneity. We conclude that H and I2, which can usually be calculated for published meta-analyses, are particularly useful summaries of the impact of heterogeneity. One or both should be presented in published meta-analyses in preference to the test for heterogeneity.     

Investigation of one-stage meta-analysis methods for joint longitudinal and time-to-event data through simulation and real data application

Background: Joint modeling of longitudinal and time-to-event data is often advantageous over separate longitudinal or time-to-event analyses as it can account for study dropout, error in longitudinally measured covariates, and correlation between longitudinal and time-to-event outcomes. The current literature on joint modeling focuses mainly on the analysis of single studies with a lack of methods available for the meta-analysis of joint data from multiple studies. Methods: We investigate a variety of one-stage methods for the meta-analysis of joint longitudinal and time-to-event outcome data. These methods are applied to the INDANA dataset to investigate longitudinally measured systolic blood pressure, with each of time to death, time to myocardial infarction, and time to stroke. Results are compared to separate longitudinal or time-to-event meta-analyses. A simulation study is conducted to contrast separate versus joint analyses over a range of scenarios. Results: The performance of the examined one-stage joint meta-analytic models varied. Models that accounted for between study heterogeneity performed better than models that ignored it. Of the examined methods to account for between study heterogeneity, under the examined association structure, fixed effect approaches appeared preferable, whereas methods involving a baseline hazard stratified by study were least time intensive. Conclusions: One-stage joint meta-analytic models that accounted for between study heterogeneity using a mix of fixed effects or a stratified baseline hazard were reliable; however, models examined that included study level random effects in the association structure were less reliable.     

Traditional reviews, meta-analyses and pooled analyses in epidemiology

BACKGROUND: The use of review articles and meta-analysis has become an important part of epidemiological research, mainly for reconciling previously conducted studies that have inconsistent results. Numerous methodologic issues particularly with respect to biases and the use of meta-analysis are still controversial. METHODS: Four methods summarizing data from epidemiological studies are described. The rationale for meta-analysis and the statistical methods used are outlined. The strengths and limitations of these methods are compared particularly with respect to their ability to investigate heterogeneity between studies and to provide quantitative risk estimation. RESULTS: Meta-analyses from published data are in general insufficient to calculate a pooled estimate since published estimates are based on heterogeneous populations, different study designs and mainly different statistical models. More reliable results can be expected if individual data are available for a pooled analysis, although some heterogeneity still remains. Large prospective planned meta-analysis of multicentre studies would be preferable to investigate small risk factors, however this type of meta-analysis is expensive and time-consuming. CONCLUSION: For a full assessment of risk factors with a high prevalence in the general population, pooling of data will become increasingly important. Future research needs to focus on the deficiencies of review methods, in particular, the errors and biases that can be produced when studies are combined that have used different designs, methods and analytic models.     

Predictive modeling and heterogeneity of baseline risk in meta-analysis of individual patient data

We developed and evaluated methods for the analysis and interpretation of the baseline risk heterogeneity in meta-analysis of individual patient data (MIPD) based on information on predictive factors. We used data from a typical MIPD of eight clinical trials (1792 patients, 2947 years of follow-up) on the efficacy of high-dose acyclovir in human immunodeficiency virus infection. Cox models with four predictive factors (age, disease state, CD4 cell count and hemoglobin levels) were used to estimate predicted individual hazards both for single trials and for various MIPD modeling methods (simple pooling, adjusted for study, stratified per study, fixed and random effects for predictors). For each study and for each method of MIPD synthesis, we estimated the odds ratio for death in the upper versus the lower quartile of predicted risk (Extreme Quartile Odds Ratio, EQuOR) and the respective rate ratio (Extreme Quartile Rate Ratio, EQuRR). Only the CD4 cell count showed a significantly heterogeneous predictive effect across the eight studies (P =.024). The EQuOR of single studies ranged from 3.5 (little heterogeneity) to 24 (intermediate heterogeneity), substantially lower than the EQuOR of the MIPD (167 to 275, depending on the model used). The EQuRR values ranged from 3.5 to 77 for single studies and from 77 to 116 for the various MIPD models. Predictive modeling can be a major strength of MIPD, when performed and interpreted with standardized approaches. All models consistently show that MIPD may be a study design with extreme heterogeneity of patient baseline risk.     

BORROWING STRENGTH FROM EXTERNAL TRIALS IN A META-ANALYSIS

There exists a variety of situations in which a random effects meta-analysis might be undertaken using a small number of clinical trials. A problem associated with small meta-analyses is estimating the heterogeneity between trials. To overcome this problem, information from other related studies may be incorporated into the meta-analysis. A Bayesian approach to this problem is presented using data from previous meta-analyses in the same therapeutic area to formulate a prior distribution for the heterogeneity. The treatment difference parameters are given non-informative priors. Further, related trials which compare one or other of the treatments of interest with a common third treatment are included in the model to improve inference on both the heterogeneity and the treatment difference. Two approaches to estimating relative efficacy are considered, namely a general parametric approach and a method explicit to binary data. The methodology is illustrated using data from 26 clinical trials which investigate the prevention of cirrhosis using beta-blockers and sclerotherapy. Both sources of external information lead to more precise posterior distributions for all parameters, in particular that representing heterogeneity.     

Investigating trial and treatment heterogeneity in an individual patient data meta-analysis of survival data by means of the penalized maximum likelihood approach

In a meta-analysis combining survival data from different clinical trials, an important issue is the possible heterogeneity between trials. Such intertrial variation can not only be explained by heterogeneity of treatment effects across trials but also by heterogeneity of their baseline risk. In addition, one might examine the relationship between magnitude of the treatment effect and the underlying risk of the patients in the different trials. Such a scenario can be accounted for by using additive random effects in the Cox model, with a random trial effect and a random treatment-by-trial interaction. We propose to use this kind of model with a general correlation structure for the random effects and to estimate parameters and hazard function using a semi-parametric penalized marginal likelihood method (maximum penalized likelihood estimators). This approach gives smoothed estimates of the hazard function, which represents incidence in epidemiology. The idea for the approach in this paper comes from the study of heterogeneity in a large meta-analysis of randomized trials in patients with head and neck cancers (meta-analysis of chemotherapy in head and neck cancers) and the effect of adding chemotherapy to locoregional treatment. The simulation study and the application demonstrate that the proposed approach yields satisfactory results and they illustrate the need to use a flexible variance-covariance structure for the random effects.     

Association between the Phytochemical Index and Overweight/Obesity: A Meta-Analysis

Some studies suggest that a higher phytochemical index (PI) is associated with a lower risk of overweight/obesity. This meta-analysis is performed to summarize published studies on the relationship of PI and the risk of overweight/obesity. We searched on PubMed, Cochrane Library and Web of Science from the inception dates to February 2022. The random-effect model was used based on heterogeneity. Meta-regression was used to explore potential sources of between-study heterogeneity. Publication bias was evaluated using Begg’s and Egger’s tests. The dose–response relationship was assessed using a restricted cubic spline model. Nine studies were included in the meta-analysis, with a total of 100,753 participants. The meta-analysis showed that the phytochemical index was associated with a decreased risk of overweight/obesity. The pooled OR (95% CI) was 0.81 (0.74–0.90). The findings from dose–response analysis showed a nonlinear association between the phytochemical index and the risk of overweight/obesity. The results of the meta-regression showed that gender and area were significant covariates influencing the heterogeneity between studies. There was no publication bias in the meta-analysis of this study. In conclusion, although this meta-analysis indicates that a high phytochemical index is associated with a reduced risk of overweight/obesity, all the studies included in this meta-analysis were cross-sectional studies with high heterogeneity. As such, more data from randomized controlled trials are required to confirm the efficacy of PI in evaluating the risk of overweight/obesity.     

Combination of direct and indirect evidence in mixed treatment comparisons

Mixed treatment comparison (MTC) meta-analysis is a generalization of standard pairwise meta-analysis for A vs B trials, to data structures that include, for example, A vs B, B vs C, and A vs C trials. There are two roles for MTC: one is to strengthen inference concerning the relative efficacy of two treatments, by including both 'direct' and 'indirect' comparisons. The other is to facilitate simultaneous inference regarding all treatments, in order for example to select the best treatment. In this paper, we present a range of Bayesian hierarchical models using the Markov chain Monte Carlo software WinBUGS. These are multivariate random effects models that allow for variation in true treatment effects across trials. We consider models where the between-trials variance is homogeneous across treatment comparisons as well as heterogeneous variance models. We also compare models with fixed (unconstrained) baseline study effects with models with random baselines drawn from a common distribution. These models are applied to an illustrative data set and posterior parameter distributions are compared. We discuss model critique and model selection, illustrating the role of Bayesian deviance analysis, and node-based model criticism. The assumptions underlying the MTC models and their parameterization are also discussed.     

Metabolic syndrome and all-cause mortality: a meta-analysis of prospective cohort studies

To synthesize the available data on the association between metabolic syndrome and all-cause mortality, we conducted a meta-analysis of prospective cohort studies. We performed a literature search using Medline, EMBASE and Cochrane Library from 2001 to December 2009, with no restrictions. We included studies if they were prospective, had an assessment of metabolic syndrome at baseline and risk of all-cause mortality. We recorded several characteristics for each study. We extracted relative risks (RR) and 95% confidence intervals (CI) and pooled them using fixed or random effects models. We performed sensitivity analysis, and assessed heterogeneity and publication bias. A total of 21 studies including 372,411 participants were included in our meta-analysis. 18,556 deaths from any cause occurred during a mean follow-up of 11.5 years. Individuals with the metabolic syndrome, compared to those without, had an increased mortality from all causes (pooled RR 1.46; 95% CI 1.35–1.57). The RR of all-cause mortality associated with metabolic syndrome was higher in studies using the National Cholesterol Education Program Adult Treatment Panel (NCEP) than the revised NCEP criteria (RR: 1.45 vs. 1.25; P = 0.0002). Metabolic syndrome is an important risk factor for all-cause mortality. The diagnosis and treatment of the underlying risk factors for the metabolic syndrome should be an important strategy for the reduction of all-cause mortality associated with metabolic syndrome in the general population.     

Meta-analysis of multiple outcomes by regression with random effects

Earlier work showed how to perform fixed-effects meta-analysis of studies or trials when each provides results on more than one outcome per patient and these multiple outcomes are correlated. That fixed-effects generalized-least-squares approach analyzes the multiple outcomes jointly within a single model, and it can include covariates, such as duration of therapy or quality of trial, that may explain observed heterogeneity of results among the trials. Sometimes the covariates explain all the heterogeneity, and the fixed-effects regression model is appropriate. However, unexplained heterogeneity may often remain, even after taking into account known or suspected covariates. Because fixed-effects models do not make allowance for this remaining unexplained heterogeneity, the potential exists for bias in estimated coefficients, standard errors and p-values. We propose two random-effects approaches for the regression meta-analysis of multiple correlated outcomes. We compare their use with fixed-effects models and with separate-outcomes models in a meta-analysis of periodontal clinical trials. A simulation study shows the advantages of the random-effects approach. These methods also facilitate meta-analysis of trials that compare more than two treatments. © 1998 John Wiley & Sons, Ltd.     

Treatments for postherpetic neuralgia--a systematic review of randomized controlled trials

BACKGROUND: A number of different therapies have been used for postherpeticneuralgia. We decided to conduct a systematic review of existingrandomized controlled trials. OBJECTIVE: To determine the efficacy of available therapies for relievingthe pain of established postherpetic neuralgia. METHODS: We performed a systematic review, including meta-analysis, ofexisting randomized controlled trials. Eleven published trialsand one unpublished trial were identified which met the inclusioncriteria and were included in the current review. RESULTS: Pooled analysis of the effect of tricyclic antidepressants demonstratestatistically significant pain relief (OR 0.15, Cl 0.08–0.27).Pooling of the results of the three trials comparing the effectsof capsaicin and placebo could not be done due to heterogeneity.This heterogeneity was mainly attributable to an unpublishedtrial which differed in terms of the dose and duration of treatment.When this study was omitted, no heterogeneity was found andthe pooled analysis revealed a statistically significant benefit(OR 0.29, 95% Cl 0.16–0.54). However, problems with blindingin patients using capsaicin may have accounted for the positiveeffect. One small study of vincristine iontophoresis comparedto placebo also yielded a favourable result (OR 0.05, 95% 0.01–0.26).Other treatment evaluated include lorazepam, acyclovir, topicalbenzydamine, and acupuncture. We found no evidence that theseare effective in relieving pain associated with postherpeticneuralgia. CONCLUSION: Based on evidence from randomized trials, tricyclic anti-depressantsappear to be the only agents of proven benefit for establishedpostherpetic neuralgia. Keywords. Postherpetic neuralgia, treatment, systematic review, meta-analysis, randomized controlled trials.     

Meta-analysis of method comparison studies

Methods for the meta-analysis of results from randomized controlled trials are well established. However, there are currently no methods for the meta-analysis of method comparison studies. Here the combination of results from studies comparing two methods of measurement on the same unit of observation is required. We compare standard methods for the pooling of k samples from the same Normal population to those for pooling parameter estimates, in order to estimate the pooled mean difference and 95 per cent limits of agreement. Methods for investigating heterogeneity across studies and for calculating random effects estimates are proposed. We postulate that for published studies either the estimated mean or variance of the difference between measurements will tend to be smaller than for unpublished studies and investigate the evidence for the existence of such publication bias. The methods are illustrated with an example evaluating the accuracy of temperature measured at the axilla compared to the rectum in children.     

Imputing variance estimates do not alter the conclusions of a meta-analysis with continuous outcomes: a case study of changes in renal function after living kidney donation

OBJECTIVE: To assess how different imputation methods used to account for missing variance data in primary studies influence tests of heterogeneity and pooled results from a meta-analysis with continuous outcomes. STUDY DESIGN AND SETTING: Point and variance estimates for changes in serum creatinine, glomerular filtration rate, systolic blood pressure, and diastolic blood pressure were variably reported among 48 primary longitudinal studies of living kidney donors (71%-78% of point estimates were reported, 8%-13% of variance data were reported). We compared the results of meta-analysis, which either were restricted to available data or used four methods to impute missing variance data. These methods used reported P-values, reported nonparametric summaries, results from other similar studies using multiple imputation, or results from estimated correlation coefficients. RESULTS: Significant heterogeneity was present in all four outcomes regardless of the imputation methods applied. The random effects point estimates and 95% confidence intervals varied little across imputation methods, and the differences were not clinically significant. CONCLUSIONS: Different methods to impute the variance data in the primary studies did not alter the conclusions from this meta-analysis of continuous outcomes. Such reproducibility increases confidence in the results. However, as with most meta-analyses, there was no gold standard of truth, and results must be interpreted judiciously. The generalization of these findings to other meta-analyses, which differ in outcomes, missing data, or between-study heterogeneity, requires further consideration.     

The effects of DRGs-based payment compared with cost-based payment on inpatient healthcare utilization: A systematic review and meta-analysis

Diagnosis related groups (DRGs)-based payment is increasingly used worldwide to control hospital costs instead of pre-existing cost-based payment, but the results of evaluations vary. A systematic analysis of the effects of DRGs-based payment is needed. This study aims to conduct a systematic review and meta-analysis to compare the effects of DRGs-based payment and cost-based payment on inpatient health utilization in terms of length of stay (LOS), total inpatient spending per admission and readmission rates. We included studies undertaken with designs approved by the Cochrane Effective Practice and Organisation of Care that reported associations between DRGs-based payment and one or more inpatient healthcare utilization outcomes. After a systematic search of eight electronic databases through October 2018, 18 studies were identified and included in the review. We extracted data and conducted quality assessment, systematic synthesis and meta-analyses on the included studies. Random-effects models were used to handle substantial heterogeneity between studies. Meta-analysis showed that DRGs-based payment was associated with lower LOS (pooled effect: -8.07 % [95 %CI -13.05 to -3.10], p = 0.001), and higher readmission rates (pooled effect: 1.36 % [95 %CI 0.45–2.27], p = 0.003). This meta-analysis revealed that DRGs-based payment may have cost-saving implications by lowering LOS, whereas hardly reduce the readmission rates. Policy-makers considering adopting DRGs-based payment should pay more attention to the hospital readmission rates compared with cost-based payment.