A refined method for multivariate meta‐analysis and meta‐regression

Making inferences about the average treatment effect using the random effects model for meta‐analysis is problematic in the common situation where there is a small number of studies. This is because estimates of the between‐study variance are not precise enough to accurately apply the conventional methods for testing and deriving a confidence interval for the average effect. We have found that a refined method for univariate meta‐analysis, which applies a scaling factor to the estimated effects’ standard error, provides more accurate inference. We explain how to extend this method to the multivariate scenario and show that our proposal for refined multivariate meta‐analysis and meta‐regression can provide more accurate inferences than the more conventional approach. We explain how our proposed approach can be implemented using standard output from multivariate meta‐analysis software packages and apply our methodology to two real examples. © 2013 The Authors. Statistics in Medicine published by John Wiley & Sons, Ltd.     

Multivariate meta‐analysis for non‐linear and other multi‐parameter associations

In this paper, we formalize the application of multivariate meta‐analysis and meta‐regression to synthesize estimates of multi‐parameter associations obtained from different studies. This modelling approach extends the standard two‐stage analysis used to combine results across different sub‐groups or populations. The most straightforward application is for the meta‐analysis of non‐linear relationships, described for example by regression coefficients of splines or other functions, but the methodology easily generalizes to any setting where complex associations are described by multiple correlated parameters. The modelling framework of multivariate meta‐analysis is implemented in the package mvmeta within the statistical environment R . As an illustrative example, we propose a two‐stage analysis for investigating the non‐linear exposure–response relationship between temperature and non‐accidental mortality using time‐series data from multiple cities. Multivariate meta‐analysis represents a useful analytical tool for studying complex associations through a two‐stage procedure. Copyright © 2012 John Wiley & Sons, Ltd.     

Multivariate network meta‐analysis to mitigate the effects of outcome reporting bias

Outcome reporting bias (ORB) is recognized as a threat to the validity of both pairwise and network meta‐analysis (NMA). In recent years, multivariate meta‐analytic methods have been proposed to reduce the impact of ORB in the pairwise setting. These methods have shown that multivariate meta‐analysis can reduce bias and increase efficiency of pooled effect sizes. However, it is unknown whether multivariate NMA (MNMA) can similarly reduce the impact of ORB. Additionally, it is quite challenging to implement MNMA due to the fact that correlation between treatments and outcomes must be modeled; thus, the dimension of the covariance matrix and number of components to estimate grows quickly with the number of treatments and number of outcomes. To determine whether MNMA can reduce the effects of ORB on pooled treatment effect sizes, we present an extensive simulation study of Bayesian MNMA. Via simulation studies, we show that MNMA reduces the bias of pooled effect sizes under a variety of outcome missingness scenarios, including missing at random and missing not at random. Further, MNMA improves the precision of estimates, producing narrower credible intervals. We demonstrate the applicability of the approach via application of MNMA to a multi‐treatment systematic review of randomized controlled trials of anti‐depressants for the treatment of depression in older adults.     

A tutorial on Bayesian bivariate meta‐analysis of mixed binary‐continuous outcomes with missing treatment effects

Bivariate random‐effects meta‐analysis (BVMA) is a method of data synthesis that accounts for treatment effects measured on two outcomes. BVMA gives more precise estimates of the population mean and predicted values than two univariate random‐effects meta‐analyses (UVMAs). BVMA also addresses bias from incomplete reporting of outcomes. A few tutorials have covered technical details of BVMA of categorical or continuous outcomes. Limited guidance is available on how to analyze datasets that include trials with mixed continuous‐binary outcomes where treatment effects on one outcome or the other are not reported. Given the advantages of Bayesian BVMA for handling missing outcomes, we present a tutorial for Bayesian BVMA of incompletely reported treatment effects on mixed bivariate outcomes. This step‐by‐step approach can serve as a model for our intended audience, the methodologist familiar with Bayesian meta‐analysis, looking for practical advice on fitting bivariate models. To facilitate application of the proposed methods, we include our WinBUGS code. As an example, we use aggregate‐level data from published trials to demonstrate the estimation of the effects of vitamin K and bisphosphonates on two correlated bone outcomes, fracture, and bone mineral density. We present datasets where reporting of the pairs of treatment effects on both outcomes was ‘partially’ complete (i.e., pairs completely reported in some trials), and we outline steps for modeling the incompletely reported data. To assess what is gained from the additional work required by BVMA, we compare the resulting estimates to those from separate UVMAs. We discuss methodological findings and make four recommendations. Copyright © 2015 John Wiley & Sons, Ltd.     

Multivariate meta‐analysis: A method to summarize non‐linear associations

Multivariate meta‐analysis represents a promising statistical tool in several research areas. Here, we provide a brief overview of the application of this methodology to combining complex multi‐parameterized relationships, such as non‐linear or delayed associations, in multi‐site studies. The discussion focuses on the advantages over simpler univariate methods, estimation and computational issues and directions for further research. Copyright © 2011 John Wiley & Sons, Ltd.     

Multivariate meta‐analysis: Potential and promise

The multivariate random effects model is a generalization of the standard univariate model. Multivariate meta‐analysis is becoming more commonly used and the techniques and related computer software, although continually under development, are now in place. In order to raise awareness of the multivariate methods, and discuss their advantages and disadvantages, we organized a one day ‘Multivariate meta‐analysis’ event at the Royal Statistical Society. In addition to disseminating the most recent developments, we also received an abundance of comments, concerns, insights, critiques and encouragement. This article provides a balanced account of the day's discourse. By giving others the opportunity to respond to our assessment, we hope to ensure that the various view points and opinions are aired before multivariate meta‐analysis simply becomes another widely used de facto method without any proper consideration of it by the medical statistics community. We describe the areas of application that multivariate meta‐analysis has found, the methods available, the difficulties typically encountered and the arguments for and against the multivariate methods, using four representative but contrasting examples. We conclude that the multivariate methods can be useful, and in particular can provide estimates with better statistical properties, but also that these benefits come at the price of making more assumptions which do not result in better inference in every case. Although there is evidence that multivariate meta‐analysis has considerable potential, it must be even more carefully applied than its univariate counterpart in practice. Copyright © 2011 John Wiley & Sons, Ltd.     

Avoiding zero between‐study variance estimates in random‐effects meta‐analysis

Fixed‐effects meta‐analysis has been criticized because the assumption of homogeneity is often unrealistic and can result in underestimation of parameter uncertainty. Random‐effects meta‐analysis and meta‐regression are therefore typically used to accommodate explained and unexplained between‐study variability. However, it is not unusual to obtain a boundary estimate of zero for the (residual) between‐study standard deviation, resulting in fixed‐effects estimates of the other parameters and their standard errors. To avoid such boundary estimates, we suggest using Bayes modal (BM) estimation with a gamma prior on the between‐study standard deviation. When no prior information is available regarding the magnitude of the between‐study standard deviation, a weakly informative default prior can be used (with shape parameter 2 and rate parameter close to 0) that produces positive estimates but does not overrule the data, leading to only a small decrease in the log likelihood from its maximum. We review the most commonly used estimation methods for meta‐analysis and meta‐regression including classical and Bayesian methods and apply these methods, as well as our BM estimator, to real datasets. We then perform simulations to compare BM estimation with the other methods and find that BM estimation performs well by (i) avoiding boundary estimates; (ii) having smaller root mean squared error for the between‐study standard deviation; and (iii) better coverage for the overall effects than the other methods when the true model has at least a small or moderate amount of unexplained heterogeneity. Copyright © 2013 John Wiley & Sons, Ltd.     

Addressing the estimation of standard errors in fixed effects meta‐analysis

Standard methods for fixed effects meta‐analysis assume that standard errors for study‐specific estimates are known, not estimated. While the impact of this simplifying assumption has been shown in a few special cases, its general impact is not well understood, nor are general‐purpose tools available for inference under more realistic assumptions. In this paper, we aim to elucidate the impact of using estimated standard errors in fixed effects meta‐analysis, showing why it does not go away in large samples and quantifying how badly miscalibrated standard inference will be if it is ignored. We also show the important role of a particular measure of heterogeneity in this miscalibration. These developments lead to confidence intervals for fixed effects meta‐analysis with improved performance for both location and scale parameters.     

Statistics for quantifying heterogeneity in univariate and bivariate meta‐analyses of binary data: The case of meta‐analyses of diagnostic accuracy

Heterogeneity in diagnostic meta‐analyses is common because of the observational nature of diagnostic studies and the lack of standardization in the positivity criterion (cut‐off value) for some tests. So far the unexplained heterogeneity across studies has been quantified by either using the I² statistic for a single parameter (i.e. either the sensitivity or the specificity) or visually examining the data in a receiver‐operating characteristic space. In this paper, we derive improved I² statistics measuring heterogeneity for dichotomous outcomes, with a focus on diagnostic tests. We show that the currently used estimate of the ‘typical’ within‐study variance proposed by Higgins and Thompson is not able to properly account for the variability of the within‐study variance across studies for dichotomous variables. Therefore, when the between‐study variance is large, the ‘typical’ within‐study variance underestimates the expected within‐study variance, and the corresponding I² is overestimated. We propose to use the expected value of the within‐study variation in the construction of I² in cases of univariate and bivariate diagnostic meta‐analyses. For bivariate diagnostic meta‐analyses, we derive a bivariate version of I² that is able to account for the correlation between sensitivity and specificity. We illustrate the performance of these new estimators using simulated data as well as two real data sets. Copyright © 2014 John Wiley & Sons, Ltd.     

Extending DerSimonian and Laird's methodology to perform multivariate random effects meta‐analyses

Multivariate meta‐analysis is increasingly used in medical statistics. In the univariate setting, the non‐iterative method proposed by DerSimonian and Laird is a simple and now standard way of performing random effects meta‐analyses. We propose a natural and easily implemented multivariate extension of this procedure which is accessible to applied researchers and provides a much less computationally intensive alternative to existing methods. In a simulation study, the proposed procedure performs similarly in almost all ways to the more established iterative restricted maximum likelihood approach. The method is applied to some real data sets and an extension to multivariate meta‐regression is described. Copyright © 2009 John Wiley & Sons, Ltd.     

A comparison of seven random‐effects models for meta‐analyses that estimate the summary odds ratio

Comparative trials that report binary outcome data are commonly pooled in systematic reviews and meta‐analyses. This type of data can be presented as a series of 2‐by‐2 tables. The pooled odds ratio is often presented as the outcome of primary interest in the resulting meta‐analysis. We examine the use of 7 models for random‐effects meta‐analyses that have been proposed for this purpose. The first of these models is the conventional one that uses normal within‐study approximations and a 2‐stage approach. The other models are generalised linear mixed models that perform the analysis in 1 stage and have the potential to provide more accurate inference. We explore the implications of using these 7 models in the context of a Cochrane Review, and we also perform a simulation study. We conclude that generalised linear mixed models can result in better statistical inference than the conventional 2‐stage approach but also that this type of model presents issues and difficulties. These challenges include more demanding numerical methods and determining the best way to model study specific baseline risks. One possible approach for analysts is to specify a primary model prior to performing the systematic review but also to present the results using other models in a sensitivity analysis. Only one of the models that we investigate is found to perform poorly so that any of the other models could be considered for either the primary or the sensitivity analysis.     

A mixed effect model for bivariate meta‐analysis of diagnostic test accuracy studies using a copula representation of the random effects distribution

Diagnostic test accuracy studies typically report the number of true positives, false positives, true negatives and false negatives. There usually exists a negative association between the number of true positives and true negatives, because studies that adopt less stringent criterion for declaring a test positive invoke higher sensitivities and lower specificities. A generalized linear mixed model (GLMM) is currently recommended to synthesize diagnostic test accuracy studies. We propose a copula mixed model for bivariate meta‐analysis of diagnostic test accuracy studies. Our general model includes the GLMM as a special case and can also operate on the original scale of sensitivity and specificity. Summary receiver operating characteristic curves are deduced for the proposed model through quantile regression techniques and different characterizations of the bivariate random effects distribution. Our general methodology is demonstrated with an extensive simulation study and illustrated by re‐analysing the data of two published meta‐analyses. Our study suggests that there can be an improvement on GLMM in fit to data and makes the argument for moving to copula random effects models. Our modelling framework is implemented in the package CopulaREMADA within the open source statistical environment R . Copyright © 2015 John Wiley & Sons, Ltd.     

A new measure of between‐studies heterogeneity in meta‐analysis

Assessing the magnitude of heterogeneity in a meta‐analysis is important for determining the appropriateness of combining results. The most popular measure of heterogeneity, I², was derived under an assumption of homogeneity of the within‐study variances, which is almost never true, and the alternative estimator, , uses the harmonic mean to estimate the average of the within‐study variances, which may also lead to bias. This paper thus presents a new measure for quantifying the extent to which the variance of the pooled random‐effects estimator is due to between‐studies variation, , that overcomes the limitations of the previous approach. We show that this measure estimates the expected value of the proportion of total variance due to between‐studies variation and we present its point and interval estimators. The performance of all three heterogeneity measures is evaluated in an extensive simulation study. A negative bias for was observed when the number of studies was very small and became negligible as the number of studies increased, while and I² showed a tendency to overestimate the impact of heterogeneity. The coverage of confidence intervals based upon was good across different simulation scenarios but was substantially lower for and I², especially for high values of heterogeneity and when a large number of studies were included in the meta‐analysis. The proposed measure is implemented in a user‐friendly function available for routine use in r and sas . will be useful in quantifying the magnitude of heterogeneity in meta‐analysis and should supplement the p‐value for the test of heterogeneity obtained from the Q test. Copyright © 2016 John Wiley & Sons, Ltd.     

Meta‐analysis of studies with bivariate binary outcomes: a marginal beta‐binomial model approach

When conducting a meta‐analysis of studies with bivariate binary outcomes, challenges arise when the within‐study correlation and between‐study heterogeneity should be taken into account. In this paper, we propose a marginal beta‐binomial model for the meta‐analysis of studies with binary outcomes. This model is based on the composite likelihood approach and has several attractive features compared with the existing models such as bivariate generalized linear mixed model (Chu and Cole, 2006) and Sarmanov beta‐binomial model (Chen et al., 2012). The advantages of the proposed marginal model include modeling the probabilities in the original scale, not requiring any transformation of probabilities or any link function, having closed‐form expression of likelihood function, and no constraints on the correlation parameter. More importantly, because the marginal beta‐binomial model is only based on the marginal distributions, it does not suffer from potential misspecification of the joint distribution of bivariate study‐specific probabilities. Such misspecification is difficult to detect and can lead to biased inference using currents methods. We compare the performance of the marginal beta‐binomial model with the bivariate generalized linear mixed model and the Sarmanov beta‐binomial model by simulation studies. Interestingly, the results show that the marginal beta‐binomial model performs better than the Sarmanov beta‐binomial model, whether or not the true model is Sarmanov beta‐binomial, and the marginal beta‐binomial model is more robust than the bivariate generalized linear mixed model under model misspecifications. Two meta‐analyses of diagnostic accuracy studies and a meta‐analysis of case–control studies are conducted for illustration. Copyright © 2015 John Wiley & Sons, Ltd.     

Bartlett‐type corrections and bootstrap adjustments of likelihood‐based inference methods for network meta‐analysis

In network meta‐analyses that synthesize direct and indirect comparison evidence concerning multiple treatments, multivariate random effects models have been routinely used for addressing between‐studies heterogeneities. Although their standard inference methods depend on large sample approximations (eg, restricted maximum likelihood estimation) for the number of trials synthesized, the numbers of trials are often moderate or small. In these situations, standard estimators cannot be expected to behave in accordance with asymptotic theory; in particular, confidence intervals cannot be assumed to exhibit their nominal coverage probabilities (also, the type I error probabilities of the corresponding tests cannot be retained). The invalidity issue may seriously influence the overall conclusions of network meta‐analyses. In this article, we develop several improved inference methods for network meta‐analyses to resolve these problems. We first introduce 2 efficient likelihood‐based inference methods, the likelihood ratio test–based and efficient score test–based methods, in a general framework of network meta‐analysis. Then, to improve the small‐sample inferences, we developed improved higher‐order asymptotic methods using Bartlett‐type corrections and bootstrap adjustment methods. The proposed methods adopt Monte Carlo approaches using parametric bootstraps to effectively circumvent complicated analytical calculations of case‐by‐case analyses and to permit flexible application to various statistical models network meta‐analyses. These methods can also be straightforwardly applied to multivariate meta‐regression analyses and to tests for the evaluation of inconsistency. In numerical evaluations via simulations, the proposed methods generally performed well compared with the ordinary restricted maximum likelihood–based inference method. Applications to 2 network meta‐analysis datasets are provided.     

An alternative pseudolikelihood method for multivariate random‐effects meta‐analysis

Recently, multivariate random‐effects meta‐analysis models have received a great deal of attention, despite its greater complexity compared to univariate meta‐analyses. One of its advantages is its ability to account for the within‐study and between‐study correlations. However, the standard inference procedures, such as the maximum likelihood or maximum restricted likelihood inference, require the within‐study correlations, which are usually unavailable. In addition, the standard inference procedures suffer from the problem of singular estimated covariance matrix. In this paper, we propose a pseudolikelihood method to overcome the aforementioned problems. The pseudolikelihood method does not require within‐study correlations and is not prone to singular covariance matrix problem. In addition, it can properly estimate the covariance between pooled estimates for different outcomes, which enables valid inference on functions of pooled estimates, and can be applied to meta‐analysis where some studies have outcomes missing completely at random. Simulation studies show that the pseudolikelihood method provides unbiased estimates for functions of pooled estimates, well‐estimated standard errors, and confidence intervals with good coverage probability. Furthermore, the pseudolikelihood method is found to maintain high relative efficiency compared to that of the standard inferences with known within‐study correlations. We illustrate the proposed method through three meta‐analyses for comparison of prostate cancer treatment, for the association between paraoxonase 1 activities and coronary heart disease, and for the association between homocysteine level and coronary heart disease. © 2014 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

A re‐evaluation of the ‘quantile approximation method’ for random effects meta‐analysis

The quantile approximation method has recently been proposed as a simple method for deriving confidence intervals for the treatment effect in a random effects meta‐analysis. Although easily implemented, the quantiles used to construct intervals are derived from a single simulation study. Here it is shown that altering the study parameters, and in particular introducing changes to the distribution of the within‐study variances, can have a dramatic impact on the resulting quantiles. This is further illustrated analytically by examining the scenario where all trials are assumed to be the same size. A more cautious approach is therefore suggested, where the conventional standard normal quantile is used in the primary analysis, but where the use of alternative quantiles is also considered in a sensitivity analysis. Copyright © 2008 John Wiley & Sons, Ltd.     

A design‐by‐treatment interaction model for network meta‐analysis with random inconsistency effects

Network meta‐analysis is becoming more popular as a way to analyse multiple treatments simultaneously and, in the right circumstances, rank treatments. A difficulty in practice is the possibility of ‘inconsistency’ or ‘incoherence’, where direct evidence and indirect evidence are not in agreement. Here, we develop a random‐effects implementation of the recently proposed design‐by‐treatment interaction model, using these random effects to model inconsistency and estimate the parameters of primary interest. Our proposal is a generalisation of the model proposed by Lumley and allows trials with three or more arms to be included in the analysis. Our methods also facilitate the ranking of treatments under inconsistency. We derive R and I² statistics to quantify the impact of the between‐study heterogeneity and the inconsistency. We apply our model to two examples. © 2014 The Authors. Statistics in Medicine published by John Wiley & Sons, Ltd.     

Quantifying the impact of between‐study heterogeneity in multivariate meta‐analyses

Measures that quantify the impact of heterogeneity in univariate meta‐analysis, including the very popular I² statistic, are now well established. Multivariate meta‐analysis, where studies provide multiple outcomes that are pooled in a single analysis, is also becoming more commonly used. The question of how to quantify heterogeneity in the multivariate setting is therefore raised. It is the univariate R² statistic, the ratio of the variance of the estimated treatment effect under the random and fixed effects models, that generalises most naturally, so this statistic provides our basis. This statistic is then used to derive a multivariate analogue of I², which we call . We also provide a multivariate H² statistic, the ratio of a generalisation of Cochran's heterogeneity statistic and its associated degrees of freedom, with an accompanying generalisation of the usual I² statistic, . Our proposed heterogeneity statistics can be used alongside all the usual estimates and inferential procedures used in multivariate meta‐analysis. We apply our methods to some real datasets and show how our statistics are equally appropriate in the context of multivariate meta‐regression, where study level covariate effects are included in the model. Our heterogeneity statistics may be used when applying any procedure for fitting the multivariate random effects model. Copyright © 2012 John Wiley & Sons, Ltd.     

Multivariate meta‐analysis of mixed outcomes: a Bayesian approach

Multivariate random effects meta‐analysis (MRMA) is an appropriate way for synthesizing data from studies reporting multiple correlated outcomes. In a Bayesian framework, it has great potential for integrating evidence from a variety of sources. In this paper, we propose a Bayesian model for MRMA of mixed outcomes, which extends previously developed bivariate models to the trivariate case and also allows for combination of multiple outcomes that are both continuous and binary. We have constructed informative prior distributions for the correlations by using external evidence. Prior distributions for the within‐study correlations were constructed by employing external individual patent data and using a double bootstrap method to obtain the correlations between mixed outcomes. The between‐study model of MRMA was parameterized in the form of a product of a series of univariate conditional normal distributions. This allowed us to place explicit prior distributions on the between‐study correlations, which were constructed using external summary data. Traditionally, independent ‘vague’ prior distributions are placed on all parameters of the model. In contrast to this approach, we constructed prior distributions for the between‐study model parameters in a way that takes into account the inter‐relationship between them. This is a flexible method that can be extended to incorporate mixed outcomes other than continuous and binary and beyond the trivariate case. We have applied this model to a motivating example in rheumatoid arthritis with the aim of incorporating all available evidence in the synthesis and potentially reducing uncertainty around the estimate of interest. © 2013 The Authors. Statistics inMedicine Published by John Wiley & Sons, Ltd.