Reply to discussion of ‘Empirical vs natural weighting in random effects meta‐analysis’

Three discussion papers raise interesting points, but at the end of the day, we continue to recommend that past meta‐analyses which have influenced public policy or clinical paradigms be reanalyzed by unweighted methods. For all future meta‐analyses that employ either (a) fixed effects where concomitant treatment and/or eligibility are diverse or (b) classical random effects weighted methods, unweighted methods should serve as the primary analysis. Other analyses would be reasonable as secondary approaches. Two commentaries suggest that weights are only random to the extent of estimation errors in the between study and within study variance components. We shall demonstrate that even if these components are known, there is still considerable random variability in the weights. In fact, methods that try to weight the estimates inversely proportional to the variance have a number of undesirable properties, including bias, incorrect standard errors, inconsistency (including coverage of confidence intervals), and counter‐intuitive properties that the expectation of the estimator changes both with the number of studies sampled and with constant multiples of sample size across all studies. These adverse properties do not exist for the unweighted approach. From the numerical example of phenylephrine 10 mg, despite the arguments of Waksman, the proper conclusion is that the collection of studies does not constitute evidence‐based support for efficacy in terms of lowering nasal airway resistance. In the final section, we present two compelling examples where questionable inferences were made, with potential major public implications. Copyright © 2010 John Wiley & Sons, Ltd.