Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy

BACKGROUND: Despite the great possibility of publication bias in studies of diagnostic test research, empirical studies about publication bias have mainly focused on studies of treatment effect. METHODS: A sample of 28 meta-analyses of diagnostic accuracy was selected from the Database of Abstracts of Reviews of Effectiveness (DARE). Methods used to deal with publication and related biases in these meta-analyses were examined. Asymmetry of funnel plot of estimated test accuracy against corresponding precision for each meta-analysis was assessed by three statistical methods: rank correlation method, regression analysis, and Trim and Fill method. RESULTS: In reviews of diagnostic accuracy, there was a general lack of consideration of appropriate literature searching to minimize publication bias, and the impact of possible publication bias has not been systematically assessed. The results of the three different statistical methods consistently showed that in a large proportion of the 28 meta-analyses evaluated, the smaller studies were associated with a greater diagnostic accuracy. Exploratory analyses found that the fewer the literature databases searched, the greater the funnel plot asymmetry in meta-analyses. Funnel plot asymmetry tended to be greater in meta-analyses that included smaller number of primary studies. Our data revealed no consistent relationship between funnel plot asymmetry and language restriction in reviews. CONCLUSIONS: Further research is required to explain why smaller studies tended to report greater test accuracy in a large proportion of meta-analyses of diagnostic tests. In systematic reviews of diagnostic studies, literature search should be sufficiently comprehensive and possible impact of publication bias should be assessed.     

Systematic reviews of diagnostic research. Considerations about assessment and incorporation of methodological quality

The aim of this paper is to provide a theoretical background for performing and reading systematic reviews of diagnostic studies. We first discuss items for assessment of methodological quality in diagnostic studies and then present methods on how to incorporate these quality measures in systematic reviews. The items of internal validity determine whether the presented results of the individual studies are unbiased and can be trusted. Items of external validity determine to what extent the results are applicable outside the population in which the study was performed. The issues concern the adequacy of the study population, the performance and interpretation of the diagnostic tests and the presentation of the results. Several methods exist for incorporation of issues of methodological quality into systematic reviews, such as subgroup analyses, meta-regression analysis, and methodological scores. Publications of diagnostic studies should provide sufficient information to enable assessment of the methodological quality. Furthermore, publication of results of subgroup analyses should be promoted. Methodological criteria lists might help to improve the quality of systematic reviews of diagnostic research. With the items of methodological quality in mind the general practitioner might be better equipped to critically read and interpret diagnostic reviews.     

A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools

BACKGROUND AND OBJECTIVE: To review existing quality assessment tools for diagnostic accuracy studies and to examine to what extent quality was assessed and incorporated in diagnostic systematic reviews. METHODS: Electronic databases were searched for tools to assess the quality of studies of diagnostic accuracy or guides for conducting, reporting or interpreting such studies. The Database of Abstracts of Reviews of Effects (DARE; 1995-2001) was used to identify systematic reviews of diagnostic studies to examine the practice of quality assessment of primary studies. RESULTS: Ninety-one quality assessment tools were identified. Only two provided details of tool development, and only a small proportion provided any indication of the aspects of quality they aimed to assess. None of the tools had been systematically evaluated. We identified 114 systematic reviews, of which 58 (51%) had performed an explicit quality assessment and were further examined. The majority of reviews used more than one method of incorporating quality. CONCLUSION: Most tools to assess the quality of diagnostic accuracy studies do not start from a well-defined definition of quality. None has been systematically evaluated. The majority of existing systematic reviews fail to take differences in quality into account. Reviewers should consider quality as a possible source of heterogeneity.     

Exploring sources of heterogeneity in systematic reviews of diagnostic tests

It is indispensable for any meta-analysis that potential sources of heterogeneity are examined, before one considers pooling the results of primary studies into summary estimates with enhanced precision. In reviews of studies on the diagnostic accuracy of tests, variability beyond chance can be attributed to between-study differences in the selected cutpoint for positivity, in patient selection and clinical setting, in the type of test used, in the type of reference standard, or any combination of these factors. In addition, heterogeneity in study results can also be caused by flaws in study design. This paper critically examines some of the potential reasons for heterogeneity and the methods to explore them. Empirical support for the existence of different sources of variation is reviewed. Incorporation of sources of variability explicitly into systematic reviews on diagnostic accuracy is demonstrated with data from a recent review. Application of regression techniques in meta-analysis of diagnostic tests can provide relevant additional information. Results of such analyses will help understand problems with the transferability of diagnostic tests and to point out flaws in primary studies. As such, they can guide the design of future studies.     

Reporting of measures of accuracy in systematic reviews of diagnostic literature

Background

There are a variety of ways in which accuracy of clinical tests can be summarised in systematic reviews. Variation in reporting of summary measures has only been assessed in a small survey restricted to meta-analyses of screening studies found in a single database. Therefore, we performed this study to assess the measures of accuracy used for reporting results of primary studies as well as their meta-analysis in systematic reviews of test accuracy studies.

Methods

Relevant reviews on test accuracy were selected from the Database of Abstracts of Reviews of Effectiveness (1994–2000), which electronically searches seven bibliographic databases and manually searches key resources. The structured abstracts of these reviews were screened and information on accuracy measures was extracted from the full texts of 90 relevant reviews, 60 of which used meta-analysis.

Results

Sensitivity or specificity was used for reporting the results of primary studies in 65/90 (72%) reviews, predictive values in 26/90 (28%), and likelihood ratios in 20/90 (22%). For meta-analysis, pooled sensitivity or specificity was used in 35/60 (58%) reviews, pooled predictive values in 11/60 (18%), pooled likelihood ratios in 13/60 (22%), and pooled diagnostic odds ratio in 5/60 (8%). Summary ROC was used in 44/60 (73%) of the meta-analyses. There were no significant differences in measures of test accuracy among reviews published earlier (1994–97) and those published later (1998–2000).

Conclusions

There is considerable variation in ways of reporting and summarising results of test accuracy studies in systematic reviews. There is a need for consensus about the best ways of reporting results of test accuracy studies in reviews.     

The impact of including different study designs in meta-analyses of diagnostic accuracy studies

Diagnostic accuracy may be overestimated when using certain study designs; thus, the inclusion of studies using different designs in meta-analyses may have important effects on their results, and influence clinical decision making. The main aim of this study was to explore the influence of heterogeneity (based on the inclusion of different study designs) on diagnostic accuracy in a sample of published meta-analyses of diagnostic accuracy studies. We identified 30 systematic reviews which included 95 separate meta-analyses combining the results from a total of 976 individual studies. We classified each individual study according to the study design (case–control studies, clinically relevant patient series or other), and each meta-analysis according to the heterogeneity of the included studies. Furthermore, we registered how the methodological quality of the individual studies was assessed. Finally, for each meta-analysis, the summary measure of diagnostic accuracy was categorised as Good, Fair or Poor. We used logistic regression to assess the relationship between reporting good diagnostic accuracy and heterogeneity. Meta-analyses with heterogeneous populations were over three times more likely to report good diagnostic accuracy compared to meta-analyses that included only clinically relevant patient series (adjusted odds ratio 3.07 95 % CI 1.16–8.11). The combination of studies that use different designs, within the same meta-analysis, may lead to higher estimates of diagnostic accuracy.     

Meta-epidemiologic study showed frequent time trends in summary estimates from meta-analyses of diagnostic accuracy studies

ObjectivesTo evaluate changes over time in summary estimates from meta-analyses of diagnostic accuracy studies.Study Design and SettingWe included 48 meta-analyses from 35 MEDLINE-indexed systematic reviews published between September 2011 and January 2012 (743 diagnostic accuracy studies; 344,015 participants). Within each meta-analysis, we ranked studies by publication date. We applied random-effects cumulative meta-analysis to follow how summary estimates of sensitivity and specificity evolved over time. Time trends were assessed by fitting a weighted linear regression model of the summary accuracy estimate against rank of publication.ResultsThe median of the 48 slopes was −0.02 (−0.08 to 0.03) for sensitivity and −0.01 (−0.03 to 0.03) for specificity. Twelve of 96 (12.5%) time trends in sensitivity or specificity were statistically significant. We found a significant time trend in at least one accuracy measure for 11 of the 48 (23%) meta-analyses.ConclusionTime trends in summary estimates are relatively frequent in meta-analyses of diagnostic accuracy studies. Results from early meta-analyses of diagnostic accuracy studies should be considered with caution.     

Coronary heart disease in primary care: accuracy of medical history and physical findings in patients with chest pain - a study protocol for a systematic review with individual patient data

ABSTRACT: BACKGROUND: Chest pain is a common complaint in primary care, with coronary heart disease (CHD) being the most concerning of many potential causes. Systematic reviews on the sensitivity and specificity of symptoms and signs summarize the evidence about which of them are most useful in making a diagnosis. Previous meta-analyses are dominated by studies of patients referred to specialists. Moreover, as the analysis is typically based on study-level data, the statistical analyses in these reviews are limited while meta-analyses based on individual patient data can provide additional information. Our patient-level meta-analysis has three unique aims. First, we strive to determine the diagnostic accuracy of symptoms and signs for myocardial ischemia in primary care. Second, we investigate associations between study- or patient-level characteristics and measures of diagnostic accuracy. Third, we aim to validate existing clinical prediction rules for diagnosing myocardial ischemia in primary care. This article describes the methods of our study and six prospective studies of primary care patients with chest pain. Later articles will describe the main results. METHODS: We will conduct a systematic review and IPD meta-analysis of studies evaluating the diagnostic accuracy of symptoms and signs for diagnosing coronary heart disease in primary care. We will perform bivariate analyses to determine the sensitivity, specificity and likelihood ratios of individual symptoms and signs and multivariate analyses to explore the diagnostic value of an optimal combination of all symptoms and signs based on all data of all studies. We will validate existing clinical prediction rules from each of the included studies by calculating measures of diagnostic accuracy separately by study. DISCUSSION: Our study will face several methodological challenges. First, the number of studies will be limited. Second, the investigators of original studies defined some outcomes and predictors differently. Third, the studies did not collect the same standard clinical data set. Fourth, missing data, varying from partly missing to fully missing, will have to be dealt with. Despite these limitations, we aim to summarize the available evidence regarding the diagnostic accuracy of symptoms and signs for diagnosing CHD in patients presenting with chest pain in primary care. Review registration Centre for Reviews and Dissemination (University of York): CRD42011001170.     

Repeated-measures modeling improved comparison of diagnostic tests in meta-analysis of dependent studies

OBJECTIVE: Current methods for meta-analysis of diagnostic tests do not allow utilizing all the information from papers in which several tests have been studied on the same patient sample. We demonstrate how to combine several studies of diagnostic tests, where each study reports on more than one test and some tests (but not necessarily all of them) are shared with other papers selected for the meta-analysis. We adopt statistical methodology for repeated measurements for the purpose of meta-analysis of diagnostic tests. STUDY DESIGN AND SETTING: The method allows for missing values of some tests for some papers, takes into account different sample sizes of papers, adjusts for background and confounding factors including test-specific covariates and paper-specific covariates, and accounts for correlations of the repeated measurements within each paper. It does not need individual-level data, although it can be modified to use them, and uses the two-by-two table of test results vs. gold standard. RESULTS: The results are translated from diagnostic odds ratios (DOR) to more clinically useful measures such as predictive values, post-test probabilities, and likelihood ratios. Models to capture between-study variation are introduced. The fit and influence of specific studies on the regression can be evaluated. Furthermore, model-based tests for homogeneity of DORs across papers are presented. CONCLUSION: The use of this new method is illustrated using a recent meta-analysis of the D-dimer test for the diagnosis of deep venous thrombosis.     

Diagnostic test systematic reviews: Bibliographic search filters (“Clinical Queries”) for diagnostic accuracy studies perform well

BackgroundSystematic reviews of health care topics are valuable summaries of all pertinent studies on focused questions. However, finding all relevant primary studies for systematic reviews remains challenging.ObjectivesTo determine the performance of the Clinical Queries sensitive search filter for diagnostic accuracy studies for retrieving studies for systematic reviews.MethodsWe compared the yield of the sensitive Clinical Queries diagnosis search filter for MEDLINE and EMBASE to retrieve studies in diagnostic accuracy systematic reviews reported in ACP Journal Club in 2006.ResultsTwelve of 22 diagnostic accuracy reviews (452 included studies) met the inclusion criteria. After excluding 11 studies not in MEDLINE or EMBASE, 95% of articles (417 of 441) were captured by the sensitive Clinical Queries diagnosis search filter (MEDLINE and EMBASE combined). Of 24 studies not retrieved by the filter, 22 were not diagnostic accuracy studies. Reanalysis of the Clinical Queries filter without these 22 nondiagnosis articles increased its performance to 99% (417 of 419). We found no substantive impact of the two articles missed by the Clinical Queries filter on the conclusions of the systematic reviews in which they were cited.ConclusionThe sensitive Clinical Queries diagnostic search filter captured 99% of articles and 100% of substantive articles indexed in MEDLINE and EMBASE in diagnostic accuracy systematic reviews.     

Systematic overview finds variation in approaches to investigating and reporting on sources of heterogeneity in systematic reviews of diagnostic studies

ObjectivesTo examine how authors explore and report on sources of heterogeneity in systematic reviews of diagnostic accuracy studies.Study Design and SettingA cohort of systematic reviews of diagnostic tests was systematically identified. Data were extracted on whether an exploration of the sources of heterogeneity was undertaken, how this was done, the number and type of potential sources explored, and how results and conclusions were reported.ResultsOf the 65 systematic reviews, 12 did not perform a meta-analysis and eight of these gave heterogeneity between studies as a reason. Of the 53 reviews containing a meta-analysis, 40 explored potential sources of heterogeneity in a formal manner and 27 identified at least one source of heterogeneity. The reviews not investigating heterogeneity were smaller than those that did (median [interquartile range {IQR}], 8 [5–15] vs. 14 [11–19] primary studies). Twelve reviews performed a sensitivity analysis, 25 stratified analyses, and 19 metaregression. Many sources of heterogeneity were explored compared with the number of primary studies in a meta-analysis (median ratio, 1:5). Review authors placed importance on the exploration of sources of heterogeneity; 37 mentioned the exploration or the findings thereof in the abstract or conclusion of the main text.resultsConclusionMethods for investigating sources of heterogeneity varied widely between reviews. Based on our findings of the review, we made suggestions on what to consider and report on when exploring sources of heterogeneity in systematic reviews of diagnostic studies.     

Adjusting for partial verification or workup bias in meta-analyses of diagnostic accuracy studies

A key requirement in the design of diagnostic accuracy studies is that all study participants receive both the test under evaluation and the reference standard test. For a variety of practical and ethical reasons, sometimes only a proportion of patients receive the reference standard, which can bias the accuracy estimates. Numerous methods have been described for correcting this partial verification bias or workup bias in individual studies. In this article, the authors describe a Bayesian method for obtaining adjusted results from a diagnostic meta-analysis when partial verification or workup bias is present in a subset of the primary studies. The method corrects for verification bias without having to exclude primary studies with verification bias, thus preserving the main advantages of a meta-analysis: increased precision and better generalizability. The results of this method are compared with the existing methods for dealing with verification bias in diagnostic meta-analyses. For illustration, the authors use empirical data from a systematic review of studies of the accuracy of the immunohistochemistry test for diagnosis of human epidermal growth factor receptor 2 status in breast cancer patients.     

Diagnostic Accuracy of IgA Anti-Transglutaminase and IgG Anti-Deamidated Gliadin for Diagnosis of Celiac Disease in Children under Two Years of Age: A Systematic Review and Meta-Analysis

The need of adding the determination of anti-deamidated gliadin peptide (DGP) IgG to anti-transglutaminase (TTG) IgA antibodies for diagnosis of celiac disease (CD) in children <2 years of age is controversial. We performed a systematic review and meta-analysis to evaluate, by head-to-head comparison, the diagnostic accuracy of TTG IgA and DGP IgG antibodies. We searched PubMed, MEDLINE, and Embase databases up to January 2021. The diagnostic reference was intestinal biopsy. We calculated the sensitivity and specificity of these tests and the odds ratio (OR) between the tests. Fifteen articles were eligible for the systematic review and ten were eligible for the meta-analysis. Sensitivity and specificity were 0.96 (95% confidence interval (CI), 0.91–0.98) and 0.96 (95% CI, 0.85–0.99) for DGP IgG and 0.93 (95% CI, 0.88–0.97) and 0.98 (95% CI, 0.96–0.99) for TTG IgA, respectively. TTG IgA specificity was significantly higher (OR 9.3 (95% CI, 2.3–37.49); p < 0.001) while the sensitivity of DGP IgG was higher without reaching statistical significance (OR: 0.6 (95% CI, 0.24–1.51); p = 0.28). Both the meta-analysis and the systematic review showed that some children with early CD are missed without the DGP IgG test. In children <2 years of age, TTG IgA is the best CD screening test; however, the addition of DGP IgG may increase the diagnostic sensitivity.     

Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract

BACKGROUND AND OBJECTIVES: Methods to identify studies for systematic reviews of diagnostic accuracy are less well developed than for reviews of intervention studies. This study assessed (1) the sensitivity and precision of five published search strategies and (2) the reliability and accuracy of reviewers screening the results of the search strategy. METHODS: We compared the results of the search filters with the studies included in two systematic reviews, and assessed the interobserver reliability of two reviewers screening the list of articles generated by a search strategy. RESULTS: In the first review, the search strategy published by van der Weijden had the greatest sensitivity, and in the second, four search strategies had 100% sensitivity. There was "substantial" agreement between two reviewers, but in the first review each reviewer working on their own would have missed one paper eligible for inclusion in the review. Ascertainment intersection techniques indicate that it is unlikely that further papers have been missed in the screening process. CONCLUSION: Published search strategies may miss papers for reviews of diagnostic test accuracy. Papers are not easily identified as studies of diagnostic test accuracy, and the lack of information in the abstract makes it difficult to assess the eligibility for inclusion in a systematic review.     

The accuracy of physical diagnostic tests for assessing meniscal lesions of the knee: a meta-analysis

OBJECTIVE: Our systematic review summarizes the evidence about the accuracy of those tests. SEARCH STRATEGY: We performed a literature search of MEDLINE (1966-1999) and EMBASE (1988-1999) with additional reference tracking. SELECTION CRITERIA: Articles written in English, French, German, or Dutch, that addressed the accuracy of at least one physical diagnostic test for meniscus injury with arthrotomy, arthroscopy, or magnetic resonance imaging as the gold standard were included. We excluded studies if no reference group or only test-positives had been included, if the study pertained to cadavers only, or if only physical examination under anesthesia was considered. DATA COLLECTION/ANALYSIS: Two reviewers independently selected studies, assessed the methodologic quality, and abstracted data using a standardized protocol. We calculated sensitivity, specificity, and likelihood ratios for each test, and summary estimates when appropriate and possible. MAIN RESULTS: Of 402 identified studies, 13 met the inclusion criteria. The results of the index and reference tests were assessed independently (blindly) of each other in only 2 studies, and in all studies verification bias seemed to be present. The study results were highly heterogeneous The summary receiver operating characteristic curves of the assessment of joint effusion, the McMurray test and joint line tenderness indicated little discriminative power for these tests. Only the predictive value of a positive McMurray test was favorable. CONCLUSIONS: The methodologic quality of studies addressing the diagnostic accuracy of meniscal tests was poor, and the results were highly heterogeneous. The poor characteristics indicate that these tests are of little value for clinical practice.     

The development of a quality appraisal tool for studies of diagnostic reliability (QAREL)

Background and ObjectiveIn systematic reviews of the reliability of diagnostic tests, no quality assessment tool has been used consistently. The aim of this study was to develop a specific quality appraisal tool for studies of diagnostic reliability.MethodsKey principles for the quality of studies of diagnostic reliability were identified with reference to epidemiologic principles, existing quality appraisal checklists, and the Standards for Reporting of Diagnostic Accuracy (STARD) and Quality Assessment of Diagnostic Accuracy Studies (QUADAS) resources. Specific items that encompassed each of the principles were developed. Experts in diagnostic research provided feedback on the items that were to form the appraisal tool. This process was iterative and continued until consensus among experts was reached.ResultsThe Quality Appraisal of Reliability Studies (QAREL) checklist includes 11 items that explore seven principles. Items cover the spectrum of subjects, spectrum of examiners, examiner blinding, order effects of examination, suitability of the time interval among repeated measurements, appropriate test application and interpretation, and appropriate statistical analysis.ConclusionsQAREL has been developed as a specific quality appraisal tool for studies of diagnostic reliability. The reliability of this tool in different contexts needs to be evaluated.     

A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations

An important quality of meta-analytic models for research synthesis is their ability to account for both within- and between-study variability. Currently available meta-analytic approaches for studies of diagnostic test accuracy work primarily within a fixed-effects framework. In this paper we describe a hierarchical regression model for meta-analysis of studies reporting estimates of test sensitivity and specificity. The model allows more between- and within-study variability than fixed-effect approaches, by allowing both test stringency and test accuracy to vary across studies. It is also possible to examine the effects of study specific covariates. Estimates are computed using Markov Chain Monte Carlo simulation with publicly available software (BUGS). This estimation method allows flexibility in the choice of summary statistics. We demonstrate the advantages of this modelling approach using a recently published meta-analysis comparing three tests used to detect nodal metastasis of cervical cancer.     

Systematic reviews of test accuracy should search a range of databases to identify primary studies

OBJECTIVE: To estimate the yield from searching a range of bibliographic databases and additional sources to identify test accuracy studies for systematic reviews. STUDY DESIGN AND SETTING: We examined eight systematic reviews and their database searches: MEDLINE, EMBASE, BIOSIS, Science Citation Index, LILACS, Pascal, and CENTRAL. We used studies included in each systematic review as the "gold standard," against which yield was estimated. For each database, we classified studies in each gold standard set as being (1) included in the database and identified by searches, (2) included and not identified, and (3) not included in the database. RESULTS: No search identified all studies in any gold standard set. EMBASE, Science Citation Index, and BIOSIS contained studies that were not on MEDLINE. Over 20% of studies in the gold standard sets were not identified by searching MEDLINE. Six studies on LILACS were not on any other database. Eight gold standard studies were not included in any of the databases, and a further 22 were not identified by the electronic search strategies. CONCLUSIONS: Systematic reviews of test accuracy studies should search a range of databases. Even searches designed to be very sensitive, that do not use study design filters, can fail to identify relevant studies.     

How objective are systematic reviews? Differences between reviews on complementary medicine

Systematic reviews are considered the most reliable tool to summarize existing evidence. To determine whether reviews that address the same questions can produce different answers we examined systematic reviews of herbal medicine, homeopathy, and acupuncture taken from a previously established database. Information on literature searching, inclusion criteria, selection process, quality assessment, data extraction, methods to summarize primary studies, number of included studies, results and conclusions was compared qualitatively. Seventeen topics (eight on acupuncture, six on herbal medicines, three on homeopathy) had been addressed by 2-5 systematic reviews each. The number of primary studies in the reviews varied greatly within most topics. The most obvious reason for discrepancies between the samples was different inclusion criteria (in thirteen topics). Methods of literature searching may have contributed with some topics but the equivalence of the searches was difficult to assess. Differences were frequently observed in other methodological aspects, in results and in conclusions. This analysis shows that, at least in the three areas examined, systematic reviews often differ considerably. Readers should be aware that apparently minor decisions in the review process can have major impact.