Pragmatic vs. explanatory: an adaptation of the PRECIS tool helps to judge the applicability of systematic reviews for daily practice

Objective

The Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) tool was designed to classify randomized clinical trials (RCT) as being more pragmatic or explanatory. We modified the PRECIS tool (called PRECIS-Review tool [PR-tool]) to grade individual trials and systematic reviews of trials. This should help policy makers, clinicians, researchers, and guideline developers to judge the applicability of individual trials and systematic reviews.

Study Design and Setting

To illustrate the usefulness and applicability of the PR-tool, we applied it to two systematic reviews. Each included RCT was scored on the 10 PRECIS domains on a scale of 1-5. After this scoring, a 10-domain average for each individual trial and for the systematic review a single domain average and an overall average was calculated.

Results

One review was more pragmatic with an average score of 3.7 (range, 2.9-4.6) on our PR-tool, whereas the other review was more explanatory with an average score of 1.9 (range, 1.1-3.3). The results also suggest that the included studies within each systematic review were rather uniform in their approach, although some domains seemed more prone to heterogeneity.

Conclusion

The PR-tool provides a useful estimate that gives insight by estimating quantitatively how pragmatic each RCT in the review is, which methodological domains are pragmatic or explanatory, and how pragmatic the review is.     

Using existing systematic reviews for developing vaccination recommendations: Results of an international expert workshop

National immunization technical advisory groups (NITAGs) develop immunization-related recommendations. Systematic reviews are recommended to be used in this process, but conducting them requires significant resources, which many NITAGs lack. Using existing systematic reviews could help address this problem.The Robert Koch Institute and collaborators set up the SYSVAC2 project to facilitate the retrieval of existing systematic reviews and offer guidance on using them. This will include an online registry of systematic reviews relevant to immunization policy and an online course on how to use existing reviews. This report describes an international expert workshop held in December 2019 to develop consensus on methods for using existing reviews and other relevant factors for the registry and course.Members from NITAGs representing different regions of the world presented their experiences of using systematic reviews and reflected on challenges inhibiting use. Three methodologists considered different aspects of using systematic reviews. Interactive sessions followed, where implications for SYSVAC2 were discussed. Participants supported having critical appraisal ratings, plain language summaries, keyword search, and data visualization functions in the registry. They suggested tailoring course content to different audiences and including overviews of reviews as a topic and examples of how NITAGs have used or could use existing reviews. Participants agreed that whether a review is out-of-date should be decided by those using the review rather than registry staff. The registry could help by highlighting the date of literature search or included primary studies. Participants recommended a visualization function to highlight overlap across reviews and guidance on handling challenges to using reviews, ideally, involving a practical element. No consensus was reached on which critical appraisal tool to use for reviews in the registry, but a majority of participants wanted registry staff to perform appraisals. Formative research is planned before the registry and online course are launched in 2020.     

Evaluations of community mental health care in low- and middle-income countries: a 10-year review of the literature

Community-based models of providing mental health services are widely considered effective ways of serving individuals diagnosed with mental illness, but more comprehensive literature on these models in low- and middle-income countries is needed. This study is a systematic review of the effects of community-based models on health outcomes of adults with depression, schizophrenia, panic disorder, or bipolar disorders in middle- and low-income countries. PubMed, PsycINFO, and Cochrane Reviews were searched, returning 500 articles. The seventeen interventions included in this review in 14 countries show us that community-based mental health services can provide improvements in mental health outcomes, and the limited cost analyses suggest cost savings associated with community models of care. These findings are in line with much of the research on higher income countries. In addition, the studies also point to the gaps in the literature on costs, rural areas, bipolar disorders, and panic disorders, and note the need for further reviews of interventions targeting additional diseases, children, and adolescents as well as studies published in languages other than English. This review of the literature serves as a stepping stone for further research in community-based mental health services in low- and middle-income countries. The works reviewed here provide a base of knowledge that will assist us in taking the important next steps in program implementation and evaluation.     

Evidence-based medicine and health economics: a case study of end stage renal disease

This paper explores the potential for use of an economic evaluation framework alongside systematic reviews. Clinical issues in dialysis therapy for end stage renal disease are used as case studies. The effectiveness data required were obtained from a systematic review of randomized controlled trials. Resource use and cost data were obtained from three sources; the identified randomized controlled trials, a separate review of observational studies and primary data collection. The results of the case studies show that, although simple economic evaluations were possible, issues arose, such as how transferable results are between settings and how appropriate it is to focus on the average patient. The interface between economic evaluation and systematic reviews needs to be further developed in order to ensure that the best available evidence can be used to inform future policy and research.     

Adding Decision Models to Systematic Reviews: Informing a Framework for Deciding When and How to Do So

ObjectiveDecision models are sometimes used alongside systematic reviews to synthesize evidence. Clarity, however, is lacking about when and how to conduct modeling studies in tandem with systematic reviews, as well as about how to evaluate and present model results. The objective of this study was to collect and analyze information from various sources to inform the development of a framework for deciding when and how a decision model should be added to a systematic review.MethodsWe collected data through 1) review and analysis of evidence reports that used decision models; 2) review and synthesis of current best practices for the development of decision models; 3) interviews of Evidence-Based Practice Center directors and selected staff, United States Preventive Services Task Force members, and decision modelers who developed models used by the United States Preventive Services Task Force; and 4) a focus group of expert modelers.ResultsModels are well suited to address gaps in the literature, better suited for certain types of research questions, and essential for determining the value of information relating to future research. Opinions differ regarding whether model outputs constitute evidence, but interviewees expressed concern over the lack of standards and directions in grading and reporting such “evidence.” Interviews of stakeholders and modelers revealed the importance of communication and presentation of model results as well as the importance of model literacy and involvement of stakeholders.ConclusionsThe study demonstrates the need for a framework for deciding when and how to use models alongside systematic reviews and provides information to develop such a framework.     

Conceptual frameworks and key dimensions to support coverage decisions for vaccines

BackgroundHealth policy makers often have to face decisions on whether and how to incorporate new vaccines into immunisation plans. This study aims to review and catalogue the relevant current frameworks and taxonomies on vaccines and connect these to the DECIDE Evidence to Decision framework (EtD), a general framework based on evidence-based criteria to guide decision-making on intervention adoption.MethodsWe systematically searched MEDLINE, EMBASE, Cochrane Library and funding agency websites from 1990 to 2013. We included systematic reviews and primary studies presenting decision-making tools for community vaccine adoption. We qualitatively summarised the reports by purpose, targeted country, principal results, and decisional models. We then extracted and compared the dimensions adopted by vaccine frameworks across studies.ResultsFourteen studies (five systematic reviews and nine primary studies) were included. Several factors frequently influenced decision-makers’ views on vaccines: the most frequent political-context factors considered were Importance of illness or problem, Vaccine characteristics, Resource use, and Feasibility. Others such as Values and preferences and Acceptability were less consistently reported. We did not find evidence on the reasons why a framework for vaccine adoption differs from that for decisions on the adoption of an intervention in general, such as the EtD. There are limited data on how dimensions are explained in practical factors and directly linked to coverage decisions.ConclusionsThis review summarises conceptual models and taxonomy of a heterogeneous and evolving area in health policy decisions. A shared and comprehensive framework on vaccine coverage remains to be achieved with its single dimensions (epidemiologic, effectiveness, economic, and social) valued differently across studies. A generic tool such as the EtD conceptualises all relevant dimensions, and might reduce inconsistencies.     

COSMOS-E：病因学观察性研究的系统综述和Meta分析制作指南与案例解读

本文针对病因学观察性研究的系统综述和Meta分析制作指南（Guidance on conducting systematic reviews and Meta-analyses of observational studies of etiology,COSMOS-E）的主要内容进行了介绍,并举例说明COSMOS-E中的关键内容,内容覆盖制作病因学观察性研究系统综述的每个步骤,从提出研究问题、定义暴露和结局、评估偏倚风险到统计分析,为研究人员开展和分析这类综述提供了指导。     

Use of methodological search filters to identify diagnostic accuracy studies can lead to the omission of relevant studies

OBJECTIVE: To determine the usefulness of methodological filters in search strategies for diagnostic studies in systematic reviews. STUDY DESIGN AND SETTING: We made an inventory of existing methodological search filters for diagnostic accuracy studies and applied them in PubMed to a reference set derived from 27 published systematic reviews in a broad range of clinical fields. Outcome measures were the fraction of not identified relevant studies and the reduction in the number of studies to read. RESULTS: We tested 12 search filters. Of the studies included in the systematic reviews, 2%-28% did not pass the sensitive search filters, 4%-24% did not pass the accurate filters, and 39%-42% did not pass the specific filters. Decrease in number-needed-to-read when a search filter was used in a search strategy for a diagnostic systematic review varied from 0% to 77%. CONCLUSION: The use of methodological filters to identify diagnostic accuracy studies can lead to omission of a considerable number of relevant studies that would otherwise be included. When preparing a systematic review, it may be preferable to avoid using methodological filters.     

Evaluation of the methodological quality of systematic reviews of health status measurement instruments

A systematic review of measurement properties of health-status instruments is a tool for evaluating the quality of instruments. Our aim was to appraise the quality of the review process, to describe how authors assess the methodological quality of primary studies of measurement properties, and to describe how authors evaluate results of the studies. Literature searches were performed in three databases. One hundred and forty-eight reviews were included. The purpose of included reviews was to identify health status instruments used in an evaluative application and to report on the measurement properties of these instruments. Two independent reviewers selected the articles and extracted the data. Reviews were often of low quality: 22% of the reviews used one database, the search strategy was often poorly described, and in many cases it was not reported whether article selection (75%) and data extraction (71%) was done by two independent reviewers. In 11 reviews the methodological quality of the primary studies was evaluated for all measurement properties, and of these 11 reviews only 7 evaluated the results. Methods to evaluate the quality of the primary studies and the results differed widely. The poor quality of reviews hampers evidence-based selection of instruments. Guidelines for conducting and reporting systematic reviews of measurement properties should be developed.     

Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice

OBJECTIVE: Heterogeneity between study results can be a problem in any systematic review or meta-analysis of clinical trials. Identifying its presence, investigating its cause and correctly accounting for it in analyses all involve difficult decisions for the researcher. Our objectives were: to collate recommendations on the subject of dealing with heterogeneity in systematic reviews of clinical trials; to investigate current practice in addressing heterogeneity in Cochrane reviews; and to compare current practice with recommendations. METHODS: We review guidelines for those undertaking systematic reviews and examine how heterogeneity is addressed in practice in a sample of systematic reviews, and their protocols, from the Cochrane Database of Systematic Reviews. RESULTS: Advice to reviewers is on the whole consistent and sensible. However, examination of a sample of Cochrane protocols and reviews demonstrates that the advice is difficult to follow given the small numbers of studies identified in many systematic reviews, the difficulty of pre-specifying important effect modifiers for subgroup analysis or meta-regression and the unresolved debate concerning fixed versus random effects meta-analyses. There was disagreement between protocols and reviews, often either regarding choice of important potential effect modifiers or due to the review identifying too few studies to perform planned analyses. CONCLUSION: Guidelines that address practical issues are required to reduce the risk of spurious findings from investigations of heterogeneity. This may involve discouraging statistical investigations such as subgroup analyses and meta-regression, rather than simply adopting a cautious approach to their interpretation, unless a large number of studies is available. The notion of a priori specification of potential effect modifiers for a retrospective review of studies is ill-defined, and the appropriateness of using a statistical test for heterogeneity to decide between analysis strategies is suspect.     

Complex interventions and their implications for systematic reviews: a pragmatic approach

Complex interventions present unique challenges for systematic reviews. Current debates tend to center around describing complexity, rather than providing guidance on what to do about it. At a series of meetings during 2009–2012, we met to review the challenges and practical steps reviewer could take to incorporate a complexity perspective into systematic reviews. Based on this, we outline a pragmatic approach to dealing with complexity, beginning, as for any review, with clearly defining the research question(s). We argue that reviews of complex interventions can themselves be simple or complex, depending on the question to be answered. In systematic reviews and evaluations of complex interventions, it will be helpful to start by identifying the sources of complexity, then mapping aspects of complexity in the intervention onto the appropriate sources of evidence (such as specific types of quantitative or qualitative study). Although we focus on systematic reviews, the general approach is also applicable to primary research that is aimed at evaluating complex interventions. Although the examples are drawn from health care, the approach may also be applied to other sectors (e.g., social policy or international development). We end by concluding that systematic reviews should follow the principle of Occam's razor: explanations should be as complex as they need to be and no more.     

Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract

BACKGROUND AND OBJECTIVES: Methods to identify studies for systematic reviews of diagnostic accuracy are less well developed than for reviews of intervention studies. This study assessed (1) the sensitivity and precision of five published search strategies and (2) the reliability and accuracy of reviewers screening the results of the search strategy. METHODS: We compared the results of the search filters with the studies included in two systematic reviews, and assessed the interobserver reliability of two reviewers screening the list of articles generated by a search strategy. RESULTS: In the first review, the search strategy published by van der Weijden had the greatest sensitivity, and in the second, four search strategies had 100% sensitivity. There was "substantial" agreement between two reviewers, but in the first review each reviewer working on their own would have missed one paper eligible for inclusion in the review. Ascertainment intersection techniques indicate that it is unlikely that further papers have been missed in the screening process. CONCLUSION: Published search strategies may miss papers for reviews of diagnostic test accuracy. Papers are not easily identified as studies of diagnostic test accuracy, and the lack of information in the abstract makes it difficult to assess the eligibility for inclusion in a systematic review.     

伞形评价——一种新型循证医学分析方法

伞形评价是一种最新的循证医学分析方法,是继系统评价和Meta分析之后的更高层级的医疗证据分析方法。它以系统评价和Meta分析为基础,又高于系统评价和Meta分析。所以在很多具体操作方面,它与Meta分析类似,但更需要注意其独特之处。本文着重介绍了伞形评价的概念、主要步骤、局限性以及发展趋势等。     

Training practitioners in preparing systematic reviews: a cross-sectional survey of participants in the Australasian Cochrane Centre training program

Background  

Although systematic reviews of health care interventions are an invaluable tool for health care providers and researchers, many potential authors never publish reviews. This study attempts to determine why some people with interest in performing systematic reviews do not subsequently publish a review; and what steps could possibly increase review completion.     

The art and science of knowledge synthesis

Objectives

To review methods for completing knowledge synthesis.

Study Design and Setting

We discuss how to complete a broad range of knowledge syntheses. Our article is intended as an introductory guide.

Results

Many groups worldwide conduct knowledge syntheses, and some methods are applicable to most reviews. However, variations of these methods are apparent for different types of reviews, such as realist reviews and mixed-model reviews. Review validity is dependent on the validity of the included primary studies and the review process itself. Steps should be taken to avoid bias in the conduct of knowledge synthesis. Transparency in reporting will help readers assess review validity and applicability, increasing its utility.

Conclusion

Given the magnitude of the literature, the increasing demands on knowledge syntheses teams, and the diversity of approaches, continuing efforts will be important to increase the efficiency, validity, and applicability of systematic reviews. Future research should focus on increasing the uptake of knowledge synthesis, how best to update reviews, the comparability between different types of reviews (eg, rapid vs. comprehensive reviews), and how to prioritize knowledge synthesis topics.     

The Aboriginal Birth Cohort Study: When is a cohort study not a cohort design?

Aims: The paper describes how a variety of different epidemiological study designs can be applied to data arising from a single prospective study. Methods: An overview of the data collection phases of the Aboriginal Birth Cohort Study is given. We illustrate how different research questions that require different analytical designs can be asked of the data collected in the present study. Results: With reference to five generic questions in health research, we showed how sixteen specific questions could be addressed in the Aboriginal Birth Cohort Study. These referred to a range of analytical designs. Conclusion: Readers need to take care not to confuse the overall design of a study with the design of a specific analysis. When conducting systematic literature reviews, studies should be classified according to the analytical design used in the specific report included in the review and not according to the design of the overall project.