Inclusion of nonrandomized studies in Cochrane systematic reviews was found to be in need of improvement

ObjectivesNonrandomized studies (NRSs) are considered to provide less reliable evidence for intervention effects. However, these are included in Cochrane reviews, despite discouragement. There has been no evaluation of when and how these designs are used. Therefore, we conducted an overview of current practice.Study Design and SettingWe included all Cochrane reviews that considered NRS, conducting inclusions and data extraction in duplicate.ResultsOf the included 202 reviews, 114 (56%) did not cite a reason for including NRS. The reasons were divided into two major categories: NRS were included because randomized controlled trials (RCTs) are wanted (N = 81, 92%) but not feasible, lacking, or insufficient alone or because RCTs are not needed (N = 7, 8%). A range of designs were included with controlled before-after studies as the most common. Most interventions were nonpharmaceutical and the settings nonmedical. For risk of bias assessment, Cochrane Effective Practice and Organisation of Care Group's checklists were used by most reviewers (38%), whereas others used a variety of checklists and self-constructed tools.ConclusionMost Cochrane reviews do not justify including NRS. When they do, most are not in line with Cochrane recommendations. Risk of bias assessment varies across reviews and needs improvement.     

用评估系统评价测评工具对国内外五篇静脉鱼油对结局影响的Meta分析的质量评价

目的　用评估系统评价测评工具（AMSTAR）对静脉鱼油对手术后及重症监护患者影响的系统评价进行质量评价。方法　计算机检索1996年1月至2013年9月美国生物医学文献检索系统（PubMed）、荷兰医学文摘检索系统（EMBASE）、Cochrane图书馆、Web of Science、中国生物医学文献数据库、中国知网、万方数据平台等数据库中研究静脉鱼油对患者临床结局影响的系统评价及Meta分析文献,同时手工检索临床营养相关期刊中论文,并回溯相关论文的参考文献。由两名研究者按照纳入标准独立筛选文献、提取资料后,采用AMSTAR量表评价纳入的系统评价质量。结果　共纳入5篇相关Meta分析。基于AMSTAR的质量评价结果显示,2篇Meta分析为高质量,1篇为中等质量,2篇为低质量。结论　已发表静脉鱼油对手术后及重症监护患者影响的Meta分析质量参差不齐,有部分为低质量Meta分析。今后需要更多样本量较大、质量较高的随机、对照、双盲研究报道来更新系统评价,观察静脉鱼油对手术后及重症监护患者的临床结局和成本-效果影响,以期得出更可靠的结论。     

Statistical methods can be improved within Cochrane pregnancy and childbirth reviews

Objectives

To assess statistical methods within systematic reviews of the Cochrane Pregnancy and Childbirth Group (CPCG).

Study Design and Setting

We extracted details about statistical methods within 75 reviews containing at least 10 studies.

Results

The median number of forest plots per review was 52 (min = 5; max = 409). Seven of the 75 reviews assessed publication bias or explained why not. Forty-four of the 75 reviews performed random-effects meta-analyses; just 1 of these justified the approach clinically and none interpreted its pooled result correctly. Of 31 reviews not using random-effects, 26 assumed a fixed-effect given potentially moderate or large heterogeneity (I² > 25%). In their Methods section, 25 (33%) of the 75 reviews said I² was used to decide between fixed-effect and random-effects; however, in 12 of these (48%) reviews, this was not carried out in their Results section. Of 72 reviews with moderate or large heterogeneity, 47 (65%) did not explore the causes of heterogeneity or justify why not.

Conclusion

Within CPCG reviews, publication bias is rarely addressed; heterogeneity is often not appropriately considered, and random-effects analyses are incorrectly interpreted. How these shortcomings impact existing review conclusions needs further investigation, but regardless of this, we recomment the Cochrane Collaboration increase “hands-on” statistical support.     

The skills and experience of GRADE methodologists can be assessed with a simple tool

ObjectivesTo suggest approaches for guideline developers on how to assess a methodologist's expertise with Grading of Recommendations Assessment, Development and Evaluation (GRADE) methods and tasks and to provide a set of minimum skills and experience required to perform specific tasks related to guideline development using GRADE.Study Design and SettingWe used an iterative and consensus-based process involving individuals with in-depth experience with GRADE. We considered four main tasks: (1) development of key questions; (2) assessment of the certainty of effect estimates; (3) development of recommendations; and (4) teaching GRADE.ResultsThere are three basic approaches to determine a methodologist's skill set. First, self-report of knowledge, skills, and experience with a standardized “GRADE curriculum vitae (CV)” focused on each of the GRADE-related tasks; second, demonstration of skills using worked examples; third, a formal evaluation using a written or oral test. We suggest that the GRADE CV is likely to be useful and feasible to implement. We also suggest minimum training including attendance at one or more full-day workshops and familiarity with the main GRADE publications and the GRADE handbook.ConclusionsThe selection of a GRADE methodologist must be a thoughtful, reasoned decision, informed by the criteria suggested in this article and tailored to the specific project. Our suggested approaches need further pilot testing and validation.     

US Food and Drug Administration documents can provide unpublished evidence relevant to systematic reviews

ObjectivesA key systematic review (SR) methodology is comprehensive searching. The Drug Effectiveness Review Project (DERP) SRs search US Food and Drug Administration (FDA) documents to identify unpublished evidence. This study evaluates the success of those efforts.Study Design and SettingWe examined DERP reports published since 2003 for the use of FDA preapproval and postmarketing documents. We categorized evidence as (1) unique unpublished studies, (2) supplemental unpublished data, or (3) FDA postmarketing data analysis. Three reviewers independently assigned predetermined impact categories (e.g., qualitative or quantitative usage, fills gaps, confirms findings, and alters conclusions), resolving disagreements through consensus.ResultsAmong 114 DERP reports, 19% included unpublished studies and/or supplemental data and 10% included postmarketing analyses. From 175 preapproval documents, 14% provided eligible unpublished studies and 4.0% supplemental unpublished data that helped confirm previous findings, identify important harms, and fill gaps in knowledge about understudied subpopulations, outcomes, and comparisons. Report conclusion statements changed in 9 of 33 instances of premarketing documents compared with 4 of 12 postmarketing analyses.ConclusionsThe FDA documents can provide important unpublished evidence for SRs, although in a small proportion of cases. Future research should identify attributes that predict which reviews may benefit most from review of FDA documents.     

中国1～12岁健康儿童接种1剂水痘减毒活疫苗保护效果的系统评价与证据质量分析

目的 系统评价1～12岁健康儿童接种1剂水痘减毒活疫苗保护效果（VE）,探索运用GRADE（Grades of Recommendation, Assessment, Development, and Evaluation）证据质量评级方法评价疫苗效果观察性研究。方法 检索2019年以前关于中国1～12岁健康儿童1剂水痘减毒活疫苗VE的文献,采用纽卡斯尔-渥太华量表进行文献质量评价,采用Meta分析计算水痘疫苗VE,对研究类型、是否为暴发疫情、文献质量、年龄分亚组计算VE。采用GRADE方法评价整合证据质量。结果 共纳入32篇文献,合并后VE值为75%（95% CI：68%～80%）,亚组分析暴发疫情VE（66%,95% CI：57%～73%）低于非暴发疫情［85%（95% CI：78%～89%）］,<6岁儿童VE［84%（95% CI：77%～89%）］高于≥6岁儿童VE［60%（95% CI：51%～68%）］,不同研究类型、不同文献质量的VE差异无统计学意义。整合证据评级VE证据质量为“极低”,分别在偏倚风险和不一致性方面降一级,在间接性、不精确性和发表偏倚方面未降级。结论 中国1～12岁健康儿童1剂次水痘减毒活疫苗可提供中等水平保护,但≥6岁儿童VE明显降低,建议对<6岁儿童开展2剂次水痘疫苗接种。GRADE的证据质量评级方法可用于疫苗效果的观察性研究,建议制定观察性研究技术指南,提高证据体的整体质量。     

Systematic review of reviews of observational studies of school-level effects on sexual health,violence and substance use

For three decades there have been reports that the quality of schools affects student health. The literature is diverse and reviews have addressed different aspects of how the school environment may affect health. This paper is the first to synthesise this evidence using a review of reviews focusing on substance-use, violence and sexual-health. Twelve databases were searched. Eleven included reviews were quality-assessed and synthesised narratively. There is strong evidence that schools’ success in engaging students is associated with reduced substance use. There is little evidence that tobacco-control policies and school sexual-health clinics on their own are associated with better outcomes.     

近10年中国医疗行业适宜卫生技术推广应用效果的系统评估

目的:系统分析近10年中国医疗行业适宜卫生技术推广应用文献,总结适宜卫生技术推广应用效果评价指标,探讨存在不足。方法:计算机检索中国学术期刊全文数据库(CNKI)、万方数据库等,同时手工检索追查纳入文献的参考文献以尽可能纳入相关研究;运用描述性方法归纳总结文献信息。结果:最终纳入文献60篇,适宜卫生技术推广应用从"需求—过程—效果"进行评价,需求评价着重供方技术需求率,过程评价主要包括技术应用率、服务人次,技术知晓率、技术掌握率、推广覆盖率等,效果评价指标主要包括患者满意度、技术有效率、费用节省状况、技术安全性等。结论:针对当前存在不足,适宜卫生技术推广应用效果应考虑需求方和供方两方面,同时也不能忽略社会影响。     

Systematic reviews and health policy: the influence of a project on perinatal care since 1988

Context: Interrelated publications between 1988 and 1992 have influenced health policy and clinical practice: The Oxford Database of Perinatal Trials (ODPT), Effective Care in Pregnancy and Childbirth (ECPC), A Guide to Effective Care in Pregnancy and Childbirth (GECPC), and Effective Care of the Newborn Infant (ECNI). These publications applied and advanced methods that had a substantial history in the medical, biological, physical, and social sciences. Their unique contribution was to demonstrate the feasibility of organizing and sustaining programs to conduct systematic reviews across an entire field of health care. The publications also influenced subsequent advances in the methodology of systematic reviews and contributed to their proliferation; in large measure, but not entirely, because their editors and many of the authors participated in organizing and developing the Cochrane Collaboration. This article describes how and why these publications attracted favorable attention and resources from policymakers in numerous countries. Methods: This article applies historical methods to the analysis of primary sources that help explain the influence of systematic reviews, mainly on health policy. These methods guide an analysis of the rhetoric of the two volumes of ECPC and of primary sources generated as systematic reviews influenced health policy. The analysis of rhetoric employs the methods of intellectual history and social studies of science. The analysis of policymaking uses the methods of political and policy history, political science, and public administration. Because the focus of this article is how science influenced policy it alludes to but does not describe in detail the literature on the methods, production, and publication of systematic reviews. Findings: The influence of the four publications on policy was mainly a result of (1) their powerful blending of the rhetoric of scientific and polemical discourse, especially but not exclusively in ECPC; (2) a growing constituency for systematic reviews as a source of “evidence‐based” health care among clinicians, journalists, and consumers in many countries; and (3) recognition by significant policymakers who allocate resources to and within the health sector that systematic reviews could contribute to making health care more effective and to containing the growth of costs. Conclusions: Analysis of this aspect of the history of producing and applying systematic reviews informs understanding of how knowledge derived from research informs policy.     

Adjusting for Treatment Switching in Oncology Trials: A Systematic Review and Recommendations for Reporting

ObjectivesTo systematically review the quality of reporting on the application of switching adjustment approaches in published oncology trials and industry submissions to the National Institute for Health and Care Excellence Although methods such as the rank preserving structural failure time model (RPSFTM) and inverse probability of censoring weights (IPCW) have been developed to address treatment switching, the approaches are not widely accepted within health technology assessment. This limited acceptance may partly be a consequence of poor reporting on their application.MethodsPublished trials and industry submissions were obtained from searches of PubMed and nice.org.uk, respectively. The quality of reporting in these studies was judged against a checklist of reporting recommendations, which was developed by the authors based on detailed considerations of the methods.ResultsThirteen published trials and 8 submissions to nice.org.uk satisfied inclusion criteria. The quality of reporting around the implementation of the RPSFTM and IPCW methods was generally poor. Few studies stated whether the adjustment approach was prespecified, more than a third failed to provide any justification for the chosen method, and nearly half neglected to perform sensitivity analyses. Further, it was often unclear how the RPSFTM and IPCW methods were implemented.ConclusionsInadequate reporting on the application of switching adjustment methods increases uncertainty around results, which may contribute to the limited acceptance of these methods by decision makers. The proposed reporting recommendations aim to support the improved interpretation of analyses undertaken to adjust for treatment switching.     

Items from patient-oriented instruments can be integrated into interval scales to operationalize categories of the International Classification of Functioning, Disability and Health

ObjectiveTo exemplify the construction of interval scales for specified categories of the International Classification of Functioning, Disability and Health (ICF) by integrating items from a variety of patient-oriented instruments.Study Design and SettingPsychometric study using data from a convenience sample of 122 patients with rheumatoid arthritis. Patients completed six different patient-oriented instruments. The contents of the instrument items were linked to the ICF. Rasch analyses for ordered-response options were used to examine whether the instrument items addressing the ICF category b130: Energy and drive functions constitute a psychometrically sound interval scale.ResultsNineteen items were linked to b130: Energy and drive functions. Sixteen of the 19 items fit the Rasch model according to the chi-square (χ²) statistic (χ²_df=32 = 38.25, P = 0.21) and the Z-fit statistic (Z_Mean = 0.451, Z_SD = 1.085 and Z_Mean = ?0.223, Z_SD = 1.132 for items and persons, respectively). The Person Separation Index r_β was 0.93.ConclusionThe ICF category interval scales to operationalize single ICF categories can be constructed. The original format of the items included in the interval scales remains unchanged. This study represents a step forward in the operationalization and future implementation of the ICF.     

Eliciting ethical and social values in health technology assessment: A participatory approach

Despite a growing consensus that ethical and social values should be addressed in health technology assessment (HTA) processes, there exist a variety of methods for doing so. There is growing interest in involving citizens in policy development to ensure that decisions are legitimate, and reflect the broad social values of the public. We sought to bring these issues together by employing a participatory approach to elicit ethical and social values in HTA. Our primary objective was to elicit a set of ethical and social values from citizens that could be used to guide Ontario's HTA evidentiary review and appraisal process. A secondary objective was to explore the feasibility of using participatory approaches to elicit these values. A 14-person Citizens' Reference Panel on Health Technologies was established to provide input to the Ontario Health Technology Advisory Committee in developing its recommendations. A mixed methods approach was used where informed, deliberative discussions were combined with pre- and post-questionnaires, which assessed the relative importance of various ethical and social values as well as their stability over time. Over the course of five meetings, panel members progressed toward the identification of a set of core values -universal access, choice and quality care. These values were consistently prioritized as the core values that should be considered in the evaluation of health technologies and ensuing recommendations. Sustained and deliberative methods, like a citizens' panel, offer a promising approach for eliciting ethical and social values into HTA.