Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report

OBJECTIVE: Scientific and ethical justification for new clinical trials requires them to have been designed in the light of scientifically defensible assessments of relevant previous research. Reliable interpretation of the results of new clinical trials entails setting them in the context of updates of the reviews upon which they were deemed scientifically and ethically justifiable. We have shown previously that most reports of randomized trials published in five general medical journals in May 1997 and in May 2001 failed to set their results in the context of the findings from similar research. In the current study, we assess whether there had been progress in this respect in 2005 and also investigate the extent to which reports begin by referring to systematic reviews providing the justification for the new research reported. DESIGN: Assessment of the Introduction and Discussion sections in all reports of randomized trials published during May 2005 in five general medical journals. SETTING: Reports of randomized trials in five general medical journals. PARTICIPANTS: Annals of Internal Medicine, BMJ, JAMA, Lancet and New England Journal of Medicine. INTERVENTIONS: None. MAIN OUTCOME MEASURES: The inclusion or mention of one or more systematic reviews in the Introduction or Discussion section of each report assessed. RESULTS: We found 18 reports of randomized trials. The Introduction sections referred to systematic reviews in five (27%) of these reports. None of the discussion sections of the 15 reports of trials that were not the first published trials to address the question studied placed the results of the new trial in the context of an updated systematic review of other research. Although reference was made to relevant systematic reviews in five of these 15 reports, there was no integration - quantitative or qualitative - of the results of the new trials in an update of these reviews. In the remaining ten reports there was no evidence that any systematic attempt had been made to set the new results in the context of previous trials. CONCLUSIONS: There is no evidence of progress between 1997 and 2005 in the proportion of reports of trials published in general medical journals which discussed new results within the context of up-to-date systematic reviews of relevant evidence from other controlled trials. Although the proportion of trials referring to systematic reviews in Discussion sections has increased, the majority of reports continued to fail even to do this. Similarly, most researchers appear not to have considered a systematic review when designing their trial. Researchers and journal editors do a disservice to the interests of the public and others involved in healthcare decision-making by acquiescing in this situation.     

Comparison of reports of randomized controlled trials and systematic reviews in surgical journals: literature review

OBJECTIVES: Randomized controlled trials and systematic reviews of such trials are the gold standard for assessing the effectiveness of interventions. There have been concerns about the anecdotal evidence underpinning many of the interventions used and introduced into surgical care. The American College of Surgeons has prioritized the need for more trials and systematic reviews of trials. To investigate the assertion that the methodological quality of studies conducted in surgery is in general poor and to assess the possible impact of new policy developments in the US, we sought to compare the number and proportion of published randomized controlled trials and systematic reviews in the leading two US and UK general surgical journals. Two reviewers systematically and independently hand searched all issues of these journals over a 12-month period to identify randomised controlled trials and systematic reviews. DESIGN: Systematic searching and independent abstraction of data from all volumes of the top two general surgical journals published in the USA and the UK in 2004. SETTING: 519 original reports in UK journals and 616 original reports in USA journals. MAIN OUTCOME MEASURES: Number and proportion of randomized controlled trials and systematic reviews. RESULTS: Overall, the proportion of randomized controlled trials in all four journals was 5.6% (95% confidence interval [CI] 4.4-7.0) and 5.2% (95% CI 4.1-6.7) for systematic reviews. For journals published in the UK 29/519 (5.6%) of the publications were reports of randomized controlled trials, and for the USA journals this figure was 34/616 (5.5%); odds ratio [OR]=0.99, 95%CI 0.6-1.6; P=0.96. Systematic reviews were significantly more commonly reported in the UK journals: UK 37/519 (7.1%) versus USA 22/616 (3.6%); OR=0.48, 95%CI 0.3-0.8; P<0.01. CONCLUSIONS: The concerns expressed almost a decade ago remain valid: there are still very few reports of randomized controlled trials and systematic reviews published in leading USA and UK surgical journals, with relatively little difference between these countries in the proportion of reported studies employing these designs. The American College of Surgeons initiative has yet to make an impact.     

Citation of randomized evidence in support of guidelines of therapeutic and preventive interventions

Guideline statements may be supported by evidence obtained from various study designs, but randomized trials are usually considered most important for making recommendations about therapeutic and preventive interventions. This study evaluated the extent to which randomized trials are cited in guidelines published in major journals. The references of 191 guidelines of therapeutic and/or preventive interventions published in Annals of Internal Medicine, BMJ, JAMA, Lancet, NEJM and Pediatrics in 1979, 1984, 1989, 1994, and 1999, were analyzed. The percentage of guidelines not citing any randomized controlled trials (RCTs) decreased gradually from 95% in 1979 to 53% in 1999. Among 4,853 references of the guidelines, there were 393 RCTs (8.1% of total), 19 systematic reviews (0.4%), and 23 meta-analyses of RCTs (0.5%). Among 19 guidelines published in 1999 or 1994 with <2 RCTs cited, in eight cases additional pertinent RCTs were identified that had not been cited by the guideline. There is a clear increase in the use of randomized evidence by guidelines over time. However, several guidelines in major journals still cite few or no RCTs.     

An analysis of systematic reviews indicated low incorpororation of results from clinical trial quality assessment

OBJECTIVE: We investigated the frequency of quality assessment of randomized controlled trials within systematic reviews and the incorporation of the quality assessment in the analysis. STUDY DESIGN AND SETTING: We included new systematic reviews of at least five trials of therapeutic or preventive interventions that appeared in issue 2, 2003, of the Cochrane Database of Systematic Reviews. All systematic reviews in the 2002 issues of the Annals of Internal Medicine, BMJ, JAMA, and Lancet were identified in Pubmed. All reviews were assessed under unblinded conditions using preprinted extraction forms. RESULTS: Trial quality was assessed in all Cochrane reviews and most (74%) of the paper reviews. When we excluded 11 paper reviews that were also published as Cochrane review, the percentage remained similar (67%). Fifty percent of all Cochrane reviews and 61% of all paper reviews incorporated the results of the quality assessment in the analysis. CONCLUSION: Half of the reviews did not incorporate the results of the quality assessment in the analysis. Authors, peer-reviewers, and editors should no longer focus exclusively on whether quality assessment has been performed but should also concentrate on incorporation of quality assessments in the analysis of the systematic review.     

Getting evidence into practice: the work of the Cochrane Effective Practice and Organization of care Group (EPOC)

Policy makers and continuing educators often face difficult decisions about which educational and quality assurance interventions to provide. Where possible, such decisions are best informed by rigorous evidence, such as that provided by systematic reviews. The Cochrane Collaboration is an international organization that aims to help people make well-informed decisions about health care by preparing, maintaining, and ensuring the accessibility of systematic reviews of the benefits and risks of health care interventions. International collaborative review groups prepare Cochrane reviews for publication in The Cochrane Library, a collection of databases available on CD-ROM and the World Wide Web and updated quarterly. The Cochrane Effective Practice and Organization of Care Group (EPOC) aims to prepare and maintain systematic reviews of professional, financial, organizational, and regulatory interventions that are designed to improve professional practice and the delivery of effective health services. EPOC has 17 reviews and 20 protocols published in Issue 3, 2000, of the Cochrane Library, with further protocols in development. We also have undertaken an overview of previously published systematic reviews of professional behavior change strategies. Our specialized register contains details of over 1,800 studies that fall within the group's scope. Systematic reviews provide a valuable source of information for policy makers and educators involved in planning continuing education and quality assurance initiatives and organizational change. EPOC will attempt to keep the Journal of Continuing Education in the Health Professions informed on an ongoing basis about new systematic reviews that it produces in the area of continuing medical education and quality assurance.     

Scientific Evidence in Health Technology Assessment Reports: An In-Depth Analysis of European Assessments on High-Risk Medical Devices

Background

The aim of this study was to examine the scientific evidence on clinical effectiveness and safety used in health technology assessments (HTAs) of high-risk medical devices (MDs) in Europe.

Interventions to promote physical activity among young and adolescent girls: a systematic review

A narrative systematic review was conducted to describe the available evidence from physical activity (PA) interventions that targeted girls aged 5-18 years and to determine their effectiveness and key characteristics of success. Systematic literature searches were conducted using four databases: PubMed, Web of Science, PsychInfo and SPORTDiscus and by examining the reference lists of included articles and published relevant reviews, to identify studies published in English from 2000 to July 2010. Randomized controlled trials or quasi-experimental designs with pre-test and post-test behavioral outcome data (objective or self-report measure) were included. Methodological quality was assessed using a checklist and conclusions were made concerning effectiveness. A total of 29 articles were reviewed, describing the evaluation of 21 interventions. Ten studies reported a favorable intervention effect upon PA outcomes, seven of which were rated as having a high methodological quality. Multi-component school-based interventions that also offer a physical education that address the unique needs of girls seemed to be the most effective. Although family support is revealed as ineffective, peer strategies showed promising evidence. The review finishes highlighting possible intervention strategies and reporting areas where further investigation is required.     

Use of quality-of-life outcome assessments in current cancer clinical trials

During the past decade, the importance of integrating quality of life with other indicators of efficacy in cancer clinical trials has been emphasized. This article reviews quality-of-life assessment in clinical trial reports. All empirical studies using patient ratings published in the Journal of Clinical Oncology from 1992 to 1996 were identified. Of these, 59 articles (4%) included quality-of-life assessment, most frequently measured by self-report questionnaires. Most articles reported clinical trials of symptom management or treatment. Future research needs to specify relationships between quality of life, toxicity, and survival. Continued development of this field requires more frequent inclusion of quality of life in clinical studies.     

Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy

ObjectiveTo describe the quality and methods of systematic reviews of physiotherapy interventions, compare Cochrane and non-Cochrane reviews, and establish the interrater reliability of the Overview Quality Assessment Questionnaire (OQAQ) quality assessment tool.Study Design and SettingA survey of 200 published systematic reviews was done. Two independent raters assessed the search strategy, assessment of trial quality, outcomes, pooling, conclusions, and overall quality (OQAQ). The study was carried out in the University research center.ResultsIn these reviews, the five most common databases searched were MEDLINE, EMBASE, Cochrane Library, CINAHL, and Cochrane Review Group Registers. The Cochrane allocation concealment system and Jadad Scale were most frequently used to assess trial quality. Cochrane reviews searched more databases and were more likely to assess trial quality, report dichotomous outcomes for individual trials, and conduct a meta-analysis than non-Cochrane reviews. Non-Cochrane reviews were more likely to conclude that there was a beneficial effect of treatment. Cochrane reviews were of higher quality than non-Cochrane reviews. There has been an increase in the quality of systematic reviews over time. The OQAQ has fair to good interrater reliability.ConclusionThe quality of systematic reviews in physiotherapy is improving, and the use of Cochrane Collaboration procedures appears to improve the methods and quality.     

Effective practices for school-based tobacco use prevention

BACKGROUND: Research evidence addressing effectiveness of tobacco use prevention interventions has accumulated since the 1970s. Systematic reviews 1985-2006 were considered, building on previous syntheses and spanning tobacco control and prevention efforts to date. Practitioners' experience was drawn upon to supplement research evidence. METHODS: A systematic, comprehensive approach was used to synthesize published literature evaluating the effectiveness of school-based tobacco use prevention interventions. Systematic reviews conducted on all populations published in English, peer-reviewed journals were included. Reviews were screened for relevance and assessed for methodological quality using pre-tested, standardized tools. The best available evidence was extracted and integrated with experiential evidence from individual interview and focus group results from practitioners involved in tobacco use prevention programming. RESULTS: Considerable consensus among the three evidence sources indicates that school-based tobacco use prevention interventions are effective in reducing smoking prevalence, reducing smoking initiation and intended smoking intentions in the short term. There is adequate evidence from over three decades of research and years of experience to recommend ongoing implementation of school-based tobacco use prevention interventions. CONCLUSION: There is strong evidence that school-based tobacco use prevention programs are largely effective for most tobacco use related outcomes, at least in the short term.     

Evidence-based medicine. (Role of the Cochrane collaboration)

The basic condition of practising evidence-based medicine is the knowledge of the evidence. Strongest of all evidences are the conclusions drawn from systematic reviews of randomized controlled trials. The over 2 million papers published annually in the biomedical literature is neither available nor possible to read for any individual, therefore it is necessary to identify and collect all relevant clinical trials, to make them available in databases, and to produce, to publish and to update systematic reviews based on the best evidence. The Cochrane Collaboration was established for these purposes in 1993. The activity of the Collaboration is summarized by a database The Cochrane Library which is updated quarterly. The user friendly Cochrane Library currently contains bibliographic data of over 218 thousand controlled trials (Cochrane Controlled Trials Register, CC-TR), and 1014 systematic reviews (Cochrane Database of Systematic reviews, CDSR) prepared by reviewers in 49 Review Groups. As the Cochrane Library contains information on much more controlled trials than any other databases including MEDLINE, decision makers in health care should be aware of this source of information. In addition to the application of the Cochrane Library as a source of information, active participation by identifying controlled trials in the Hungarian medical literature and registering them in the CCTR is another important task.     

The health impact of worksite nutrition and cholesterol intervention programs

PURPOSE: To summarize and provide a critical review of worksite health promotion program evaluations published between 1980 and 1995 that address nutrition and hypercholesterolemia. The article discusses and critiques both intervention methods and research methodologies to identify the most effective strategies. METHODS: Core articles are 26 original, data-based studies that report on measures of health status, behavior, attitudes, and knowledge as outcomes of worksite nutrition and cholesterol interventions. Only work published since 1980 that clearly describes nutrition or cholesterol interventions and that includes identifiable nutrition-related outcomes is reviewed. The main search method was the same one used for this special issue; supplementary sources included those found in earlier reviews or identified through backward searches or expert contact. SUMMARY OF IMPORTANT FINDINGS: Ten worksite nutrition education programs were reviewed and were categorized as group education, group education plus individual counseling/instruction, cafeteria-based programs, and group education plus cafeteria-based programs. Four of these were randomized studies, and one used the worksite as the unit of randomization and analysis. Sixteen worksite cholesterol programs were reviewed, in five categories: monitoring; individual counseling; group sessions or classes; mediated methods using print, audiovisual, telephone, and self-help kits; and combination approaches. Of these, eight were randomized controlled trials; most tested interventions for persons with elevated cholesterol levels, although four studies reported cholesterol education programs for the general employee population. Six large controlled trials of worksite nutrition and cholesterol interventions in progress are also described. MAJOR CONCLUSIONS: The conclusions that can be drawn from this review are limited by the study designs used, which often lacked control groups, used nonrandomized designs, or relied on self-selected high-risk or volunteer participants. Our rating for the quality of the evidence in the literature as a whole lies between suggestive and indicative. It is clear that worksite nutrition and cholesterol programs are feasible and that participants benefit in the short-term. Conclusive evidence about a causal relationship between worksite nutrition and cholesterol programs and improved behavior or health is not yet available, although studies currently underway hold promise for providing more solid evidence about the potential efficacy of these interventions.     

Development of the Cochrane Collaboration's CENTRAL Register of controlled clinical trials

The Cochrane Collaboration has established a centralized database of controlled trials and other studies of health care interventions (called CENTRAL) that serves as the best available resource for all those preparing and maintaining systematic reviews or otherwise searching for trials. CENTRAL is available on The Cochrane Library. This article describes the history and methods of CENTRAL's development and the results of an analysis of the current composition of CENTRAL. As of September 2000, CENTRAL contained almost 300,000 citations to reports of trials, contributed mainly by Cochrane Groups and Centers around the world. Development of CENTRAL has been an ambitious, scholarly undertaking and has resulted in a valuable resource: CENTRAL includes citations to controlled trials that may not be indexed in MEDLINE, EMBASE, or other bibliographic databases; citations published internationally in many languages; and citations that are available only in conference proceedings or other hard-to-access sources.     

Of mites and men: reference bias in narrative review articles: a systematic review

BACKGROUND: Citations in scientific articles may tend to favor the views presented. We studied whether there is such reference bias in narrative review articles that discuss interventions against house dust mites for people with asthma. DESIGN: Systematic review of reviews identified in a Medline search that expressed an opinion about the clinical effects of physical or chemical intervention methods. MAIN OUTCOME MEASURE: Positive bias was judged to have occurred if the reference list contained a higher proportion of trial references with significant results than among all trials available to the authors (published 2 years or more prior to the review). RESULTS: Seventy reviews were included, of which 63 (90%) recommended physical interventions. Forty-six reviews had trial references, 4 of these only to chemical interventions. In the remaining 42 reviews, reference bias was detected (P=2 x 10-8). The most quoted trial had only 7 patients per group, its claimed significant result was probably erroneous, and it did not report a clinical outcome. Intervention recommendations were often based on nonrandomized evidence, and the most quoted nonrandomized controlled study had included only 10 patients per group but claimed very positive results. CONCLUSION: The narrative review articles were severely biased, and their positive intervention recommendations are at variance with the systematic Cochrane Review on this topic and a recent very large trial of physical intervention, both of which failed to find an effect.     

Searching for unpublished trials in Cochrane reviews may not be worth the effort

ObjectiveTo assess the value of searching for unpublished data by exploring the extent to which Cochrane reviews include unpublished data and by evaluating the quality of unpublished trials.Study Design and SettingWe screened all 2,462 completed Cochrane reviews published since 2000 in the Cochrane Database of Systematic Reviews Issue 3, 2006. In a random sample (n = 61) of 292 reviews, including unpublished trials, we studied all 116 references.ResultsUnpublished trials make up 8.8% of all included trials in our sample. Thirty-eight percent of the “unpublished” trials have in fact been published. Allocation concealment was “unclear” or not adequate in 54.3% and 61.3% reported blinding. In 47.2% reported withdrawal rates were >20%. Trials that were eventually published had larger mean population sizes (P-value, 0.02). Of the reported sponsors, 87.3% were drug companies. Methodological quality and publication bias are mentioned in half of the reviews and explored in a third. Quality ratings did not have consequences for pooling, because 82.8% was included in the forest plots.ConclusionsA minority of Cochrane reviews include “unpublished trials” and many of these are eventually published. Truly unpublished studies have poor or unclear methodological quality. Therefore, it may be better to invest in regular updating of reviews, rather than in extensive searching for unpublished data.     

Knowledge for knowledge translation: the role of the Cochrane Collaboration

Knowledge-translation (KT) activities, including continuing education, should be informed by the totality of available research evidence. Systematic reviews are a generic methodology used to synthesize evidence from a broad range of research methods addressing different questions. Over the past decade, there has been a dramatic increase in the availability of systematic reviews that could support KT activities. However, the conduct of systematic reviews is technically challenging, and it is not surprising that the quality of available reviews is variable. In addition, unless attempts are made to update systematic reviews, they rapidly become out of date. The Cochrane Collaboration is a unique, worldwide, not-for-profit organization that aims to help people make well-informed decisions about all forms of health care by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of health care interventions. Globally, over 13, 000 consumers, clinicians, policymakers, and researchers are involved with The Cochrane Collaboration and have to date produced over 2, 500 systematic reviews that can be used to inform KT activities. The Cochrane Collaboration publishes its reviews quarterly in The Cochrane Library. Cochrane reviews have been used to develop a number of KT-derivative products for professionals, consumers, and policymakers. Whereas most Cochrane Review groups focus on specific clinical areas, the Cochrane Effective Practice and Organisation of Care Group undertakes reviews of interventions to improve health care delivery and health care systems, including reviews of different KT activities. We summarize the activities of The Cochrane Collaboration and how these can contribute to KT activities.     

An update of a systematic review of controlled clinical trials on the primary prevention of back pain at the workplace

OBJECTIVE: To update the evidence on the effectiveness of lumbar supports, education and exercise in the primary prevention of low back pain at the workplace. METHODS: A computerized search for controlled clinical trials published between 1997 and 2002 was conducted, and the methodological quality of the studies was assessed using a criteria list. The available evidence was graded with a rating system for the level of evidence. Effect sizes of individual studies were combined if the studies were sufficiently similar. RESULTS: Five new papers were identified for the update. These trials were added to the previously available trials (n = 11). The methodological quality of most studies was low. Since three of four RCTs on lumbar supports reported no effect, there is no evidence for the effectiveness of lumbar supports. No evidence for education could be found either, since all six RCTs showed negative results. The four RCTs on exercise consistently reported a positive effect, indicating limited evidence for the effectiveness of exercise. CONCLUSION: There is no evidence for the effectiveness of lumbar supports or education in the primary prevention of low back pain at the workplace. There is limited evidence for the efficacy of exercise, and the effect that can be obtained is moderate. There is still a need for methodologically sound studies and studies on the cost-effectiveness of exercise. Also the possible effect of lumbar supports in the treatment of back pain needs further investigation.     

A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals

This study set out to compare Cochrane reviews and reviews published in paper-based journals. Two assessment tools were used to collect the data, a 23-item checklist developed by Sacks and a nine-item scale developed by Oxman. Cochrane reviews were found to be better at reporting some items and paper-based review at reporting others. The overall quality was found to be low. This represents a serious situation because clinicians, health policy makers, and consumers are often told that systematic reviews represent "the best available evidence." In the period since this study, the Cochrane Collaboration has taken steps to improve the quality of its reviews through, for example, more thorough prepublication refereeing, developments in the training and support offered to reviewers, and improvements in the system for postpublication peer review. In addition, the use of evidence-based criteria (i.e., the QUOROM statement) for reporting systematic reviews may help further to improve their quality.     

The contribution of handsearching European general health care journals to the Cochrane Controlled Trials Register

A fundamental aim of any systematic review is that all relevant studies should be identified and considered for inclusion. Limitations with searching bibliographic databases led the Cochrane Collaboration to search journals by hand for reports of trials. This article presents the results of a 3-year project to identify and make accessible reports of randomized trials published in European general health care journals. Overall, 21,620 reports of controlled trials were identified from 119 journals from 16 countries. More than three quarters (76%) were published in U.K. or German journals. Only 3,640 (17%) reports were indexed in MEDLINE as controlled trials, and 6,554 (30%) were not indexed in MEDLINE at all. Bibliographic details for all reports are available by searching The Cochrane Controlled Trials Register in The Cochrane Library. This project has ensured that a large proportion of trial reports not previously identifiable has been made accessible to those preparing systematic reviews.     

偏倚风险评估系列:(四)整群设计随机对照试验

介绍整群设计随机对照试验Cochrane偏倚评估工具2.0版本（RoB2.0）的主要内容,阐述与平行设计RoB2.0的不同之处,并举例说明整群设计RoB2.0的使用方法和注意事项。RoB2.0针对整群设计的自身特点,设置了相应的信号问题,为将整群设计试验纳入系统综述进行证据整合提供偏倚风险信息。