Identification and analysis of randomised controlled trials in nursing: a preliminary study.

OBJECTIVES: To describe preliminary work undertaken for development of a nursing contribution to the Cochrane Collaboration. To ascertain whether there are randomised controlled trials (RCTs) on nursing care which need to be identified for inclusion in systematic reviews of the effects of health care. DESIGN: Searches by Medline (1966-94) and by hand of 11 nursing research journals from inception to the end of 1994 to identify RCTs and systematic reviews; and a comparison of searches by hand and by Medline for three nursing research journals. MAIN MEASURES: Total number of RCTs identified and number of RCTs published in nursing journals; the sensitivity of Medline searches; and aspects of nursing care evaluated by RCT. RESULTS: The work is ongoing and 522 reports of RCTs and 20 systematic reviews of effectiveness have been identified so far. The sensitivity of Medline searches for RCTs in nursing journals is as low as 36% for one journal and the lack of reference to research design in the title or abstract was the main reason for the lack of sensitivity. CONCLUSIONS: There are RCTs that evaluate aspects of nursing care, and are published in nursing and non-nursing journals, and are largely undertaken by nurses. These must be reviewed in ongoing systematic reviews of the effects of health care (including those undertaken as part of the Cochrane Collaboration). Nursing journals must be hand searched to identify these studies as the lack of reference to study design in the titles and abstracts of nursing trials leads to poor indexing in electronic databases such as Medline.     

The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus

Most systematic reviews rely substantially on the assessment of the methodological quality of the individual trials. The aim of this study was to obtain consensus among experts about a set of generic core items for quality assessment of randomized clinical trials (RCTs). The invited participants were experts in the field of quality assessment of RCTs. The initial item pool contained all items from existing criteria lists. Subsequently, we reduced the number of items by using the Delphi consensus technique. Each Delphi round comprised a questionnaire, an analysis, and a feedback report. The feedback report included staff team decisions made on the basis of the analysis and their justification. A total of 33 international experts agreed to participate, of whom 21 completed all questionnaires. The initial item pool of 206 items was reduced to 9 items in three Delphi rounds. The final criteria list (the Delphi list) was satisfactory to all participants. It is a starting point on the way to a minimum reference standard for RCTs on many different research topics. This list is not intended to replace, but rather to be used alongside, existing criteria lists.     

Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria

OBJECTIVES: The aim of the Consensus on Health Economic Criteria (CHEC) project is to develop a criteria list for assessment of the methodological quality of economic evaluations in systematic reviews. The criteria list resulting from this CHEC project should be regarded as a minimum standard. METHODS: The criteria list has been developed using a Delphi method. Three Delphi rounds were needed to reach consensus. Twenty-three international experts participated in the Delphi panel. RESULTS: The Delphi panel achieved consensus over a generic core set of items for the quality assessment of economic evaluations. Each item of the CHEC-list was formulated as a question that can be answered by yes or no. To standardize the interpretation of the list and facilitate its use, the project team also provided an operationalization of the criteria list items. CONCLUSIONS: There was consensus among a group of international experts regarding a core set of items that can be used to assess the quality of economic evaluations in systematic reviews. Using this checklist will make future systematic reviews of economic evaluations more transparent, informative, and comparable. Consequently, researchers and policy-makers might use these systematic reviews more easily. The CHEC-list can be downloaded freely from http://www.beoz.unimaas.nl/chec/.     

The reporting quality of meta-analyses improves: a random sampling study

OBJECTIVES: To determine the overall quality of reporting of meta-analyses (QUOROMs) of randomized controlled trials (RCTs) based on the QUOROM statement, to compare the reporting quality of paper-based articles and Cochrane reviews, and to determine whether compliance with the statement improves over time. STUDY DESIGN AND SETTING: A random sample of systematic reviews or meta-analyses of RCTs was selected from Medline (2000-2005). RESULTS: A total of 161 articles were included. The mean QUOROM score was 12.3 (95% confidence interval [CI]: 12.0, 12.6), which rose from 10.5 (95% CI: 8.8, 12.1) in 2000 to 13.0 (95% CI: 12.2, 13.8) in 2005. The mean QUOROM scores of Cochrane reviews and paper-based articles were 14.2 (95% CI: 13.9, 14.5) and 11.7 (95% CI: 11.3, 12.1), respectively. Compared with the paper-based articles, the Cochrane reviews had better reporting quality in the abstract section, while the quality of their trial flows was poor. The fulfillment of most QUOROM items improved with time. A linear relation of the QUOROM score with time was revealed. CONCLUSIONS: The reporting quality of meta-analyses improves with time. The reporting quality of Cochrane reviews is better compared with paper-based articles. Room still exists for improvements in the reporting quality of both Cochrane and paper-based articles.     

Organisation of a proposed Cochrane Diet and Nutrition Field

The Cochrane Collaboration helps people make well-informed decisions about health care by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions. In all, 51 Cochrane Review Groups are responsible for preparing and maintaining the reviews. Most of these Review Groups are problem-based. However, there are a number of areas or dimensions of health care, such as the setting of care (eg primary care) or the type of patient/consumer (eg older persons), that cannot be usefully conceptualised as 'health problems'. In order to reflect the interests of these dimensions, or 'fields', of health care more effectively, the Cochrane Collaboration created another type of entity: Cochrane Fields or Networks. The core functions of these Fields are described. It is proposed that a new Cochrane Subfield for Diet and Nutrition should be set up to attract a cadre of new reviewers in order to ensure the proper representation of diet and nutrition expertise in the relevant Collaborative Review Groups. The methodological pitfalls of the use of nonrandomised studies will be a key issue for the new Subfield, since the focus of the Cochrane Collaboration's programme of searching databases, journals and conference proceedings has been on the identification of reports of randomised controlled trials (RCTs). Some of these sources will be need to be searched again for non-RCT studies. The financial considerations of establishing of a Cochrane Subfield for Diet and Nutrition are discussed.     

Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM Statement

BACKGROUND: The Quality of Reporting of Meta-analyses (QUOROM) Conference was convened to address standards for improving the quality of reporting of meta-analyses of clinical randomised controlled trials (RCTs). METHODS: The QUOROM group consists of 30 clinical epidemiologists, clinicians, statisticians, editors, and researchers. In conference, the group was asked to identify items they thought should be included in a checklist of standards. Whenever possible, checklist items were guided by research evidence suggesting that failure to adhere to the item proposed could lead to biased results. A modified Delphi technique was used in assessing candidate items. RESULTS: The conference resulted in the QUOROM statement, a checklist, and a flow diagram. The checklist describes our preferred way to present the abstract, introduction, methods, results, and discussion sections of a report of a meta-analysis. it is organized into 21 headings and subheadings regarding searches, selection, validity assessment, data abstraction, study characteristics, and quantitative data synthesis, and in the results with 'trial flow', study characteristics, and quantitative data synthesis; research documentation was identified for eight of the 18 items. The flow diagram provides information about both the numbers of RCTs identified, included, and excluded and the reasons for exclusion of trials. INTERPRETATION: We hope this report will generate further thought about ways to improve the quality of reports of meta-analyses of RCTs and that interested readers, reviewers, researchers, and editors will use the QUOROM statement and generate ideas for its improvement.     

A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals

This study set out to compare Cochrane reviews and reviews published in paper-based journals. Two assessment tools were used to collect the data, a 23-item checklist developed by Sacks and a nine-item scale developed by Oxman. Cochrane reviews were found to be better at reporting some items and paper-based review at reporting others. The overall quality was found to be low. This represents a serious situation because clinicians, health policy makers, and consumers are often told that systematic reviews represent "the best available evidence." In the period since this study, the Cochrane Collaboration has taken steps to improve the quality of its reviews through, for example, more thorough prepublication refereeing, developments in the training and support offered to reviewers, and improvements in the system for postpublication peer review. In addition, the use of evidence-based criteria (i.e., the QUOROM statement) for reporting systematic reviews may help further to improve their quality.     

The contribution of handsearching European general health care journals to the Cochrane Controlled Trials Register

A fundamental aim of any systematic review is that all relevant studies should be identified and considered for inclusion. Limitations with searching bibliographic databases led the Cochrane Collaboration to search journals by hand for reports of trials. This article presents the results of a 3-year project to identify and make accessible reports of randomized trials published in European general health care journals. Overall, 21,620 reports of controlled trials were identified from 119 journals from 16 countries. More than three quarters (76%) were published in U.K. or German journals. Only 3,640 (17%) reports were indexed in MEDLINE as controlled trials, and 6,554 (30%) were not indexed in MEDLINE at all. Bibliographic details for all reports are available by searching The Cochrane Controlled Trials Register in The Cochrane Library. This project has ensured that a large proportion of trial reports not previously identifiable has been made accessible to those preparing systematic reviews.     

Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches

OBJECTIVES: To analyze sources searched in Cochrane reviews, to determine the proportion of trials included in reviews that are indexed in major databases, and to compare the quality of these trials with those from other sources. METHODS: All new systematic reviews in the Cochrane Library, Issue1 2001, that were restricted to randomized controlled trials (RCTs) or quasi-RCTs were selected. The sources searched in the reviews were recorded, and the trials included were checked to see whether they were indexed in four major databases. Trials not indexed were checked to determine how they could be identified. The quality of trials found in major databases was compared with those found from other sources. RESULTS: The range in the number of databases searched per review ranged between one and twenty-seven. The proportion of the trials in the four databases were Cochrane Controlled Trials Register = 78.5%, MEDLINE = 68.8%, Embase = 65.0%, and Science/Social Sciences Citation Index = 60.7%. Searching another twenty-six databases after Cochrane Controlled Trials Register (CCTR), MEDLINE, and Embase only found 2.4% additional trials. There was no significant difference between trials found in the CCTR, MEDLINE, and Embase compared with other trials, with respect to adequate allocation concealment or sample size. CONCLUSIONS: There was a large variation between reviews in the exhaustiveness of the literature searches. CCTR was the single best source of RCTs. Additional database searching retrieved only a small percentage of extra trials. Contacting authors and manufacturers to find unpublished trials appeared to be a more effective method of obtaining the additional better quality trials.     

Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary

OBJECTIVE: To assess whether language of publication restrictions impact the estimates of an intervention's effectiveness, whether such impact is similar for conventional medicine and complementary medicine interventions, and whether the results are influenced by publication bias and statistical heterogeneity. STUDY DESIGN AND SETTING: We set out to examine the extent to which including reports of randomized controlled trials (RCTs) in languages other than English (LOE) influences the results of systematic reviews, using a broad dataset of 42 language-inclusive systematic reviews, involving 662 RCTs, including both conventional medicine (CM) and complementary and alternative medicine (CAM) interventions. RESULTS: For CM interventions, language-restricted systematic reviews, compared with language-inclusive ones, did not introduce biased results, in terms of estimates of intervention effectiveness (random effects ration of odds rations ROR=1.02; 95% CI=0.83-1.26). For CAM interventions, however, language-restricted systematic reviews resulted in a 63% smaller protective effect estimate than language-inclusive reviews (random effects ROR=1.63; 95% CI=1.03-2.60). CONCLUSION: Language restrictions do not change the results of CM systematic reviews but do substantially alter the results of CAM systematic reviews. These findings are robust even after sensitivity analyses, and do not appear to be influenced by statistical heterogeneity and publication bias.     

Assessment of clinical trial quality and its impact on meta-analyses

OBJECTIVE: To evaluate whether different quality assessment tools applied to a group of clinical trials could be correlated, and what would be their impact on meta-analysis results. METHODS: Thirty-eight randomized controlled clinical trials were analyzed. These had been selected for a systematic review of the therapeutic efficacy of alpha interferon for treating chronic hepatitis B. The following tools were utilized: Maastricht (M), Delphi (D), Jadad (J) and the Cochrane Collaboration (CC) method (gold standard). The Spearman correlation coefficient was used to compare the results from the three methods. The Kappa test was used to assess the concordance between the reviewers in applying the tools, and the weighted Kappa test was applied to compare the quality ranking determined by the tools. The outcomes assessed in the meta-analyses were clearance of HBV-DNA and HBeAg. RESULTS: The studies presented regular to low quality. The concordance between reviewers varied according to the instrument utilized: D=0.12; J=0.29; M=0.33; and CC=0.53. The correlation was moderate and homogeneous (D/J=0.51; D/M=0.53; and J/M=0.52). The meta-analysis result relating to HBV-DNA ranged from RR=0.71 (95% CI: 0.66-0.77) to RR=0.67 (95% CI: 0.58-0.79). For HBeAg, the results ranged from RR=0.85 (95% CI: 0.80-0.90) to RR=0.85 ( 95% CI: 0.77-0.93). These results depended on the quality of the studies included. CONCLUSIONS: The quality assessment tools presented good correlation. In systematic reviews with the same direction of effect, the quality assessment may not significantly change the results. The Cochrane Collaboration method was the most reproducible method and easiest to apply.     

Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture

BACKGROUND: A key aim when conducting systematic reviews of randomized controlled trials (RCTs) is to include all of the evidence, if possible. Serious bias may result if trials are missed through inadequate search strategies. OBJECTIVE: The objective was to evaluate the search plan for identifying RCTs in nutrition as part of a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture. DESIGN: We identified potential studies by searching the electronic databases BIOSIS, CABNAR, CINAHL, EMBASE, HEALTHSTAR, and MEDLINE; reference lists in trial reports; and other relevant articles. We also contacted investigators and other experts for information and searched 4 nutrition journals by hand. RESULTS: We identified 15 RCTs that met the predefined inclusion criteria. The search plan identified 8 trials each in EMBASE, HEALTHSTAR, and MEDLINE and 7 in BIOSIS and CABNAR. BIOSIS was the only electronic database source of 2 trials. Eleven trials were identified by searching electronic databases and 2 unpublished trials were identified via experts in the field. We found one trial, published only as a conference abstract, by searching nutrition journals by hand. After publication of the protocol for the review in The Cochrane Library, we were informed of another unpublished trial. CONCLUSIONS: We found that a limited search plan based on only MEDLINE or one of the other commonly available databases would have failed to locate nearly one-half of the studies. To protect against bias, the search plan for a systematic review of nutritional interventions should be comprehensive.     

Evidence-based medicine. (Role of the Cochrane collaboration)

The basic condition of practising evidence-based medicine is the knowledge of the evidence. Strongest of all evidences are the conclusions drawn from systematic reviews of randomized controlled trials. The over 2 million papers published annually in the biomedical literature is neither available nor possible to read for any individual, therefore it is necessary to identify and collect all relevant clinical trials, to make them available in databases, and to produce, to publish and to update systematic reviews based on the best evidence. The Cochrane Collaboration was established for these purposes in 1993. The activity of the Collaboration is summarized by a database The Cochrane Library which is updated quarterly. The user friendly Cochrane Library currently contains bibliographic data of over 218 thousand controlled trials (Cochrane Controlled Trials Register, CC-TR), and 1014 systematic reviews (Cochrane Database of Systematic reviews, CDSR) prepared by reviewers in 49 Review Groups. As the Cochrane Library contains information on much more controlled trials than any other databases including MEDLINE, decision makers in health care should be aware of this source of information. In addition to the application of the Cochrane Library as a source of information, active participation by identifying controlled trials in the Hungarian medical literature and registering them in the CCTR is another important task.     

The Oxford Implementation Index: a new tool for incorporating implementation data into systematic reviews and meta-analyses

ObjectivesThis article presents a new tool that helps systematic reviewers to extract and compare implementation data across primary trials. Currently, systematic review guidance does not provide guidelines for the identification and extraction of data related to the implementation of the underlying interventions.Study Design and SettingA team of systematic reviewers used a multistaged consensus development approach to develop this tool. First, a systematic literature search on the implementation and synthesis of clinical trial evidence was performed. The team then met in a series of subcommittees to develop an initial draft index. Drafts were presented at several research conferences and circulated to methodological experts in various health-related disciplines for feedback. The team systematically recorded, discussed, and incorporated all feedback into further revisions. A penultimate draft was discussed at the 2010 Cochrane–Campbell Collaboration Colloquium to finalize its content.ResultsThe Oxford Implementation Index provides a checklist of implementation data to extract from primary trials. Checklist items are organized into four domains: intervention design, actual delivery by trial practitioners, uptake of the intervention by participants, and contextual factors. Systematic reviewers piloting the index at the Cochrane–Campbell Colloquium reported that the index was helpful for the identification of implementation data.ConclusionThe Oxford Implementation Index provides a framework to help reviewers assess implementation data across trials. Reviewers can use this tool to identify implementation data, extract relevant information, and compare features of implementation across primary trials in a systematic review. The index is a work-in-progress, and future efforts will focus on refining the index, improving usability, and integrating the index with other guidance on systematic reviewing.     

Development of the Cochrane Collaboration's CENTRAL Register of controlled clinical trials

The Cochrane Collaboration has established a centralized database of controlled trials and other studies of health care interventions (called CENTRAL) that serves as the best available resource for all those preparing and maintaining systematic reviews or otherwise searching for trials. CENTRAL is available on The Cochrane Library. This article describes the history and methods of CENTRAL's development and the results of an analysis of the current composition of CENTRAL. As of September 2000, CENTRAL contained almost 300,000 citations to reports of trials, contributed mainly by Cochrane Groups and Centers around the world. Development of CENTRAL has been an ambitious, scholarly undertaking and has resulted in a valuable resource: CENTRAL includes citations to controlled trials that may not be indexed in MEDLINE, EMBASE, or other bibliographic databases; citations published internationally in many languages; and citations that are available only in conference proceedings or other hard-to-access sources.     

Blinding was judged more difficult to achieve and maintain in nonpharmacologic than pharmacologic trials

OBJECTIVE: To compare the feasibility of blinding and the perceived risk of unblinding in trials evaluating pharmacologic (PT) and nonpharmacologic treatments (NPT) of hip or knee osteoarthritis. STUDY DESIGN AND SETTING: Two independent reviewers assessed the feasibility of blinding patients, care providers, and outcome assessors, the perceived risk of unblinding, and whether blinding was reported in 110 reports of randomized controlled trials (RCTs) evaluating PT and NPT in patients with hip or knee osteoarthritis. RESULTS: Blinding was considered to be possible less often in NPT trials than in PT trials for patients (42 vs. 96%; P <.001), care providers (12 vs. 96%; P <.001), and outcome assessors (34 vs. 98%; P <.001). When blinding was judged feasible, the perceived risk of unblinding was more often considered moderate or important in NPT than PT trials for patients (35 vs. 14%, P=.02) and outcome assessors (44 vs. 10%, P=.0004). When blinding was judged feasible, it was reported less often in NPT reports than in PT reports for patients (46 vs. 98%, P <.001), care providers (43 vs. 83%, P=.03), and outcome assessors (72 vs. 98%, P=.0006). CONCLUSION: Blinding appears to be more difficult to achieve and unblinding may occur more often in NPT than PT trials.     

Hand searching the Journal of Epidemiology and Community Health as part of th Cochrane Collaboration.

STUDY OBJECTIVE: To identify randomised controlled trials (RCTs) published in the Journal of Epidemiology and Community Health and to explore the contribution of these to the evaluation of public health issues. DESIGN: Hand searching of the journal by both authors with independent assessment of topics of the reports and of their relevance to the Cochrane Collaboration. Agreement was assessed using kappa scores. SETTING: All papers and letters published in the journal from the first issue to the end of 1994. SUBJECTS: Reports that might be RCTs were collected and classified into seven categories: definitely/probably/possibly RCTs or quasi-RCTs; or none of these. MAIN RESULTS: Eighty two definite RCTs were identified and a further 23 were probably/possibly RCTs or quasi-RCTs. Most reports dealt with health education, drug treatments, or "other" health service interventions. Both authors failed to identify a number of trials on hand searching. CONCLUSIONS: The journal has published many trials of importance to the development of evidence-based public health policy. Hand searching may need to be done independently by more than one person.     

For randomized controlled trials, the quality of reports of complementary and alternative medicine was as good as reports of conventional medicine

OBJECTIVE: To compare the quality of reporting of reports randomized controlled trials (RCTs) published in English and in languages other than English (LOE), and to determine whether there were differences between conventional medicine (CM) and complementary and alternative medicine (CAM) reports. STUDY DESIGN AND SETTING: We examined more than 600 RCTs associated with 125 systematic reviews. We extracted characteristics of each RCT using a standardized data collection form. We assessed quality using the Jadad scale and the adequacy of allocation concealment. RESULTS: There were only minor differences in the quality of reports of RCTs published in English compared with other languages (median quality score of 3 vs. 2, P=.10), and the quality of reports of CAM RCTs was similar to the CM reports (median score of 3 vs. 2, P=.14). There was no effect of language of publication on quality of reporting for CM trials (median score of 2 vs. 2, P=.12). Among CAM trials, however, overall quality scores were higher for reports in English than for reports in other languages (median score of 3 vs. 2, P=.04). CONCLUSION: The overall quality of reports published in languages other than English is similar to that of English-language reports. Moreover, the overall quality of reporting of RCTs of CAM interventions is as good as that for CM interventions.