Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy

ObjectiveTo describe the quality and methods of systematic reviews of physiotherapy interventions, compare Cochrane and non-Cochrane reviews, and establish the interrater reliability of the Overview Quality Assessment Questionnaire (OQAQ) quality assessment tool.Study Design and SettingA survey of 200 published systematic reviews was done. Two independent raters assessed the search strategy, assessment of trial quality, outcomes, pooling, conclusions, and overall quality (OQAQ). The study was carried out in the University research center.ResultsIn these reviews, the five most common databases searched were MEDLINE, EMBASE, Cochrane Library, CINAHL, and Cochrane Review Group Registers. The Cochrane allocation concealment system and Jadad Scale were most frequently used to assess trial quality. Cochrane reviews searched more databases and were more likely to assess trial quality, report dichotomous outcomes for individual trials, and conduct a meta-analysis than non-Cochrane reviews. Non-Cochrane reviews were more likely to conclude that there was a beneficial effect of treatment. Cochrane reviews were of higher quality than non-Cochrane reviews. There has been an increase in the quality of systematic reviews over time. The OQAQ has fair to good interrater reliability.ConclusionThe quality of systematic reviews in physiotherapy is improving, and the use of Cochrane Collaboration procedures appears to improve the methods and quality.     

High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews

BACKGROUND AND OBJECTIVE: Extracting data from primary articles is an essential component in conducting systematic reviews. Incorrect data extraction can lead to false conclusions. The objective of this study was to retrospectively repeat the data extraction in all systematic reviews conducted by the Cochrane Cystic Fibrosis and Genetic Disorders Group. STUDY DESIGN AND SETTING: For each review, data extraction was conducted, by an experienced statistician, for the same publications used by the reviewers. Results were compared with those obtained by the reviewers. RESULTS: Errors were found in 20 of 34 reviews, including incorrect calculations made when converting data in primary articles into data required for the review (2 reviews) and misinterpretation of data that were reported in the primary article (7 reviews). All data-handling errors led to changes in the summary results, but none of these affected the review conclusions. CONCLUSIONS: Important errors were identified in a high proportion of reviews. A variety of problems relating to the reporting of results within a review were identified, but these did not lead to substantial changes in any conclusion.     

Patients view errors more broadly than clinicians

Patients' definition of medical errors include staff not communicating effectively, or not being responsive to patient requests. While 94% of patients believed their medical safety had been good to excellent, 39% experienced at least one error-related concern. "Perception gap" can have a significant impact on efforts to improve patient safety.     

Incorporating data from dissertations in systematic reviews

OBJECTIVES: To determine whether unpublished dissertations are an important source of trials for systematic reviews. METHODS: In a review of infant massage, we identified 17 dissertations. We recorded whether each dissertation was included in the review and, if so, whether it contributed data to any analyses. RESULTS: Thirteen dissertations were excluded, 11 because of poor reporting of methodology. Three dissertations were also published as journal papers. The data from the remaining dissertation were not included in any analysis. Reviewing data on the Cochrane Library, only one of 878 reviews included data from a dissertation that might have changed a review's conclusions. CONCLUSIONS: Searching for and retrieving unpublished dissertations involves considerable time and effort and appears to influence the conclusions of a review only rarely.     

Choice of data extraction tools for systematic reviews depends on resources and review complexity

ObjectiveTo assist investigators planning, coordinating, and conducting systematic reviews in the selection of data-extraction tools for conducting systematic reviews.Study Design and SettingWe constructed an initial table listing available data-collection tools and reflecting our experience with these tools and their performance. An international group of experts iteratively reviewed the table and reflected on the performance of the tools until no new insights and consensus resulted.ResultsSeveral tools are available to manage data in systematic reviews, including paper and pencil, spreadsheets, web-based surveys, electronic databases, and web-based specialized software. Each tool offers benefits and drawbacks: specialized web-based software is well suited in most ways, but is associated with higher setup costs. Other approaches vary in their setup costs and difficulty, training requirements, portability and accessibility, versatility, progress tracking, and the ability to manage, present, store, and retrieve data.ConclusionAvailable funding, number and location of reviewers, data needs, and the complexity of the project should govern the selection of a data-extraction tool when conducting systematic reviews.     

The Cochrane Library: more systematic reviews on nutrition needed

The knowledge and relevance of nutrition as well as the demand for well-funded advices increase. The Cochrane Collaboration plays a leading role within the evidence-based medicine and practice. We advocate therefore more specialized nutritional interest within the Cochrane Collaboration. In case 'Nutrition' needs more attention within the Cochrane Library, one of the first priorities is deciding about whether to include non-randomized studies into the Specialized Register and generating lists of journals to handsearch for such a Specialized Register. Preparatory to these activities an inventory of Nutritional content within the Cochrane Library is needed. We estimate that reviews directly related to nutrition and those of borderline interest to nutrition represent less than 4% of all published reviews in The Cochrane Library.     

Semi-automated sensitivity analysis to assess systematic errors in observational data

BACKGROUND: Published epidemiologic research usually provides a quantitative assessment of random error for effect estimates, but no quantitative assessment of systematic error. Sensitivity analysis can provide such an assessment. METHODS: We describe a method to reconstruct epidemiologic data, accounting for biases, and to display the results of repeated reconstructions as an assessment of error. We illustrate with a study of the effect of less-than-definitive therapy on breast cancer mortality. RESULTS: We developed SAS code to reconstruct the data that would have been observed had a set of systematic errors been absent, and to convey the results. After 4,000 reconstructions of the example data, we obtained a median estimate of relative hazard equal to 1.5 with a 95% simulation interval of 0.8-2.8. The relative hazard obtained by conventional analysis equaled 2.0, with a 95% confidence interval of 1.2-3.4. CONCLUSIONS: Our method of sensitivity analysis can be used to quantify the systematic error for an estimate of effect and to describe that error in figures, tables, or text. In the example, the sources of error biased the conventional relative hazard away from the null, and that error was not accurately communicated by the conventional confidence interval.     

PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews

Objective

To compare the performance of Ovid-MEDLINE vs. PubMed for identifying randomized controlled trials of methotrexate (MTX) in patients with rheumatoid arthritis (RA).

Study Design and Setting

We created search strategies for Ovid-MEDLINE and PubMed for a systematic review of MTX in RA. Their performance was evaluated using sensitivity, precision, and number needed to read (NNR).

Results

Comparing searches in Ovid-MEDLINE vs. PubMed, PubMed retrieved more citations overall than Ovid-MEDLINE; however, of the 20 citations that met eligibility criteria for the review, Ovid-MEDLINE retrieved 17 and PubMed 18. The sensitivity was 85% for Ovid-MEDLINE vs. 90% for PubMed, whereas the precision and NNR were comparable (precision: 0.881% for Ovid-MEDLINE vs. 0.884% for PubMed and NNR: 114 for Ovid-MEDLINE vs. 113 for PubMed).

Conclusion

In systematic reviews of RA, PubMed has higher sensitivity than Ovid-MEDLINE with comparable precision and NNR. This study highlights the importance of well-designed database-specific search strategies.     

Criminalization of medical errors: When is a mistake more than just a mistake¿

This article will review widely publicized cases involving healthcare providers against whom criminal charges have been brought, just at the moment that the idea of medical error disclosure is getting traction. This article will attempt to find similarities or patterns and propose that some level of control can be exercised over these matters. This article will address approaches to the issue that risk managers can take to prepare for possible cases, and ways that cases can be managed once charges have been filed. The author concludes that errors should be treated as what they are — errors, not criminal acts.     

Do Internet interventions for consumers cause more harm than good? A systematic review

Objective  To systematically review the effect of consumer use of online health information on decision-making, attitudes, knowledge, satisfaction and health outcomes and utilization.
Search strategy  Electronic databases searched included the Cochrane Controlled Trials Register, MEDLINE, PREMEDLINE (to 14 March 2001), CINAHL, Australian Medical Index, Health and Society, National Institutes of Health Clinical Trials Database and CenterWatch.
Inclusion criteria  All post-1995 comparative studies (including controlled studies, before and after studies, and interrupted time series analyses) of Internet users vs. non-Internet users and other communications mediums, and Internet characteristics such as e-mail vs. other communication mediums, were included. Outcomes included consumer decision-making, attitudes, knowledge, satisfaction and measurable changes in health status or health utilization.
Data extraction and synthesis  One reviewer screened all papers then two reviewers independently assessed studies against the selection criteria and any discrepancies were resolved by discussion with a third reviewer. No attempt was made to combine the data for further statistical analysis.
Main results  We identified 10 comparative studies. Studies evaluated the effectiveness of using the Internet to deliver a smoking cessation programme, cardiac and nutrition educational programmes, behavioural interventions for headache and weight loss, and pharmacy and augmentative services. All studies showed some positive effects on health outcomes, although the methodological quality of many studies was poor.
Conclusions  Despite widespread consumer Internet use to obtain health-care information, there is almost a complete lack of evidence of any effects this may have on health outcomes.     

Unpublished data can be of value in systematic reviews of adverse effects: methodological overview

ObjectiveTo assess the impact of including unpublished data on adverse effects in systematic reviews.Study Design and SettingWe carried out a systematic review of methodological evaluations that compared the quantitative reporting of adverse effects data between published and unpublished sources, in particular, the frequency, rate, or risk of reported adverse effects. Included studies were sought from 10 databases as well as by checking references, handsearching, searching citations, and contacting experts.ResultsWe identified 6,218 potential articles yielding 10 relevant methodological evaluations.One evaluation found that adverse effects were reported more often in unpublished trials. For anecdotal case reports, two evaluations found a higher frequency of unpublished cases, whereas one study identified a greater number of published cases. Another evaluation indicated that differences in frequency of published and unpublished case reports were topic dependent.A comparison of relative risk estimates from five studies suggested no major systematic variation in risk estimates from published and unpublished studies.ConclusionInclusion of unpublished studies can provide additional adverse effects information and more precise risk estimates. However, there is insufficient evidence to indicate whether inclusion of unpublished studies has a major influence on the pooled risk estimates in meta-analyses of adverse effects.