Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy

ObjectiveTo describe the quality and methods of systematic reviews of physiotherapy interventions, compare Cochrane and non-Cochrane reviews, and establish the interrater reliability of the Overview Quality Assessment Questionnaire (OQAQ) quality assessment tool.Study Design and SettingA survey of 200 published systematic reviews was done. Two independent raters assessed the search strategy, assessment of trial quality, outcomes, pooling, conclusions, and overall quality (OQAQ). The study was carried out in the University research center.ResultsIn these reviews, the five most common databases searched were MEDLINE, EMBASE, Cochrane Library, CINAHL, and Cochrane Review Group Registers. The Cochrane allocation concealment system and Jadad Scale were most frequently used to assess trial quality. Cochrane reviews searched more databases and were more likely to assess trial quality, report dichotomous outcomes for individual trials, and conduct a meta-analysis than non-Cochrane reviews. Non-Cochrane reviews were more likely to conclude that there was a beneficial effect of treatment. Cochrane reviews were of higher quality than non-Cochrane reviews. There has been an increase in the quality of systematic reviews over time. The OQAQ has fair to good interrater reliability.ConclusionThe quality of systematic reviews in physiotherapy is improving, and the use of Cochrane Collaboration procedures appears to improve the methods and quality.     

Inclusion of nonrandomized studies in Cochrane systematic reviews was found to be in need of improvement

ObjectivesNonrandomized studies (NRSs) are considered to provide less reliable evidence for intervention effects. However, these are included in Cochrane reviews, despite discouragement. There has been no evaluation of when and how these designs are used. Therefore, we conducted an overview of current practice.Study Design and SettingWe included all Cochrane reviews that considered NRS, conducting inclusions and data extraction in duplicate.ResultsOf the included 202 reviews, 114 (56%) did not cite a reason for including NRS. The reasons were divided into two major categories: NRS were included because randomized controlled trials (RCTs) are wanted (N = 81, 92%) but not feasible, lacking, or insufficient alone or because RCTs are not needed (N = 7, 8%). A range of designs were included with controlled before-after studies as the most common. Most interventions were nonpharmaceutical and the settings nonmedical. For risk of bias assessment, Cochrane Effective Practice and Organisation of Care Group's checklists were used by most reviewers (38%), whereas others used a variety of checklists and self-constructed tools.ConclusionMost Cochrane reviews do not justify including NRS. When they do, most are not in line with Cochrane recommendations. Risk of bias assessment varies across reviews and needs improvement.     

Clinical reasoning and the occupational therapy curriculum

Several reasoning styles are used by occupational therapists when they evaluate clients' problems. This study investigated the influence of the occupational therapy curriculum in Hong Kong on therapists' clinical reasoning styles. Two groups of therapists with different clinical experience were recruited. Through interviews with the therapists after identifying clients' problems using the Canadian Occupational Performance Measure, their clinical reasoning styles were explored. The local occupational therapy curriculum was analysed to isolate the components that influence clinical reasoning. Results showed that more experienced therapists use conditional reasoning that considers clients' needs in their future lives whereas junior therapists use procedural reasoning that focuses on clients' disabilities. The analysis of the occupational therapy curriculum indicated that it prepared the students with an equal emphasis on theoretical and clinical subjects and fieldwork practice. The present curriculum was useful in providing educational preparation for novice therapists. However, the period of fieldwork practice can be lengthened to allow adequate maturation of clinical reasoning skills. Problem‐based learning can be incorporated to facilitate students' problem‐solving and self‐directed learning skills. Copyright © 2000 Whurr Publishers Ltd.     

A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals

This study set out to compare Cochrane reviews and reviews published in paper-based journals. Two assessment tools were used to collect the data, a 23-item checklist developed by Sacks and a nine-item scale developed by Oxman. Cochrane reviews were found to be better at reporting some items and paper-based review at reporting others. The overall quality was found to be low. This represents a serious situation because clinicians, health policy makers, and consumers are often told that systematic reviews represent "the best available evidence." In the period since this study, the Cochrane Collaboration has taken steps to improve the quality of its reviews through, for example, more thorough prepublication refereeing, developments in the training and support offered to reviewers, and improvements in the system for postpublication peer review. In addition, the use of evidence-based criteria (i.e., the QUOROM statement) for reporting systematic reviews may help further to improve their quality.     

Converting systematic reviews to Cochrane format: a cross-sectional survey of Australian authors of systematic reviews

Background

Despite the growing reputation and subject coverage of the Cochrane Database of Systematic Reviews, many systematic reviews continue to be published solely in paper-based health care journals. This study was designed to determine why authors choose to publish their systematic reviews outside of the Cochrane Collaboration and if they might be interested in converting their reviews to Cochrane format for publication in the Cochrane Database of Systematic Reviews.

Methods

Cross-sectional survey of Australian primary authors of systematic reviews not published on the Cochrane Database of Systematic Reviews identified from the Database of Abstracts of Reviews of Effectiveness.

Results

We identified 88 systematic reviews from the Database of Abstracts of Reviews of Effectiveness with an Australian as the primary author. We surveyed 52 authors for whom valid contact information was available. The response rate was 88 per cent (46/52). Ten authors replied without completing the survey, leaving 36 valid surveys for analysis. The most frequently cited reasons for not undertaking a Cochrane review were: lack of time (78%), the need to undergo specific Cochrane training (46%), unwillingness to update reviews (36%), difficulties with the Cochrane process (26%) and the review topic already registered with the Cochrane Collaboration (21%). (Percentages based on completed responses to individual questions.) Nearly half the respondents would consider converting their review to Cochrane format. Dedicated time emerged as the most important factor in facilitating the potential conversion process. Other factors included navigating the Cochrane system, assistance with updating and financial support. Eighty-six per cent were willing to have their review converted to Cochrane format by another author.

Conclusion

Time required to complete a Cochrane review and the need for specific training are the primary reasons why some authors publish systematic reviews outside of the Cochrane Collaboration. Encouragingly, almost half of the authors would consider converting their review to Cochrane format. Based on the current number of reviews in the Database of Abstracts of Reviews of Effectiveness, this could result in more than 700 additional Cochrane reviews. Ways of supporting these authors and how to provide dedicated time to convert systematic reviews needs further consideration.

     

Clinical decision making, risk and occupational therapy

Little is known about the processes involved in risk taking, and complex decision making of the type encountered on acute medical wards in general hospitals. The purpose of the present study was to examine certain of these processes in the context of decisions to discharge elderly inpatients from hospital. Vignettes of hypothetical frail and disabled elderly in-patients were presented to student and qualified occupational therapists (OTs). All the vignettes represented high risk discharges and each of the patients expressed the desire to return to their own home. The additional diagnosis of early dementia was manipulated within and between vignettes. Subjects were asked to decide on the appropriateness of discharging each patient to (a) home and (b) a residential or nursing home. In addition, subjects completed a short demographic questionnaire which also probed aspects of their knowledge about elderly people. Results indicated that qualified OTs were less likely to overrule patients' wishes by favouring discharge of patients to residential or nursing home, and more likely to favour discharge to patients' own homes than student OTs. There was no overall main effect of dementia on subjects' decisions. In addition to clinical experience, certain areas of knowledge about elderly people were significantly associated with different approaches to risk taking in response to the vignettes. Undergraduate syllabi may need to be modified to incorporate more information about elderly people, the prevalence of different diseases, and direct experience of clinical decision making in the context of uncertain and risky situations.     

Clinical heterogeneity in random-effect meta-analysis: Between-study boundary estimate problem

Random-effect meta-analysis is commonly applied to estimate overall effects with unexplained heterogeneity across studies. However, standard methods, including (restricted) maximum likelihood (ML or REML), frequently produce (near) zero estimates for between-study variance parameters. Consequently, these methods are reduced to simple and unrealistic fixed-effect models, resulting in an ignorance of the substantial clinical heterogeneity and sometimes leading to incorrect conclusions. To solve the boundary estimate problem, we propose (1) an adjusted maximum likelihood method for the between-study variance that maximizes a likelihood defined as a product of a standard likelihood and a Gaussian class of adjustment factor and (2) a framework using sensitivity analysis by developing a new criterion to check for the occurrence of the boundary estimate. Although the adjustment introduces bias to the overall effects to ensure strictly positive estimates of the between-study variance when the number of studies K is small, the bias asymptotically approaches zero, resulting in the same estimates derived from the REML method. Moreover, the adjusted maximum likelihood estimator of the between-study variance is consistent for large K, and interestingly, the REML method and our method are equivalent in terms of mean squared error criterion, up to O(K⁻¹). We illustrate our approach with a motivating example to examine the controversial result of a meta-analysis for 24 randomized controlled trials of human albumin. Numerical evaluations show that our approach produces no boundary estimates but similar synthesized results with the standard maximum likelihood methods as those produced by conventional methods, especially with a small number of studies.     

The Cochrane Library: a resource for current reviews of clinical evidence

The Cochrane Library is a tool for physicians and other health care practitioners seeking evidence to inform their decisions. The systematic reviews provide a high-quality synthesis of the current literature, saving time for busy people. To make the library more user-friendly, the collaboration plans to produce a consumer summary for each topic in the Database of Systematic Reviews and a cancer library aimed at the general public. In contrast to some of the questionable health-related resources on the Web, the Cochrane Library is an authoritative reference that can help physicians with everyday treatment decisions. The collaboration's commitment to keeping its resources up to date through vigorous support of reviewers should ensure that it remains a valuable Internet tool for physicians.     

Investigating causes of heterogeneity in systematic reviews

What causes heterogeneity in systematic reviews of controlled trials? First, it may be an artefact of the summary measures used, of study design features such as duration of follow-up or the reliability of outcome measures. Second, it may be due to real variation in the treatment effect and hence provides the opportunity to identify factors that may modify the impact of treatment. These factors may include features of the population such as: severity of illness, age and gender; intervention factors such as dose, timing or duration of treatment; and comparator factors such as the control group treatment or the co-interventions in both groups. The ideal way to study causes of true variation is within rather than between studies. In most situations however, we will have to make do with a study level investigation and hence need to be careful about adjusting for potential confounding by artefactual factors such as study design features. Such investigation of artefactual and true causes of heterogeneity form essential steps in moving from a combined effect estimate to application to particular populations and individuals.     

Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information

ObjectiveTo measure the effects of a summary-of-findings (SoF) table on user satisfaction, understanding, and time spent finding key results in a Cochrane review.Study Design and SettingWe randomized participants in an evidence-based practice workshop (randomized controlled trial [RCT] I) and a Cochrane Collaboration entities meeting (RCT II) to receive a Cochrane review with or without an SoF table. In RCT I, we measured user satisfaction. In RCT II, we measured correct comprehension and time spent finding key results.ResultsRCT I: Participants with the SoF table (n = 47) were more likely to “agree” or “strongly agree” that it was easy to find results for important outcomes than (n = 25) participants without the SoF table—68% vs. 40% (P = 0.021). RCT II: Participants with the SoF table (n = 18) were more likely to correctly answer two questions regarding results than (n = 15) those without the SoF table: 93% vs. 44% (P = 0.003) and 87% vs. 11% (P < 0.001). Participants with the SoF table spent an average of 90 seconds to find key information compared with 4 minutes for participants without the SoF table (P = 0.002).ConclusionIn two small trials, we found that inclusion of an SoF table in a review improved understanding and rapid retrieval of key findings compared with reviews with no SoF table.     

Core competencies for health care professionals: what medicine, nursing, occupational therapy, and physiotherapy share

This paper describes the amalgamation of the core competencies identified for medicine, nursing, physical therapy, and occupational therapy and the "harmonization" of these competencies into a framework for interprofessional education. The study was undertaken at a Canadian university with a Faculty of Health Sciences comprised of three schools (namely, medicine, nursing, and rehabilitation therapy). Leaders in interprofessional education began to identify the common standards for the core competencies expected of learners in all three schools at commensurate levels to facilitate the integration of educational curricula aimed at interprofessional education across the Faculty. The model that was created serves as a basis for curriculum design and assessment of individuals and groups of learners from different domains across and within the four professions. It particularly highlights the relevance of cross-disciplinary competency teaching and 360-degree evaluation in teams. Most importantly, it provides a launch pad for clarifying performance standards and expectations in interdisciplinary learning.