The art of quality assessment of RCTs included in systematic reviews.

The best evidence on the efficacy of medical interventions is provided by high-quality trials summarized in high-quality systematic reviews or meta-analyses. The methodological quality of studies included in a systematic review can have a substantial impact on the estimates of the treatment effect and therefore on the conclusions of such a review. But what is the empirical evidence to support quality assessment of randomized clinical trials (RCTs)? We elaborate on questions such as: what is the concept of quality of individual studies (RCTs), can quality be measured validly and reliably? Plans for future research on this issue are proposed.     

An encouraging assessment of methods to inform priorities for updating systematic reviews

ObjectiveTo consider the use of statistical methods that aim to prioritize the updating of a collection of systematic reviews based on preliminary literature searches.Study Design and SettingA new simulation-based method estimating statistical power and the ratio of the weights assigned to the predicted new and old evidence, and the existing Barrowman n approach is considered. Using only information on the numbers of subjects randomized in the “new” trials, these were applied retrospectively, by removing recent studies, to existing systematic reviews from the Cochrane Infectious Diseases Group.ResultsTwelve systematic reviews were included. When the removed studies were reinstated, inferences changed in five of them. These reviews were ranked, in order of update priority, 1, 2, 3, 4, and 11 and 1, 2, 3, 4, and 12 by the Barrowman n and simulation-based power approaches, respectively. The low ranking of one significant meta-analysis by both methods was due to unexpectedly favorable results in the reinstated study.ConclusionThis study demonstrates the feasibility of the use of analytical methods to inform update prioritization strategies. Under conditions of homogeneity, Barrowman's n and simulated power were in close agreement. We encourage further, prospective, evaluation of these methods.     

Health-related quality of life after fast-track treatment results from a randomized controlled clinical equivalence trial

Purpose

This randomized clinical equivalence trial was designed to evaluate health-related quality of life (HRQoL) after fast-track treatment for low-risk coronary artery bypass (CABG) patients.

Methods

Four hundred and ten CABG patients were randomly assigned to undergo either short-stay intensive care treatment (SSIC, 8 h of intensive care stay) or control treatment (care as usual, overnight intensive care stay). HRQoL was measured at baseline and 1 month, and one year after surgery using the multidimensional index of life quality (MILQ), the EQ-5D, the Beck Depression Inventory and the State-Trait Anxiety Inventory.

Results

At one month after surgery, no statistically significant difference in overall HRQoL was found (MILQ-score P-value = .508, overall MILQ-index P-value = .543, EQ-5D VAS P-value = .593). The scores on the MILQ-domains, physical, and social functioning were significantly higher at one month postoperatively in the SSIC group compared to the control group (P-value = .049; 95%CI: 0.01–2.50 and P-value = .014, 95% CI: 0.24–2.06, respectively). However, these differences were no longer observed at long-term follow-up.

Conclusions

According to our definition of clinical equivalence, the HRQoL of SSIC patients is similar to patients receiving care as usual. Since safety and the financial benefits of this intervention were demonstrated in a previously reported analysis, SSIC can be considered as an adequate fast-track intensive care treatment option for low-risk CABG patients.     

Practice of systematic reviews. VII. Pooling of results from observational studies

Knowledge concerning aetiologic hypotheses can be obtained through systematic reviews of observational studies. In observational studies, heterogeneity between studies is expected, because of differences in study design and analysis. Data from observational studies have to be made comparable before pooling of results from component studies is possible. In this, definition of exposure and disease and adjustment for confounding have to be taken into account. Three methods to deal with heterogeneity are presented: ignoring (fixed effects model), modelling (random effects model) and exploring (meta regression). Any combination of data in meta-analysis of observational studies has to be based on statistical, methodological and clinical considerations. The same considerations are applied for the choice which method is used.     

Methodological quality of systematic reviews on influenza vaccination

Background

There is a growing body of evidence on the risks and benefits of influenza vaccination in various target groups. Systematic reviews are of particular importance for policy decisions. However, their methodological quality can vary considerably.

Objectives

To investigate the methodological quality of systematic reviews on influenza vaccination (efficacy, effectiveness, safety) and to identify influencing factors.

Methods

A systematic literature search on systematic reviews on influenza vaccination was performed, using MEDLINE, EMBASE and three additional databases (1990–2013). Review characteristics were extracted and the methodological quality of the reviews was evaluated using the assessment of multiple systematic reviews (AMSTAR) tool. U-test, Kruskal–Wallis test, chi-square test, and multivariable linear regression analysis were used to assess the influence of review characteristics on AMSTAR-score.

Results

Fourty-six systematic reviews fulfilled the inclusion criteria. Average methodological quality was high (median AMSTAR-score: 8), but variability was large (AMSTAR range: 0–11). Quality did not differ significantly according to vaccination target group. Cochrane reviews had higher methodological quality than non-Cochrane reviews (p = 0.001). Detailed analysis showed that this was due to better study selection and data extraction, inclusion of unpublished studies, and better reporting of study characteristics (all p < 0.05). In the adjusted analysis, no other factor, including industry sponsorship or journal impact factor had an influence on AMSTAR score.

Conclusions

Systematic reviews on influenza vaccination showed large differences regarding their methodological quality. Reviews conducted by the Cochrane collaboration were of higher quality than others. When using systematic reviews to guide the development of vaccination recommendations, the methodological quality of a review in addition to its content should be considered.     

The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus

Most systematic reviews rely substantially on the assessment of the methodological quality of the individual trials. The aim of this study was to obtain consensus among experts about a set of generic core items for quality assessment of randomized clinical trials (RCTs). The invited participants were experts in the field of quality assessment of RCTs. The initial item pool contained all items from existing criteria lists. Subsequently, we reduced the number of items by using the Delphi consensus technique. Each Delphi round comprised a questionnaire, an analysis, and a feedback report. The feedback report included staff team decisions made on the basis of the analysis and their justification. A total of 33 international experts agreed to participate, of whom 21 completed all questionnaires. The initial item pool of 206 items was reduced to 9 items in three Delphi rounds. The final criteria list (the Delphi list) was satisfactory to all participants. It is a starting point on the way to a minimum reference standard for RCTs on many different research topics. This list is not intended to replace, but rather to be used alongside, existing criteria lists.     

Identifying studies for systematic reviews. An example from medical imaging

OBJECTIVES: To determine if published figures on the proportion of articles included in systematic reviews and identified in electronic databases are applicable to an example from medical imaging. METHODS: A systematic review was performed. Additionally, sensitivity and precision of a MEDLINE search were compared with values from three published searches, each customized for a specific field. RESULTS: All articles included in the systematic review were in electronic databases. The MEDLINE search had low precision compared with searches in other fields. CONCLUSIONS: In a specific area of medical imaging, electronic databases, including MEDLINE, are reliable sources of articles.     

The practice of systematic reviews. IV. Pooling results from separate studies]

Meta-analysis comprises the analysis of the data gathered in a systematic review. Meta-analysis can be defined as the combination of pooling results of similar studies into one overall estimate of the effect on the one hand and the exploration of sources of heterogeneity between studies on the other. A general statistical model is available for pooling the estimates of various types of effect parameters. Pooling enables the estimation of the effect of an intervention or exposure as precisely as possible (with narrow confidence intervals) which allows detection of minor effects. The Mantel-Haenszel method is also an excellent method for pooling effect parameters of dichotomous outcomes. Pooling provides an efficient means of summarizing the available knowledge.     

Endocrine clinical practice guidelines in North America. A systematic assessment of quality

ObjectiveTo assess the quality of endocrine guidelines developed in North America.Study Design and SettingA systematic review of the literature was conducted to identify all endocrine clinical practice guidelines developed in North America and published between January 1, 2007 and January 13, 2010. Two independent reviewers used the Appraisal of Guidelines, Research and Evaluation instrument to evaluate the quality of the guidelines in six domains: scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, and editorial independence.ResultsOne hundred eligible endocrine guidelines had high scores in the scope-and-purpose (mean pooled standardized score [MPSD] of 82 ± 14) and clarity domains (MPSD = 64 ± 17) and low scores in the stakeholder-involvement (MPSD of 36 ± 12) and editorial independence domains (MPSD = 36 ± 36). Only 29% of guidelines scored above 60% for more than three domains. Rigor-of-development domain score was significantly higher in guidelines using the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) approach, nondiabetes guidelines, and in published in-print vs. online publications.ConclusionsThe quality of endocrine guidelines published in 2007–2009 is moderate and can be improved by (1) using methodologically sound development frameworks, (2) increasing stakeholder involvement, and (3) paying more attention to resource implications of guideline implementation.     

Measures of success of computerized clinical decision support systems: An overview of systematic reviews

IntroductionAlthough increasing number of clinical decision support systems (CDSSs) were introduced, a lack of rigorous evaluations of information systems (IS) success limits these systems’ usage and diffusion. The purpose of this overview is to identify specific features which are critical to information system success based on the updated DeLone & McLean IS Success Model from published systematic reviews (SRs) over the last 10 years.MethodsThis systematic review was prepared in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.ResultsThere were 39 measures were synthesized from 45 SRs published during 2009–2020 and were classified into 16 categories and 6 dimensions. The most frequently displayed dimension was “net benefit”, which was divided into four categories: “process of care”, “professional competency”, “patient outcomes” and “cost-effectiveness”. Very limited publications summarized evaluation metrics under the remaining five dimensions, excepted “validity” in “system performance” category and “acceptance” in “use behavior” category.ConclusionThe findings of this overview may have meaningful and useful implications to the evaluation of CDSSs. Future research needs to focus on developing a comprehensive evaluation framework through summarizing evaluation metrics on the base of theoretical model, which is critical to CDSSs development, utilization and research. Furthermore, multi-centered, large sample size and long-term follow up studies are required to ensure more reliable evidence on patient outcomes of CDSSs usage.     

The quality of systematic reviews of health-related outcome measurement instruments

Background

Systematic reviews of outcome measurement instruments are important tools for the selection of instruments for research and clinical practice. Our aim was to assess the quality of systematic reviews of health-related outcome measurement instruments and to determine whether the quality has improved since our previous study in 2007.

Methods

A systematic literature search was performed in MEDLINE and EMBASE between July 1, 2013, and June 19, 2014. The quality of the reviews was rated using a study-specific checklist.

Results

A total of 102 reviews were included. In many reviews the search strategy was considered not comprehensive; in only 59 % of the reviews a search was performed in EMBASE and in about half of the reviews there was doubt about the comprehensiveness of the search terms used for type of measurement instruments and measurement properties. In 41 % of the reviews, compared to 30 % in our previous study, the methodological quality of the included studies was assessed. In 58 %, compared to 55 %, the quality of the included instruments was assessed. In 42 %, compared to 7 %, a data synthesis was performed in which the results from multiple studies on the same instrument were somehow combined.

Conclusion

Despite a clear improvement in the quality of systematic reviews of outcome measurement instruments in comparison with our previous study in 2007, there is still room for improvement with regard to the search strategy, and especially the quality assessment of the included studies and the included instruments, and the data synthesis.

     

Bayesian cost-effectiveness analysis from clinical trial data

A key tool for assessing the relative cost-effectiveness of two treatments in health economics is the incremental C/E acceptability curve. We present Bayesian computations for this curve in the case where data on both costs and efficacy are available from a clinical trial. Analysis is given under various formulations of prior information. A case study is analysed in which reasonable prior information is shown to strengthen substantially the posterior inference, leading to a more conclusive assessment of cost-effectiveness. Calculations can be performed using readily available Bayesian software.     

An analysis of ambulatory care quality assessment research.

This article offers a framework that distinguishes quality as an attribute of the processes of medical care from effectiveness as the results of that care. The review of process-outcome studies in the light of this framework reveals not only a confusion of concepts but a variety of methodological and measurement problems. Among the measurement problems is the lack of assessment of the reliability and validity of the process and outcome measures in most quality studies in ambulatory medical care. This article proposes a minimum set of elements to be considered in developing a valid quality assessment method.     

Quality assessment of systematic reviews on international migrant healthcare interventions: a systematic review

Journal of Public Health - The significant increase in international cross-border migrations remains a global public health concern; therefore, the need for migrant-specific evidence-based...     

A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals

This study set out to compare Cochrane reviews and reviews published in paper-based journals. Two assessment tools were used to collect the data, a 23-item checklist developed by Sacks and a nine-item scale developed by Oxman. Cochrane reviews were found to be better at reporting some items and paper-based review at reporting others. The overall quality was found to be low. This represents a serious situation because clinicians, health policy makers, and consumers are often told that systematic reviews represent "the best available evidence." In the period since this study, the Cochrane Collaboration has taken steps to improve the quality of its reviews through, for example, more thorough prepublication refereeing, developments in the training and support offered to reviewers, and improvements in the system for postpublication peer review. In addition, the use of evidence-based criteria (i.e., the QUOROM statement) for reporting systematic reviews may help further to improve their quality.     

Quality assessment of observational studies is not commonplace in systematic reviews

BACKGROUND AND OBJECTIVE: To review current practice in the assessment of the quality of original observational studies included in systematic reviews. MATERIALS AND METHODS: Examination of all systematic reviews identified by a basic PubMed search for the years 1999-2000 (32 reviews) and 2003-2004 (98 reviews). English language systematic reviews published in peer-reviewed journals was the setting. Each review was evaluated for the use of quality assessment of original observational studies and if quality assessment occurred, what type of assessment was used. RESULTS: Quality assessment occurred in 22% of systematic reviews identified in 1999-2000 compared with 50% of reviews identified from 2003-2004. All earlier reviews devised their own quality assessment criteria, whereas in 2003-2004 10 different quality assessment techniques were identified. CONCLUSIONS: Quality assessment does not routinely occur in systematic reviews of observational studies. Where it does occur, there is no clear consensus in the method used.