The value of including observational studies in systematic reviews was unclear: a descriptive study

ObjectivesTo evaluate (1) how often observational studies are included in comparative effectiveness reviews (CERs); (2) the rationale for including observational studies; (3) how data from observational studies are appraised, analyzed, and graded; and (4) the impact of observational studies on strength of evidence (SOE) and conclusions.Study Design and SettingDescriptive study of 23 CERs published through the Effective Health Care Program of the U.S. Agency for Healthcare Research and Quality.ResultsAuthors searched for observational studies in 20 CERs, of which 18 included a median of 11 (interquartile range, 2–31) studies. Sixteen CERs incorporated the observational studies in their SOE assessments. Seventy-eight comparisons from 12 CERs included evidence from both trials and observational studies; observational studies had an impact on SOE and conclusions for 19 (24%) comparisons. There was diversity across the CERs regarding decisions to include observational studies; study designs considered; and approaches used to appraise, synthesize, and grade SOE.ConclusionReporting and methods guidance are needed to ensure clarity and consistency in how observational studies are incorporated in CERs. It was not always clear that observational studies added value in light of the additional resources needed to search for, select, appraise, and analyze such studies.