Systematic reviewers commonly contact study authors but do so with limited rigor

ObjectivesAuthor contact can enhance the quality of systematic reviews. We conducted a systematic review of the practice of author contact in recently published systematic reviews to characterize its prevalence, quality, and results.Study Design and SettingEligible studies were systematic reviews of efficacy published in 2005–2006 in the 25 journals with the highest impact factor publishing systematic reviews in clinical medicine and the Cochrane Library, identified by searching MEDLINE, EMBASE, and the Cochrane Library. Two researchers determined whether and why reviewers contacted authors. To assess the accuracy of the abstracted data, we surveyed reviewers by e-mail.ResultsForty-six (50%) of the 93 eligible systematic reviews published in top journals and 46 (85%) of the 54 eligible Cochrane reviews reported contacting authors of eligible studies. Requests were made most commonly for missing information: 40 (76%) clinical medicine reviews and 45 (98%) Cochrane reviews. One hundred and nine of 147 (74%) reviewers responded to the survey, and reported a higher rate of author contact than apparent from the published record.ConclusionAlthough common, author contact is not a universal feature of systematic reviews published in top journals and the Cochrane Library. The conduct and reporting of author contact purpose, procedures, and results require improvement.     

Diagnostic test systematic reviews: Bibliographic search filters (“Clinical Queries”) for diagnostic accuracy studies perform well

BackgroundSystematic reviews of health care topics are valuable summaries of all pertinent studies on focused questions. However, finding all relevant primary studies for systematic reviews remains challenging.ObjectivesTo determine the performance of the Clinical Queries sensitive search filter for diagnostic accuracy studies for retrieving studies for systematic reviews.MethodsWe compared the yield of the sensitive Clinical Queries diagnosis search filter for MEDLINE and EMBASE to retrieve studies in diagnostic accuracy systematic reviews reported in ACP Journal Club in 2006.ResultsTwelve of 22 diagnostic accuracy reviews (452 included studies) met the inclusion criteria. After excluding 11 studies not in MEDLINE or EMBASE, 95% of articles (417 of 441) were captured by the sensitive Clinical Queries diagnosis search filter (MEDLINE and EMBASE combined). Of 24 studies not retrieved by the filter, 22 were not diagnostic accuracy studies. Reanalysis of the Clinical Queries filter without these 22 nondiagnosis articles increased its performance to 99% (417 of 419). We found no substantive impact of the two articles missed by the Clinical Queries filter on the conclusions of the systematic reviews in which they were cited.ConclusionThe sensitive Clinical Queries diagnostic search filter captured 99% of articles and 100% of substantive articles indexed in MEDLINE and EMBASE in diagnostic accuracy systematic reviews.     

Exploratory Evaluation of Medication Classes Most Commonly Involved in Nursing Home Errors

BackgroundMedication errors may potentially pose significant risk of harmful outcomes in vulnerable nursing home residents. Current literature lacks data regarding the drug classes most frequently involved in errors in this population and their risk relative to underlying drug class utilization rates.ObjectivesThis study (1) describes the frequency and error characteristics for the drug classes most commonly involved in medication errors in nursing homes, and (2) examines the correlation between drug class utilization rates and their involvement in medication errors in nursing home residents.DesignA cross-sectional analysis of individual medication error incidents reported by North Carolina nursing homes to the Medication Error Quality Initiative during fiscal years 2010 to 2011 was conducted.ParticipantsAll nursing home residents in the state of North Carolina.Main measuresThe 10 drug classes most frequently involved in medication errors were identified. Characteristics and patient impact of these medication errors were further examined as frequencies and proportions within each drug class. Medication error data were combined with data from the 2004 National Nursing Home Survey to capture nationally representative estimates of medication use by drug class in nursing home patients. The correlation between medication utilization and error involvement was assessed.ResultsThere were 32,176 individual medication errors reported to Medication Error Quality Initiative in years 2010–2011. The 10 drug classes most commonly involved in medication errors were analgesics (12.27%), anxiolytics/sedative/hypnotics (8.39%), antidiabetic agents (5.86%), anticoagulants (5.04%), anticonvulsants (4.05%), antidepressants (4.05%), laxatives (3.13%), ophthalmic preparations (2.77%), antipsychotics (2.47%), and diuretics (2.34%). The correlation between utilization and medication error involvement was not statistically significant (P value for spearman correlation coefficient = .88), suggesting certain drug classes are more likely to be involved in medication errors in nursing home patients regardless of the extent of their use.ConclusionsThe drug classes frequently and disproportionately involved in errors in nursing homes include anxiolytics/sedatives/hypnotics, antidiabetic agents, anticoagulants, anticonvulsants, and ophthalmic preparations. Better understanding of the causes and prevention strategies to reduce these errors may improve nursing home patient safety.     

Meta-epidemiologic study showed frequent time trends in summary estimates from meta-analyses of diagnostic accuracy studies

ObjectivesTo evaluate changes over time in summary estimates from meta-analyses of diagnostic accuracy studies.Study Design and SettingWe included 48 meta-analyses from 35 MEDLINE-indexed systematic reviews published between September 2011 and January 2012 (743 diagnostic accuracy studies; 344,015 participants). Within each meta-analysis, we ranked studies by publication date. We applied random-effects cumulative meta-analysis to follow how summary estimates of sensitivity and specificity evolved over time. Time trends were assessed by fitting a weighted linear regression model of the summary accuracy estimate against rank of publication.ResultsThe median of the 48 slopes was −0.02 (−0.08 to 0.03) for sensitivity and −0.01 (−0.03 to 0.03) for specificity. Twelve of 96 (12.5%) time trends in sensitivity or specificity were statistically significant. We found a significant time trend in at least one accuracy measure for 11 of the 48 (23%) meta-analyses.ConclusionTime trends in summary estimates are relatively frequent in meta-analyses of diagnostic accuracy studies. Results from early meta-analyses of diagnostic accuracy studies should be considered with caution.     

Patient partners’ perspectives of meaningful engagement in synthesis reviews: A patient‐oriented rapid review

BackgroundA growing literature describes promising practices for patient‐oriented research (POR) generally; however, those for systematic reviews are largely derived through the lens of a researcher. This rapid review sought to understand meaningful engagement in synthesis reviews from the patient partner (PP) perspective.DesignThe review team comprised PPs, librarians, SCPOR staff and academic faculty. We searched OVID MEDLINE and EMBASE, ProQuest Nursing and Allied Health, and core POR websites. Documents describing PP reflections on their involvement in synthesis reviews were included. Screening and data extraction were conducted by two independent reviewers. Thematic analysis was employed to identify themes in the data regarding PP perceptions of engagement in synthesis reviews.ResultsThe literature search yielded 1386 citations. Eight journal articles and one blog post were included. Seven studies focused on conducting systematic reviews on a particular health or patient‐related topic to which PP involvement was an important part and two studies focused specifically on the experience of including PP in synthesis reviews. PPs engaged in the review process through a variety of mechanisms, levels and stages of the review process. Three major themes emerged from the data: (1) foster partnerships through team development, (2) provide opportunities for outcomes valued by PP and (3) strengthen the research endeavour.ConclusionFostering partnerships through team development is foundational for meaningful engagement in synthesis reviews. It requires sensitively balancing of various needs (eg overburdening with contributions). Meaningful involvement in reviews has both personal and research benefits.Patient InvolvementPatient partners were equal collaborators in all aspects of the review.     

Prevalence and types of vaccination errors from 2009 to 2018: A systematic review of the medical literature

IntroductionVaccination practices and the programmatic factors that influence them are essential for public health. Several barriers impact vaccination efforts, including vaccination errors, which pose the risk of reduced population-wide vaccination efficacy and individual adverse drug events. This study aimed to define the prevalence of vaccination errors documented in English language medical literature between 2009 and 2018 and to identify the common types of errors that occurred during this period.MethodsThis systematic review followed the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) guidelines. The study protocol was registered with the International Prospective Register of Systematic Reviews prior to research activities. The Cochrane Library, Cumulative Index to Nursing and Allied Health Literature, Google Scholar, ProQuest Central, PubMed, Scopus, and Web of Science were searched using predetermined search terms. Included data were from primary studies or retrospective analyses that assessed the prevalence and/or type of vaccination errors and that were peer-reviewed, conducted between 2009 and 2018, and published in English. Data were extracted using the Cochrane Data Extraction and Assessment Template and assessed using the Appraisal tool for Cross-Sectional Studies. Pooled vaccination error prevalence was then calculated.ResultsOf the 1310 independent records that were identified and screened, 17 studies from five countries met all inclusion criteria. Pooled vaccination error prevalence was calculated to be 1.15 per 10,000 vaccine doses (range, 0.005–141.69 per 10,000 doses). The most commonly reported vaccination errors were “wrong vaccine administered” and “off-schedule administration.”ConclusionsInternational rates of vaccination error reporting remain low, with few reports of significant adverse reactions. Vaccination programs should consider the impact of vaccination errors on individual and population health, particularly focusing on the impact of “wrong vaccine” administration. Continued monitoring and promotion of error reporting will enable further understanding of this topic.     

Mapping Studies to Estimate Health-State Utilities From Nonpreference-Based Outcome Measures: A Systematic Review on How Repeated Measurements are Taken Into Account

ObjectivesMapping algorithms are developed using data sets containing patient responses to a preference-based questionnaire and another health-related quality-of-life questionnaire. When data sets include repeated measurements from the same individuals over time, the assumption of observations’ independence, required by standard models, is violated, and standard errors are underestimated. This review aimed to identify how studies deal with methodological challenges of repeated measurements, provide an overview of practice to date, and potential implications for future work.MethodsWe conducted a systematic literature search of MEDLINE, Cumulative Index to Nursing and Allied Health Literature, specialized databases, and previous systematic reviews. A data template was used to extract, among others, start and target instruments if the data set(s) used for estimation and validation had repeated measurements per patient, used regression techniques, and which (if any) adjustments were made for repeated measurements.ResultsWe identified 278 publications developing at least 1 mapping algorithm. Of the 278 publications, 121 used a data set with repeated measurements, among which 92 used multiple time points for estimation, and 39 selected specific time points to have 1 observation per participant. A total of 36 studies did not account for repeated measurements. An adjustment was conducted using cluster-robust standard errors (21), random-effects models (30), generalized estimating equations (7), and other methods (7).ConclusionsThe inconsistent use of methods to account for interdependent observations in the literature indicates that mapping guidelines should include recommendations on how to deal with repeated measurements, and journals should update their guidelines accordingly.     

The elderly were under-represented in osteoarthritis clinical trials

ObjectivesOsteoarthritis is the most common disease affecting joints in the elderly. We aimed to evaluate if elderly patients are properly represented in clinical trials of diverse osteoarthritis interventions.Study Design and SettingClinical trials of osteoarthritis interventions were retrieved from Cochrane Library systematic reviews (2006, issue 2). We examined the age distribution of the trial participants and eligibility criteria.ResultsWe analyzed data from 219 eligible trials from 18 systematic reviews. The average mean age of the participants was 63 years. Only 13 trials (6.4%) had a mean age between 71 and 80 years and only one trial had a mean age exceeding 80 years. Among trials where the age range of participants was available or could be approximately inferred, we estimated that 66 (38%) trials had not included any patients over 80 years old. Only 23 trials specifically excluded patients over 70 based on reported eligibility criteria, but 168 trials excluded patients with various comorbidities and 142 trials excluded patients receiving other specific treatments.ConclusionsElderly patients are considerably under-represented in clinical trials of osteoarthritis. This causes an important deficit in the utility, relevance, and generalizability of trial results for this very common condition.     

Vaccine administration error rates at a large academic medical center and its affiliated clinics – Familiarity matters

ObjectiveThe purpose of this study was to track and describe the absolute number of vaccine administration errors and corresponding error rates over time and by patient age and vaccine type.MethodsTotal vaccines administered to patients aged 0 through 19 years 364 days from 1/1/2006 through 12/31/2017 at a large academic health system in the Midwest United States with primary, specialty and school-based clinics, and a pediatric hospital were obtained from an electronic medical record. Vaccine administration errors over the same time period for the same patient criteria were analyzed from the health system’s incident reporting system and further compared to the frequency of all incidents reported. Vaccine administration error rates were calculated. Data were analyzed by patient age, vaccine type and year administered.ResultsOf the 1,431,206 vaccine doses given, 552 vaccine administration errors were identified (0.04%). The highest error rates occurred in children aged 2, 3, and 19 years. Vaccine types with the highest error rate were Td, rabies and pneumococcal polysaccharide vaccines. Overall vaccine doses given and errors reported increased over the study period. However, the increase was disproportionate, resulting in an increase in the error rate initially followed by a stabilization at the end of the study period.ConclusionsVaccine administration errors are uncommon. The error rate appears to be stabilizing. Errors are more likely at ages when vaccines are not commonly given, with vaccines that have age-specific dosing and with vaccines that are given less often. This suggests more safety checks are needed for vaccines that are rarely used or given off-schedule, and manufacturers should avoid vaccines with age-specific dosing.     

Addressing continuous data for participants excluded from trial analysis: a guide for systematic reviewers

BackgroundNo methods directly address the impact of missing participant data for continuous outcomes in systematic reviews on risk of bias.MethodsWe conducted a consultative, iterative process to develop a framework for handling missing participant data for continuous outcomes. We considered sources reflecting real observed outcomes in participants followed-up in individual trials included in the systematic review, and developed a range of plausible strategies. We applied our approach to two systematic reviews.ResultsWe used five sources of data for imputing the means for participants with missing data. To impute standard deviation (SD), we used the median SD from the control arms of all included trials. Using these sources, we developed four progressively more stringent imputation strategies. In the first example review, effect estimates diminished and lost significance as strategies became more stringent, suggesting rating down confidence in estimates of effect for risk of bias. In the second, effect estimates maintained statistical significance using even the most stringent strategy, suggesting missing data does not undermine confidence in results.ConclusionsOur approach provides a useful, reasonable, and relatively simple, quantitative guidance for judging the impact of risk of bias as a result of missing participant data in systematic reviews of continuous outcomes.     

识别诊断试验准确性系统综述的方法学异质性

目的 分析诊断试验系统综述的方法学异质性来源。方法 通过检索2008年1月1日至2012年12月31日发表在Medline、Embase、Cochrane(CDSR和DARE)数据库中关于诊断试验准确性(DTA)的Meta分析文献,纳入了至少包含10篇原始研究的Meta分析。两名人员独立地对研究特点以及原始研究的数据进行提取,使用混合线性模型对14种潜在的方法学异质性因素来源进行探讨,进而计算出14种异质性因素的诊断比值比(DOR)之比(RDOR)值及其95% CI,从而判断其异质性大小和方向。结果 最终纳入了23篇DTA的系统综述,涵盖550篇原始研究。纳入的550篇原始研究的质量良好。单因素混合线性模型分析显示,"金标准是否足够准确"和"金标准与待检试验是否相互独立"是DTA系统综述的异质性来源。多因素混合线性模型分析显示,金标准不准确的诊断试验,其DOR合并值低于金标准足够准确的诊断试验,RDOR=0.018 6(95% CI:0.001 0~0.358 5);金标准与待检试验不独立的诊断试验,其DOR合并值高于金标准与待检试验相互独立的诊断试验,RDOR=2.396 6(95% CI:1.242 8~4.622 7)。结论 对于诊断试验系统综述,原始研究的金标准是否足够准确、金标准与待检试验是否相互独立为其方法学异质性的来源。     

Quality scores do not predict discrepant statistical significances among meta-analyses on different targets of glycemic control in type 2 diabetes

ObjectivesTo examine differences in conclusions, statistical significances, and quality of systematic reviews on preventive effects of different targets of blood glucose lowering on macrovascular events in patients with type 2 diabetes.Study Design and SettingWe searched MEDLINE, the Cochrane Database of Systematic Reviews, and Embase until October 15, 2011. Systematic reviews using meta-analyses to compare macrovascular events in patients with type 2 diabetes randomized to different therapeutic targets of blood glucose were eligible for inclusion in this study. Effect measures were extracted and quality was assessed by Overview Quality Assessment Questionnaire, Assessment of Multiple Systematic Reviews, and Preferred Reporting Items for Systematic Reviews and Meta-Analyses. An additional checklist was used to control for completeness and suitability of included trials, correctness of data extraction, and choice of outcome parameters.ResultsThe 16 included systematic reviews were partially discrepant in statistical significances of reported outcome parameters. Twelve systematic reviews did not include available relevant trials and included eight systematic reviews that should not have been included according to the systematic review's aim and search strategies. Quality differed considerably among systematic reviews.ConclusionPhysicians, researchers, and policy makers should bear in mind that common quality assessment instruments are necessary but not sufficient to guarantee reliable results and conclusions obtained from meta-analyses.     

High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews

BACKGROUND AND OBJECTIVE: Extracting data from primary articles is an essential component in conducting systematic reviews. Incorrect data extraction can lead to false conclusions. The objective of this study was to retrospectively repeat the data extraction in all systematic reviews conducted by the Cochrane Cystic Fibrosis and Genetic Disorders Group. STUDY DESIGN AND SETTING: For each review, data extraction was conducted, by an experienced statistician, for the same publications used by the reviewers. Results were compared with those obtained by the reviewers. RESULTS: Errors were found in 20 of 34 reviews, including incorrect calculations made when converting data in primary articles into data required for the review (2 reviews) and misinterpretation of data that were reported in the primary article (7 reviews). All data-handling errors led to changes in the summary results, but none of these affected the review conclusions. CONCLUSIONS: Important errors were identified in a high proportion of reviews. A variety of problems relating to the reporting of results within a review were identified, but these did not lead to substantial changes in any conclusion.     

Prenotification but not envelope teaser increased response rates in a bulimia nervosa mental health literacy survey: A randomized controlled trial

ObjectivesUnderstanding attitudes to mental health issues can inform public health interventions. However, low response rates may contribute to nonresponse bias. In a randomized controlled trial we examined the effect of sending a prenotification postcard before the questionnaire and the placement of a short message on the survey envelope (teaser) on response rates to a mailed questionnaire about bulimia nervosa “mental health literacy”.Study Design and SettingQuestionnaires were mailed to 3,010 adults (50.6% female and 49.4% male) aged 18–65 years. In a 2 (pre-notification–present; absent) by 2 (teaser–present; absent) design, questionnaire recipients were randomly allocated to the experimental strategies. Outcomes considered were response rate, response time, and cost.ResultsThe overall response rate was 22.0%. Significant main effects showed higher response rates for the use of prenotification (present = 23.6%; absent = 20.3%), among female participants, and older participants. A significant interaction of teaser by gender indicated lower response rates for men who received the teaser but not for women. Older participants returned the questionnaire more promptly than younger participants. Females—but not males—who received the teaser were slower to return the questionnaire. Higher response rates for participants receiving the postcard compensated for increased costs, particularly for males and older participants.ConclusionResponse rates to a mental health postal survey can be increased through the use of prenotification.     

An evidence-based practice guideline for the peer review of electronic search strategies

ObjectiveComplex and highly sensitive electronic literature search strategies are required for systematic reviews; however, no guidelines exist for their peer review. Poor searches may fail to identify existing evidence because of inadequate recall (sensitivity) or increase the resource requirements of reviews as a result of inadequate precision. Our objective was to create an annotated checklist for electronic search strategy peer review.Study DesignA systematic review of the library and information retrieval literature for important elements in electronic search strategies was conducted, along with a survey of individuals experienced in systematic review searching.ResultsSix elements with a strong consensus as to their importance in peer review were accurate translation of the research question into search concepts, correct choice of Boolean operators and of line numbers, adequate translation of the search strategy for each database, inclusion of relevant subject headings, and absence of spelling errors. Seven additional elements had partial support and are included in this guideline.ConclusionThis evidence-based guideline facilitates the improvement of search quality through peer review, and thus the improvement in quality of systematic reviews. It is relevant for librarians/information specialists, journal editors, developers of knowledge translation tools, research organizations, and funding bodies.     

Systematic overview finds variation in approaches to investigating and reporting on sources of heterogeneity in systematic reviews of diagnostic studies

ObjectivesTo examine how authors explore and report on sources of heterogeneity in systematic reviews of diagnostic accuracy studies.Study Design and SettingA cohort of systematic reviews of diagnostic tests was systematically identified. Data were extracted on whether an exploration of the sources of heterogeneity was undertaken, how this was done, the number and type of potential sources explored, and how results and conclusions were reported.ResultsOf the 65 systematic reviews, 12 did not perform a meta-analysis and eight of these gave heterogeneity between studies as a reason. Of the 53 reviews containing a meta-analysis, 40 explored potential sources of heterogeneity in a formal manner and 27 identified at least one source of heterogeneity. The reviews not investigating heterogeneity were smaller than those that did (median [interquartile range {IQR}], 8 [5–15] vs. 14 [11–19] primary studies). Twelve reviews performed a sensitivity analysis, 25 stratified analyses, and 19 metaregression. Many sources of heterogeneity were explored compared with the number of primary studies in a meta-analysis (median ratio, 1:5). Review authors placed importance on the exploration of sources of heterogeneity; 37 mentioned the exploration or the findings thereof in the abstract or conclusion of the main text.resultsConclusionMethods for investigating sources of heterogeneity varied widely between reviews. Based on our findings of the review, we made suggestions on what to consider and report on when exploring sources of heterogeneity in systematic reviews of diagnostic studies.     

Three challenges described for identifying participants with missing data in trials reports,and potential solutions suggested to systematic reviewers

ObjectiveTo categorize the challenges in determining the extent of missing participant data in randomized trials and suggest potential solutions for systematic review authors.Study Design and SettingDuring the process of updating a series of Cochrane systematic reviews on the topic of anticoagulation in patients with cancer, we identified challenges and used an iterative approach to improve, and a consensus process to agree on the challenges identified, and to suggest potential ways of dealing with them. The five systematic reviews included 58 trials and 75 meta-analyses for patient-important dichotomous outcomes with 27,037 randomized participants.ResultsWe identified three categories of challenges: (1) Although systematic reviewers require information about missing data to be reported by outcome, trialists typically report the information by participant; (2) It is not always clear whether the trialists followed up participants in certain categories (e.g., noncompliers), that is, whether some categories of participants did or did not have missing data; (3) It is not always clear how the trialists dealt with missing data in their analysis (e.g., exclusion from the denominator vs. assumptions made for the numerator). We discuss potential solutions for each one of these challenges and suggest further research work.ConclusionCurrent reporting of missing data is often not explicit and transparent, and although our potential solutions to problems of suboptimal reporting may be helpful, reliable and valid characterization of the extent and nature of missing data remains elusive. Reporting of missing data in trials needs further improvement.     

Single data extraction generated more errors than double data extraction in systematic reviews

BACKGROUND AND OBJECTIVE: To conduct a pilot study to compare the frequency of errors that accompany single vs. double data extraction, compare the estimate of treatment effect derived from these methods, and compare the time requirements for these methods. METHODS: Reviewers were randomized to the role of data extractor or data verifier, and were blind to the study hypothesis. The frequency of errors associated with each method of data extraction was compared using the McNemar test. The data set for each method was used to calculate an efficacy estimate by each method, using standard meta-analytic techniques. The time requirement for each method was compared using a paired t-test. RESULTS: Single data extraction resulted in more errors than double data extraction (relative difference: 21.7%, P = .019). There was no substantial difference between methods in effect estimates for most outcomes. The average time spent for single data extraction was less than the average time for double data extraction (relative difference: 36.1%, P = .003). CONCLUSION: In the case that single data extraction is used in systematic reviews, reviewers and readers need to be mindful of the possibility for more errors and the potential impact these errors may have on effect estimates.     

Choice of data extraction tools for systematic reviews depends on resources and review complexity

ObjectiveTo assist investigators planning, coordinating, and conducting systematic reviews in the selection of data-extraction tools for conducting systematic reviews.Study Design and SettingWe constructed an initial table listing available data-collection tools and reflecting our experience with these tools and their performance. An international group of experts iteratively reviewed the table and reflected on the performance of the tools until no new insights and consensus resulted.ResultsSeveral tools are available to manage data in systematic reviews, including paper and pencil, spreadsheets, web-based surveys, electronic databases, and web-based specialized software. Each tool offers benefits and drawbacks: specialized web-based software is well suited in most ways, but is associated with higher setup costs. Other approaches vary in their setup costs and difficulty, training requirements, portability and accessibility, versatility, progress tracking, and the ability to manage, present, store, and retrieve data.ConclusionAvailable funding, number and location of reviewers, data needs, and the complexity of the project should govern the selection of a data-extraction tool when conducting systematic reviews.