Addressing continuous data for participants excluded from trial analysis: a guide for systematic reviewers

BackgroundNo methods directly address the impact of missing participant data for continuous outcomes in systematic reviews on risk of bias.MethodsWe conducted a consultative, iterative process to develop a framework for handling missing participant data for continuous outcomes. We considered sources reflecting real observed outcomes in participants followed-up in individual trials included in the systematic review, and developed a range of plausible strategies. We applied our approach to two systematic reviews.ResultsWe used five sources of data for imputing the means for participants with missing data. To impute standard deviation (SD), we used the median SD from the control arms of all included trials. Using these sources, we developed four progressively more stringent imputation strategies. In the first example review, effect estimates diminished and lost significance as strategies became more stringent, suggesting rating down confidence in estimates of effect for risk of bias. In the second, effect estimates maintained statistical significance using even the most stringent strategy, suggesting missing data does not undermine confidence in results.ConclusionsOur approach provides a useful, reasonable, and relatively simple, quantitative guidance for judging the impact of risk of bias as a result of missing participant data in systematic reviews of continuous outcomes.     

Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews

ObjectiveTo collate and categorize the ways in which selective inclusion and reporting can occur in randomized controlled trials (RCTs) and systematic reviews.Study Design and SettingSearches of the Cochrane Methodology Register, PubMed, and PsycInfo were conducted in April 2011. Methodological reports describing empirically investigated or hypothetical examples of selective inclusion or reporting were eligible for inclusion. Examples were extracted from the reports by one author and categorized by three authors independently. Discrepancies in categorization were resolved via discussion.ResultsTwo hundred ninety reports were included. The majority were empirical method studies (45.5%) or commentaries (29.3%). Eight categories (30 examples) of selective reporting in RCTs, eight categories (27 examples) of selective inclusion in systematic reviews, and eight categories (33 examples) of selective reporting in systematic reviews were collated. Broadly, these describe scenarios in which multiple outcomes or multiple data for the same outcome are available, yet only a subset is included or reported; outcome data are reported with inadequate detail; or outcome data are given different prominence through its placement across or within reports.ConclusionAn extensive list of examples of selective inclusion and reporting was collated. Increasing trialists’ and systematic reviewers’ awareness of these examples may minimize their occurrence.     

Addressing continuous data measured with different instruments for participants excluded from trial analysis: a guide for systematic reviewers

BackgroundWe previously developed an approach to address the impact of missing participant data in meta-analyses of continuous variables in trials that used the same measurement instrument. We extend this approach to meta-analyses including trials that use different instruments to measure the same construct.MethodsWe reviewed the available literature, conducted an iterative consultative process, and developed an approach involving a complete-case analysis complemented by sensitivity analyses that apply a series of increasingly stringent assumptions about results in patients with missing continuous outcome data.ResultsOur approach involves choosing the reference measurement instrument; converting scores from different instruments to the units of the reference instrument; developing four successively more stringent imputation strategies for addressing missing participant data; calculating a pooled mean difference for the complete-case analysis and imputation strategies; calculating the proportion of patients who experienced an important treatment effect; and judging the impact of the imputation strategies on the confidence in the estimate of effect. We applied our approach to an example systematic review of respiratory rehabilitation for chronic obstructive pulmonary disease.ConclusionsOur extended approach provides quantitative guidance for addressing missing participant data in systematic reviews of trials using different instruments to measure the same construct.     

Systematic reviewers commonly contact study authors but do so with limited rigor

ObjectivesAuthor contact can enhance the quality of systematic reviews. We conducted a systematic review of the practice of author contact in recently published systematic reviews to characterize its prevalence, quality, and results.Study Design and SettingEligible studies were systematic reviews of efficacy published in 2005–2006 in the 25 journals with the highest impact factor publishing systematic reviews in clinical medicine and the Cochrane Library, identified by searching MEDLINE, EMBASE, and the Cochrane Library. Two researchers determined whether and why reviewers contacted authors. To assess the accuracy of the abstracted data, we surveyed reviewers by e-mail.ResultsForty-six (50%) of the 93 eligible systematic reviews published in top journals and 46 (85%) of the 54 eligible Cochrane reviews reported contacting authors of eligible studies. Requests were made most commonly for missing information: 40 (76%) clinical medicine reviews and 45 (98%) Cochrane reviews. One hundred and nine of 147 (74%) reviewers responded to the survey, and reported a higher rate of author contact than apparent from the published record.ConclusionAlthough common, author contact is not a universal feature of systematic reviews published in top journals and the Cochrane Library. The conduct and reporting of author contact purpose, procedures, and results require improvement.     

Qualitative “trial-sibling” studies and “unrelated” qualitative studies contributed to complex intervention reviews

ObjectiveTo compare the contribution of “trial-sibling” and “unrelated” qualitative studies in complex intervention reviews.Study Design and SettingResearchers are using qualitative “trial-sibling” studies undertaken alongside trials to provide explanations to understand complex interventions. In the absence of qualitative “trial-sibling” studies, it is not known if qualitative studies “unrelated” to trials are helpful. Trials, “trial-sibling,” and “unrelated” qualitative studies looking at three health system interventions were identified. We looked for similarities and differences between the two types of qualitative studies, such as participants, intervention delivery, context, study quality and reporting, and contribution to understanding trial results.ResultsReporting was generally poor in both qualitative study types. We detected no substantial differences in participant characteristics. Interventions in qualitative “trial-sibling” studies were delivered using standardized protocols, whereas interventions in “unrelated” qualitative studies were delivered in routine care. Qualitative “trial-sibling” studies alone provided insufficient data to develop meaningful transferrable explanations beyond the trial context, and their limited focus on immediate implementation did not address all phenomena of interest. Together, “trial-sibling” and “unrelated” qualitative studies provided larger, richer data sets across contexts to better understand the phenomena of interest.ConclusionsFindings support inclusion of “trial-sibling” and “unrelated” qualitative studies to explore complexity in complex intervention reviews.     

A methodological review of recent meta-analyses has found significant heterogeneity in age between randomized groups

BackgroundThere is evidence to suggest that component randomized controlled trials (RCTs) within systematic reviews may be biased. It is important that these reviews are identified to prevent erroneous conclusions influencing health care policies and decisions.PurposeTo assess the likelihood of bias in trials in 12 meta-analyses.DesignA review of 12 systematic reviews.Data SourcesTwelve recently published systematic reviews with 503 component randomized trials, published in the British Medical Journal, The Lancet, Journal of the American Medical Association, and The Annals of Internal Medicine before May 2012.Study Selection and Data ExtractionSystematic reviews were eligible for inclusion if they included only RCTs. We obtained the full text for the component RCTs of the 12 systematic reviews (in English only). We extracted summary data on age, number of participants in each treatment group, and the method of allocation concealment for each RCT.Data SynthesisFive of the 12 meta-analyses exhibited heterogeneity in age differences (I² > 0.30), when there should have been none. In two meta-analyses, the age of the intervention group was significantly greater than that of the control group. Inadequate allocation concealment was a statistically significant predictor of heterogeneity in one trial as observed by a metaregression.ConclusionsMost of the sample of recent meta-analyses showed that there were signs of imbalance and/or heterogeneity in ages between treatment groups, when there should have been none. Systematic reviewers might consider using the techniques described here to assess the validity of their findings.     

Is there a solution to publication bias? Researchers call for changes in dissemination of clinical research results

ObjectivesTo explore opinions of authors of published reports of clinical trials and Cochrane systematic reviews on the causes and methods of preventing publication bias.Study Design and SettingAn online questionnaire was developed and sent to researchers publishing in high-impact or national general medical journals, authors of Cochrane systematic reviews, and a general population of researchers. Open-ended questions about publication bias were qualitatively analyzed. We also held a focus group with experienced researchers and/or Cochrane reviewers.ResultsPublication bias was common: 48 (36%) respondents had own unpublished trials and 40 (30%) admitted selective outcome reporting; but researchers felt strongly that blame rested also with the system that promotes and augments publication bias practices. Qualitative analysis of both survey responses and focus group discussion identified possible ways of reducing publication bias through increased transparency, improvements to trial registries, search engines and databases, enhancement of the role of institutional review boards, positive encouragement of scientists, and policy changes.ConclusionAlthough well aware of the problem, clinical researchers knowingly contribute to problems of selective reporting and nonpublication of trials. They call for changes in current practices of journal-based communication of trial reporting and for systematic evaluation of measures to decrease publication bias.     

The elderly were under-represented in osteoarthritis clinical trials

ObjectivesOsteoarthritis is the most common disease affecting joints in the elderly. We aimed to evaluate if elderly patients are properly represented in clinical trials of diverse osteoarthritis interventions.Study Design and SettingClinical trials of osteoarthritis interventions were retrieved from Cochrane Library systematic reviews (2006, issue 2). We examined the age distribution of the trial participants and eligibility criteria.ResultsWe analyzed data from 219 eligible trials from 18 systematic reviews. The average mean age of the participants was 63 years. Only 13 trials (6.4%) had a mean age between 71 and 80 years and only one trial had a mean age exceeding 80 years. Among trials where the age range of participants was available or could be approximately inferred, we estimated that 66 (38%) trials had not included any patients over 80 years old. Only 23 trials specifically excluded patients over 70 based on reported eligibility criteria, but 168 trials excluded patients with various comorbidities and 142 trials excluded patients receiving other specific treatments.ConclusionsElderly patients are considerably under-represented in clinical trials of osteoarthritis. This causes an important deficit in the utility, relevance, and generalizability of trial results for this very common condition.     

A pragmatic–explanatory continuum indicator summary (PRECIS): a tool to help trial designers

ObjectiveTo propose a tool to assist trialists in making design decisions that are consistent with their trial's stated purpose.Study Design and SettingRandomized trials have been broadly categorized as either having a pragmatic or explanatory attitude. Pragmatic trials seek to answer the question, “Does this intervention work under usual conditions?,” whereas explanatory trials are focused on the question, “Can this intervention work under ideal conditions?” Design decisions make a trial more (or less) pragmatic or explanatory, but no tool currently exists to help researchers make the best decisions possible in accordance with their trial's primary goal. During the course of two international meetings, participants with experience in clinical care, research commissioning, health care financing, trial methodology, and reporting defined and refined aspects of trial design that distinguish pragmatic attitudes from explanatory.ResultsWe have developed a tool (called PRECIS) with 10 key domains and which identifies criteria to help researchers determine how pragmatic or explanatory their trial is. The assessment is summarized graphically.ConclusionWe believe that PRECIS is a useful first step toward a tool that can help trialists to ensure that their design decisions are consistent with the stated purpose of the trial.     

Challenges and Solutions for the Benefit Assessment of Tumor-Agnostic Therapies in Germany

ObjectivesPrecision medicine is increasingly important in cancer treatment. Tumor-agnostic therapies are used regardless of tumor entity because they target specific biomarkers in tumors. In Germany, the benefit assessment of oncological pharmaceuticals has traditionally been entity specific. Thus, the assessment of tumor-agnostic therapies leaves stakeholders with various challenges. Our aim was to systematically identify challenges and possible solutions for the benefit assessment of therapies in tumor-agnostic indications using a 2-step sequential qualitative approach.MethodsTo identify relevant challenges, we conducted qualitative interviews with different stakeholders who were involved in previous benefit assessments of tumor-agnostic therapies in Germany. To identify possible solutions for these challenges, we systematically searched MEDLINE, Embase, and the websites of European health technology assessment bodies for relevant literature.ResultsWe identified 9 categories of challenges of which the following were deemed particularly relevant: the absence of direct comparative studies, challenges regarding the use of basket studies and indirect comparisons, challenges in determining the appropriate comparative therapy in a tumor-agnostic indication, and challenges on the system side. Seven categories of solutions were identified, including an increased use of real-world evidence, making conditional decisions in the context of systematic reassessments, splitting the field of application, and finding (new) ways to design and analyze basket studies.ConclusionA range of possible solutions, which can help to meet the identified challenges in Germany, have been found. Future research should investigate the acceptance and feasibility of these solutions.     

Reporting of health-related quality of life (HRQOL) data in oncology trials: a comparison of the European Organization for Research and Treatment of Cancer Quality of Life (EORTC QLQ-C30) and the Functional Assessment of Cancer Therapy-General (FACT-G)

Purpose

The inclusion of patient-reported outcome (PRO) instruments to record patient health-related quality of life (HRQOL) data has virtually become the norm in oncology randomised controlled trials (RCTs). Despite this fact, recent concerns have focused on the quality of reporting of HRQOL. The primary aim of this study was to evaluate the quality of reporting of HRQOL data from two common instruments in oncology RCTs.

Design

A meta-review was undertaken of systematic reviews reporting HRQOL data collected using PRO instruments in oncology randomised controlled trials (RCTs). English language articles published between 2000 and 2012 were included and evaluated against a methodology checklist.

Results

Four hundred and thirty-five potential articles were identified. Six systematic reviews were included in the analysis. A total of 70,403 patients had completed PROs. The European Organization for Research and Treatment of Cancer QLQ-C30 and Functional Assessment of Cancer Therapy-General questionnaire accounted for 55 % of RCTs. Eighty per cent of RCTs had used psychometrically validated instruments; 70 % reported culturally valid instruments and almost all reported the assessment timing (96 %). Thirty per cent of RCTS reported clinical significance and missing data. In terms of methodological design, only 25 % of RCTs could be categorised as probably robust.

Conclusion

The majority of oncology RCTs has shortcomings in terms of reporting HRQOL data when assessed against regulatory and methodology guidelines. These limitations will need to be addressed if HRQOL data are to be used to successfully support clinical decision-making, treatment options and labelling claims in oncology.     

Unpublished data can be of value in systematic reviews of adverse effects: methodological overview

ObjectiveTo assess the impact of including unpublished data on adverse effects in systematic reviews.Study Design and SettingWe carried out a systematic review of methodological evaluations that compared the quantitative reporting of adverse effects data between published and unpublished sources, in particular, the frequency, rate, or risk of reported adverse effects. Included studies were sought from 10 databases as well as by checking references, handsearching, searching citations, and contacting experts.ResultsWe identified 6,218 potential articles yielding 10 relevant methodological evaluations.One evaluation found that adverse effects were reported more often in unpublished trials. For anecdotal case reports, two evaluations found a higher frequency of unpublished cases, whereas one study identified a greater number of published cases. Another evaluation indicated that differences in frequency of published and unpublished case reports were topic dependent.A comparison of relative risk estimates from five studies suggested no major systematic variation in risk estimates from published and unpublished studies.ConclusionInclusion of unpublished studies can provide additional adverse effects information and more precise risk estimates. However, there is insufficient evidence to indicate whether inclusion of unpublished studies has a major influence on the pooled risk estimates in meta-analyses of adverse effects.     

Kawasaki disease and immunisation: A systematic review

BackgroundKawasaki disease is a complex and potentially serious condition. It has been observed in temporal relation to immunisation.MethodsWe conducted a systematic literature review using various reference sources to review the available evidence published in the literature.ResultsWe identified twenty seven publications reporting a temporal association between immunisation and Kawasaki disease. We present a systematic review of data drawn from randomised controlled trials, observational studies, case series and reports, and reviews. Overall there was a lack of standardised case definitions, making data interpretation and comparability challenging.ConclusionsAlthough a temporal relationship between immunisation and Kawasaki disease is suggested, evidence for an increased risk or a causal association is lacking. Implementation of a standardised Kawasaki disease case definition would increase confidence in the findings and add value to future studies of pre- or post-licensure vaccine safety studies.     

Insufficient evidence to determine the impact of patient preferences on clinical outcomes in acupuncture trials: a systematic review

ObjectiveTo review reporting of preferences in acupuncture studies and their effect on clinical outcomes.Study Design and SettingSystematic review of published randomized and quasi-randomized controlled trials of acupuncture reporting participant preferences for randomization or treatment or using a preference design.ResultsOf the 31 included trials, 5 reported on randomization preference, 18 on treatment preference, and 1 reported on both. Seven used a preference design. Four out of seven trials noted that the group with preferences had different baseline characteristics (less education, worse baseline measure score, and greater or fewer years with pain). There was a tendency for greater attrition in nonpreference arms at 6 months but not earlier. Around three-quarters of participants turned down randomization in favor of nontrial treatment, and preference for acupuncture was around 20% when offered multiple treatments. Questions used to elicit preferences varied across trials and were poorly reported. Ten trials reported the effects of preferences on outcomes; only one detected a statistically (but not clinically) significant difference.ConclusionThere is little evidence that preferences cause detectable effects on outcomes in acupuncture trials; however, trials use inconsistent methods and poorly report these data. Monitoring the level and effect of preferences in trials is recommended.     

Alzheimer’s Disease Services,Staffing, and Outcomes in Adult Day Health Centers

ObjectivesIncreasing rates of Alzheimer disease and related dementia (ADRD) has resulted in greater reliance on adult day health centers (ADHCs) and their skilled workforce. Little is known about staffing in ADHCs that provide ADRD services compared with ADHCs that do not. This study examines whether there are differences in staffing between ADHCs that offer ADRD services versus those that do not, and whether the percentage of ADHC participants with ADRD is associated with staffing levels. It also examines whether staffing levels and provision of ADRD services are associated with participant outcomes.DesignCross-sectional analysis of secondary survey data.Setting and ParticipantsWe used facility-level data from the 2014 National Post-acute and Long-term Care Study Adult Day Services Center module. This survey is completed by administrators of ADHCs, who provide information about their ADHC’s organization, services, participants, sources of payment, staffing, and participant outcomes.MethodsBivariate comparisons and multivariate regressions were used to compare scope of services, staffing, and participant outcomes for ADHCs that offered ADRD programs compared with those that did not.ResultsADHCs with ADRD programs had similar average daily attendance, less revenue from Medicaid and self-payment, and greater proportions of Black and female participants. ADHCs with ADRD programs had similar staff hours per participant day for all staff categories; licensed nurse staffing increased and social worker staffing decreased with the proportion of participants with ADRD. Staffing had significant associations with participant outcomes.Conclusions and ImplicationsADHCs that have more participants with ADRD have greater staffing of licensed nurses but fewer social workers. Participant outcomes are associated with staffing, but the results suggest that there are unmeasured dimensions of participant risk that confound the relationship.