The quality of reporting of trial abstracts is suboptimal: Survey of major general medical journals

ObjectiveTo evaluate the quality of reporting of abstracts describing randomized controlled trials (RCTs) published in four major general medical journals.Study Design and SettingSystematic survey of published RCT abstracts, with two reviewers independently extracting data. We searched MEDLINE and identified 227 RCT abstracts published in the New England Journal of Medicine (NEJM), Journal of the American Medical Association (JAMA), British Medical Journal (BMJ), and The Lancet in the year 2006.ResultsMost abstracts identified the study as a randomized trial (98.7%), reported the objectives (92.5%), described the population (90.3%), detailed the intervention (81.5%), and defined the primary outcome (71.3%). Methodological quality was poorly reported: one (0.4%) described allocation concealment; 21 (9.3%) clearly specified blinding; 51 (22.5%) described intention-to-treat analysis; and 32 (14.1%) outlined losses to follow-up. Most of the abstracts reported the effect size and the confidence intervals (62.3%), but just half of them reported side effects or harms.ConclusionThe quality of reporting of RCT abstracts published in main general medical journals is suboptimal. Space limitations notwithstanding, with the recent recommendations from the CONSORT for Abstracts, it is expected that the transparency of abstract reporting can and should improve.     

Reporting of methods was better in the Clinical Trials Registry-India than in Indian journal publications

ObjectiveWe sought to evaluate if editorial policies and the reporting quality of randomized controlled trials (RCTs) had improved since our 2004–05 survey of 151 RCTs in 65 Indian journals, and to compare reporting quality of protocols in the Clinical Trials Registry-India (CTRI).Study Design and SettingAn observational study of endorsement of Consolidated Standards for the Reporting of Trials (CONSORT) and International Committee of Medical Journal Editors (ICMJE) requirements in the instructions to authors in Indian journals, and compliance with selected requirements in all RCTs published during 2007–08 vs. our previous survey and between all RCT protocols in the CTRI on August 31, 2010 and published RCTs from both surveys.ResultsJournal policies endorsing the CONSORT statement (22/67, 33%) and ICMJE requirements (35/67, 52%) remained suboptimal, and only 4 of 13 CONSORT items were reported in more than 50% of the 145 RCTs assessed. Reporting of ethical issues had improved significantly, and that of methods addressing internal validity had not improved. Adequate methods were reported significantly more frequently in 768 protocols in the CTRI, than in the 296 published trials.ConclusionThe CTRI template facilitates the reporting of valid methods in registered trial protocols. The suboptimal compliance with CONSORT and ICMJE requirements in RCTs published in Indian journals reduces credibility in the reliability of their results.     

A tool to improve quality of reporting published economic analyses

OBJECTIVES: To test the feasibility of obtaining a baseline level of quality of reporting for cost-utility analysis (CUA) studies using the British Medical Journal economic submissions checklist, test interrater reliability of this tool, and discuss its longer term implications. METHODS: CUA studies in peer-reviewed English language journals in 1996, assessed using the British Medical Journal checklist, a quality index, and interrater reliability correlations. RESULTS: Forty-three CUA studies were assessed, with 23 checklist items acceptable and 10 items inadequate. Lowest quality scores were reported in specialist medical journals. Proportional agreement between assessors was over 80%. CONCLUSIONS: The British Medical Journal checklist is a feasible tool to collect baseline information on the quality of reporting in journals other than the British Medical Journal. Editors of specialist medical journals are in greatest need of economic guidance. If handled carefully, they might consider adopting the British Medical Journal checklist.     

Randomized trials published in higher vs. lower impact journals differ in design,conduct, and analysis

ObjectiveTo compare methodological characteristics of randomized controlled trials (RCTs) published in higher vs. lower impact Core Clinical Journals.Study Design and SettingWe searched MEDLINE for RCTs published in 2007 in Core Clinical Journals. We randomly sampled 1,140 study reports in a 1:1 ratio in higher (five general medicine journals with the highest total citations in 2007) and lower impact journals.ResultsFour hundred sixty-nine RCTs proved eligible: 219 in higher and 250 in lower impact journals. RCTs in higher vs. lower impact journals had larger sample sizes (median, 285 vs. 39), were more likely to receive industry funding (53% vs. 28%), declare concealment of allocation (66% vs. 36%), declare blinding of health care providers (53% vs. 41%) and outcome adjudicators (72% vs. 54%), report a patient-important primary outcome (69% vs. 50%), report subgroup analyses (64% vs. 26%), prespecify subgroup hypotheses (42% vs. 20%), and report a test for interaction (54% vs. 27%); P < 0.05 for all differences.ConclusionRCTs published in higher impact journals were more likely to report methodological safeguards against bias and patient-important outcomes than those published in lower impact journals. However, sufficient limitations remain such that publication in a higher impact journal does not ensure low risk of bias.     

Influence of trial registration on reporting quality of randomized trials: Study from highest ranked journals

ObjectiveTo evaluate the reporting quality of key methodological items of randomized control trials (RCTs) in 55 of the highest ranked journals.Study Design and SettingA list of the highest top ranked journals was identified, and a search for detecting RCTs in those journals was made. Two hundred sixty four journals were screened and 55 of them were identified having at least one RCT. Three RCTs were randomly selected a priori from each journal; 148 RCTs were finally included. RCTs were assessed by two reviewers using the Consolidated Standards of Reporting Trials (CONSORT) statement.ResultsOnly 11 (8%) RCTs had all items adequately reported. In addition, 36% of RCTs reported that the study was registered in any trial registry. We found a significant difference in the quality of reporting for baseline characteristics, recruitment, participant's flow, and randomization implementation between those studies having reported the registration of their RCT in a trial registry and those that have not. Adherence to key methodological items of the CONSORT statement was as follows: sample size determination (60%), sequence generation (49%), allocation concealment (40%), and blinding (25%).ConclusionsReporting of varied CONSORT items remains suboptimal. Registration in a trial registry was associated with improved reporting. Further efforts to enhance RCT registration could contribute to this improvement.     

An absence of pediatric randomized controlled trials in general medical journals, 1985-2004

OBJECTIVE: There are numerous potential barriers to conducting randomized controlled trials (RCTs) in children. The purpose of this study was to compare the quantity, trends over time, characteristics, and quality of pediatric RCTs published in general medical journals (GMJs) with adult RCTs. STUDY DESIGN AND SETTING: We conducted an electronic search of adult and pediatric RCTs from 1985-2004 and a manual search of published RCTs in the year 2000 in five high-impact GMJs (New England Journal of Medicine, Journal of the American Medical Association [JAMA], the Lancet, British Medical Journal [BMJ], Canadian Medical Association Journal [CMAJ]). Linear trends were identified and the 1-year sample was analyzed for publication characteristics (location of recruitment, sample size, number of centers, funding sources, and results) and quality scoring (Jadad score, intention-to-treat analysis, and citation frequency since publication). RESULTS: Adult RCTs increased by 4.71 RCTs/year (95% confidence interval (CI) 3.62-5.80; P<0.001), which was significantly higher (P<0.0001) than pediatric RCTs, which increased by 0.4 RCTs/year (95% CI -0.02 to 0.9; P=0.06). Adult RCTs were more likely to be hospital-based (P=.001) and to involve more centers in multicenter studies (P=0.02). Quality scores were similar, although adult RCTs were cited more frequently (P=0.003). CONCLUSION: There may be significant barriers to the publication of high-quality pediatric RCTs in GMJs.     

A methodological review of recent meta-analyses has found significant heterogeneity in age between randomized groups

BackgroundThere is evidence to suggest that component randomized controlled trials (RCTs) within systematic reviews may be biased. It is important that these reviews are identified to prevent erroneous conclusions influencing health care policies and decisions.PurposeTo assess the likelihood of bias in trials in 12 meta-analyses.DesignA review of 12 systematic reviews.Data SourcesTwelve recently published systematic reviews with 503 component randomized trials, published in the British Medical Journal, The Lancet, Journal of the American Medical Association, and The Annals of Internal Medicine before May 2012.Study Selection and Data ExtractionSystematic reviews were eligible for inclusion if they included only RCTs. We obtained the full text for the component RCTs of the 12 systematic reviews (in English only). We extracted summary data on age, number of participants in each treatment group, and the method of allocation concealment for each RCT.Data SynthesisFive of the 12 meta-analyses exhibited heterogeneity in age differences (I² > 0.30), when there should have been none. In two meta-analyses, the age of the intervention group was significantly greater than that of the control group. Inadequate allocation concealment was a statistically significant predictor of heterogeneity in one trial as observed by a metaregression.ConclusionsMost of the sample of recent meta-analyses showed that there were signs of imbalance and/or heterogeneity in ages between treatment groups, when there should have been none. Systematic reviewers might consider using the techniques described here to assess the validity of their findings.     

The quality of safety reporting in trials is still suboptimal: survey of major general medical journals

Objective

To evaluate whether the quality of reporting harms improved after the publication of the Extension of the Consolidated Standards of Reporting Trials (CONSORT) statement and predictors that influence the safety reporting in randomized controlled trials (RCTs)

Study Design and Setting

Systematic survey of published RCTs assessing drugs. In MEDLINE, we identified 228 RCTs published in Annals of Internal Medicine, British Medical Journal, Journal of American Medical Association, The Lancet, and The New England Journal of Medicine in 2003 and 2006.

Results

The reporting of harms have improved over time both in quality and extent of space. However, the mean score as an overall measure of adequacy in reporting harms was 0.58 in 2003 and increased to 0.67 in 2006, indicating a moderate safety reporting. Safety was more adequate in trials with statistically significant results for efficacy, private funding, primary harms outcome, and anti-infective, antineoplasmatic, or immunosuppressive agents.

Conclusion

The use of the Extension of the CONSORT statement may be associated with improving the quality of safety reporting in RCTs, but there are still deficiencies that need to be corrected to use quantitative objective evidence for harms in performing meta-analyses and making therapeutic decisions.     

Changing author counts in five major general medicine journals: effect of author contribution forms

ObjectivesObjective and indirect evidence was used to determine whether required author contribution forms were associated with a decrease in author counts in four major general medicine journals (British Medical Journal [BMJ], Journal of the American Medical Association [JAMA], Canadian Medical Association Journal [CMAJ], and the Lancet). The number of authors listed per article before and after the introduction of explicit author contribution requirements were counted and compared with that found for the New England Journal of Medicine (NEJM) that did not require such disclosure. The primary hypothesis was that author counts decreased more in the BMJ, CMAJ, JAMA, and the Lancet after introduction of the rules than they did in the NEJM.Study Design and SettingThe number of authors listed per original research article published in the five general medical journals with the greatest 2004 Impact Factors in the first issue of each month in the years before and after introduction of required author contribution forms was compared.ResultsIntroduction of the required author contribution forms by the four leading general medical journals did not result in a drop in the rate of increasing authors per article per year, or in the number of authors per article compared with the control. Overall, there was a trend of an increasing number of authors listed per article.ConclusionBased on the presented objective and indirect evidence, required author contribution forms were not associated with a decrease in author counts.     

Covariate adjustments in randomized controlled trials increased study power and reduced biasedness of effect size estimation

ObjectivesThis study aims to show that under several assumptions, in randomized controlled trials (RCTs), unadjusted, crude analysis will underestimate the Cohen's d effect size of the treatment, and an unbiased estimate of effect size can be obtained only by adjusting for all predictors of the outcome.Study Design and SettingFour simulations were performed to examine the effects of adjustment on the estimated effect size of the treatment and power of the analysis. In addition, we analyzed data from the Advanced Cognitive Training for Independent and Vital Elderly (ACTIVE) study (older adults aged 65–94), an RCT with three treatment arms and one control arm.ResultsWe showed that (1) the number of unadjusted covariates was associated with the effect size of the treatment; (2) the biasedness of effect size estimation was minimized if all covariates were adjusted for; (3) the power of the statistical analysis slightly decreased with the number of adjusted noise variables; and (4) exhaustively searching the covariates and noise variables adjusted for can lead to exaggeration of the true effect size. Analysis of the ACTIVE study data showed that the effect sizes adjusting for covariates of all three treatments were 7.39–24.70% larger than their unadjusted counterparts, whereas the effect size would be elevated by at most 57.92% by exhaustively searching the variables adjusted for.ConclusionAll covariates of the outcome in RCTs should be adjusted for, and if the effect of a particular variable on the outcome is unknown, adjustment will do more good than harm.     

Reported effects in randomized controlled trials were compared with those of nonrandomized trials in cholecystectomy

ObjectivesBecause external validity of randomized controlled trials (RCTs) may be insufficient, the performance of nonrandomized controlled trials (nRCTs) is increasingly debated. RCTs and nRCTs were compared using the example of laparoscopic vs. open cholecystectomy (LC vs. OC).Study Design and SettingRCTs and nRCTs comparing LC and OC were identified by searching PubMed. To assess internal and external validity of the studies, patient characteristics, relative risks, and mean differences of RCTs and nRCTs were compared by meta-analytic techniques.ResultsIn total, 162 studies were analyzed (136 nRCTs and 26 RCTs). Significant discrepancies between RCT- and nRCT-based results were revealed for 3 of 15 variables: overall complications (P < 0.021), wound infections (P < 0.014), and length of hospital stay (P < 0.005). In RCTs and in nRCTs, length of hospital stay and return to work were significantly reduced when using LC compared with OC. The results of nRCTs were more often heterogeneous among themselves (11 of 15) as compared with RCTs (4 of 15).ConclusionThe results of RCTs and nRCTs differ significantly in at least 20% of the variables. External validities of RCTs and nRCTs in LC vs. OC appear to be similar. Between-study heterogeneity was larger in nRCTs than in RCTs of cholecystectomy.     

Publication bias for CAM trials in the highest impact factor medicine journals is partly due to geographical bias

OBJECTIVE: To assess the presence of publication bias and its relation to geographical bias in clinical trials involving complementary and alternative medicine (CAM) published in the highest impact factor general medicine journals. STUDY DESIGN AND SETTING: All CAM clinical trials published in the four highest impact factor general medicine journals, Lancet and British Medical Journal (European), and New England Journal of Medicine and Journal of American Medical Association (U.S.), between 1965 and 2004 were abstracted using Medline. Three reviewers abstracted data from the individual studies. In a multivariate analysis, factors predictive of a positive study were assessed. RESULTS: A total of 259 studies met the inclusion criteria. CAM trials published in the European journals were significantly more likely to be positive compared to those published in the U.S. journals (76% vs. 50%, odds ratio [OR]=3.15, P<0.0001). Studies originating outside of the United States were significantly more likely to be positive compared to the U.S. studies (75% vs. 49%, P<0.0001). Adjusting for location and other variables in a multivariate model, the OR for European vs. U.S. journals to publish a positive CAM trial was 1.95 (P=0.11). CONCLUSION: Publication bias related to CAM trials among the highest impact factor general medicine journals is partly due to geographical bias.     

Empirical assessment suggests that existing evidence could be used more fully in designing randomized controlled trials

BackgroundMeta-analyses of randomized controlled trials (RCTs) provide the highest level of evidence regarding the effectiveness of interventions. Less is known about how they are used to inform the design and reporting of RCTs.MethodsA sample of RCTs published in leading medical journals in 2007 was assessed to establish whether authors considered previous trials in the design of their trial. An approach to calculate the sample size required for a significant pooled effect in an updated meta-analysis was applied to a subsample of the RCTs to illustrate the ways in which the results of an existing meta-analysis can be incorporated into the planning and reporting of new RCTs.ResultsSix of the 27 trials assessed (22%) reported the use of previous trial(s) for sample size calculations. Meta-analyses relating the results of the trial to previous research were cited in 37% (10 out of 27) of the report discussion sections. Previous evidence is formally incorporated into retrospective sample size calculations for three of the trials.Discussion/ConclusionConsulting previous research before embarking on a new trial and basing decisions about future research on the impact on an updated meta-analysis will make the reporting of research more coherent and the design of new RCTs more efficient.     

Nonrandomized quality improvement intervention trials might overstate the strength of causal inference of their findings

ObjectiveTo assess the strength of causal inferences reported in randomized and nonrandomized evaluations of quality improvement (QI) interventions in relation to the study design and the direction of results for the primary outcomes.Study Design and SettingWe searched 11 journals for QI intervention studies that aimed to change clinician behavior. Statements that addressed the causal inference between intervention and outcomes were extracted and were rated by 34 researchers for the strength of causality.ResultsWe found 38 randomized controlled trials (RCTs) and 35 non-RCTs, and extracted 68 quotes from the abstracts and 139 from the main text. A significant interaction was found between study design and direction of results for the abstract quotes (P = 0.022). The ratings for non-RCTs were higher when the results were mixed, but for RCTs, they were higher if the results were positive or no effect, although none of the differences were statistically significant at α = 0.05 after adjusting for multiple comparisons. For the main text quotes, the causality rating was higher by 0.43 for RCTs than for non-RCTs after adjusting for the direction of results (P < 0.001).ConclusionAuthors might have overstated the strength of causal inference in the abstracts of non-RCTs, but appeared to report causality appropriately in the main text.     

Comparing vaccines: A systematic review of the use of the non-inferiority margin in vaccine trials

BackgroundNon-inferiority (NI) randomized controlled trials (RCTs) aim to demonstrate that a new treatment is no worse than a comparator that has already shown its efficacy over placebo within a pre-specified margin. However, clear guidelines on how the NI margin should be determined are lacking for vaccine trials. A difference (seroprevalence/risk) of 10% or a geometric mean titre/concentration (GMT) ratio of 1.5 or 2.0 in antibody levels is implicitly recommended for vaccine trials. We aimed to explore which NI margins were used in vaccine RCTs and how they were determined.MethodsA systematic search for NI vaccine RCTs yielded 177 eligible articles. Data were extracted from these articles using a standardized form and included general characteristics and characteristics specific for NI trials. Relations between the study characteristics and the NI margin used were explored.ResultsAmong the 143 studies using an NI margin based on difference (n = 136 on immunogenicity, n = 2 on efficacy and n = 5 on safety), 66% used a margin of 10%, 23% used margins lower than 10% (range 1–7.5%) and 11% used margins larger than 10% (range 11.5–25%). Of the 103 studies using a NI margin based on the GMT ratio, 50% used a margin of 0.67/1.5 and 49% used 0.5/2.0. As observed, 85% of the studies did not discuss the method of margin determination; and 19% of the studies lacked a confidence interval or p-value for non-inferiority.ConclusionMost NI vaccine RCTs used an NI margin of 10% for difference or a GMT ratio of 1.5 or 2.0 without a clear rationale. Most articles presented enough information for the reader to make a judgement about the NI margin used and the conclusions. The reporting on the design, margins used and results of NI vaccine trials could be improved; more explicit guidelines may help to achieve this end.     

Time-dependent bias was common in survival analyses published in leading clinical journals

OBJECTIVE: In survival analysis, "baseline immeasurable" time-dependent factors cannot be recorded at baseline, and change value after patient observation starts. Time-dependent bias can occur if such variables are not analyzed appropriately. This study sought to determine the prevalence of such time-dependent bias in highly-cited medical journals. STUDY DESIGN AND SETTING: We searched Medline databases to identify all observational studies that used a survival analysis in American Journal of Medicine, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal, Chest, Circulation, Journal of the American Medical Association, Lancet, and New England Journal of Medicine between 1998 and 2002. Studies with "baseline immeasurable" time-dependent factors were susceptible to time-dependent bias if a time-dependent covariate analysis was not used. RESULTS: Of 682 eligible studies, 127 (18.6%, 95% CI 15.8-21.8%) contained a "baseline immeasurable" time-dependent factor and 52 (7.6% [5.8-9.9%] of all survival analyses/40.9% [32.3-50.0%] of studies with a time-dependent factor) were susceptible to time-dependent bias. In 35 studies (5.1% [3.7-7.1%]/27.6% [20.5-35.9%]), the bias affected a variable highlighted in the study abstract and correction of the bias could have qualitatively changed the study's conclusion in over half of studies. CONCLUSION: In medical journals, time-dependent bias is concerningly common and frequently affects key factors and the study's conclusion.     

No differential attrition was found in randomized controlled trials published in general medical journals: a meta-analysis

ObjectiveDifferential attrition is regarded as a major threat to the internal validity of a randomized controlled trial (RCT). This study identifies the degree of differential attrition in RCTs covering a broad spectrum of clinical areas and factors that are related to this.Study Design and SettingA PubMed search was conducted to obtain a random sample of 100 RCTs published between 2008 and 2010 in journals from the ISI Web of Knowledge^SM category of medicine, general and internal. Eligibility criteria for selecting studies were primary publications of two-arm parallel randomized clinical trials, containing human participants and one or multiple follow-up measurements whose availability depended on the patients' willingness to participate.ResultsA significant amount of differential attrition was observed in 8% of the trials. The average differential attrition rate was 0.99 (95% confidence interval: 0.97–1.01), indicating no general difference in attrition rates between intervention and control groups. Moreover, no indication of heterogeneity was found, suggesting that the occurrence of differential attrition in the published literature is mostly a chance finding, unrelated to any particular design factors.ConclusionDifferential attrition did not generally occur in RCTs covering a broad spectrum of clinical areas within general and internal medicine.     

Has Female Authorship in Family Medicine Research Evolved Over Time?

PURPOSEStudies conducted in medical fields other than family medicine show that gender differences in publication rates are pronounced in many, but not all, fields of medicine. Our objective was to assess possible gender differences in publication rates in family medicine journals.METHODSUsing MEDLINE, we collected information on all journal articles published in 3 family medicine journals in the United States (Family Medicine, Journal of the American Board of Family Medicine, and Annals of Family Medicine) during the period 2008 to 2017. Gender of first and last author for each article was assigned using first names. The gender breakdown of the editorial boards during this time period was also examined.RESULTSFor the 3 journals combined during the period 2008 to 2017, 46.1% (1,209/2,623) of first authors were female, and 38.6% (857/2,223) of last authors were female. For all journals combined, there was a statistically significant increase in first authorship (43.2% in 2008 vs 52.1% in 2017; P<.001) and last authorship (28.8% in 2008 vs 41.8% in 2017; P <.001) over time. The editorial boards of the journals combined were 37.2% (279/749) female, and this did not increase significantly over the time period studied (35.5% in 2008 vs 39.2% in 2017; P=.49).CONCLUSIONSRepresentation of female authors in family medicine journals is increasing, yet last authorship remains low, and there is variation between journals in terms of gender equity. Future studies can evaluate the reason for these differences and offer solutions to publications as they try to increase their female authorship.Key words: bibliometrics, authorship, gender gap, academic medicine