Specific instructions for estimating unclearly reported blinding status in randomized trials were reliable and valid

ObjectiveTo test the reliability and validity of specific instructions to classify blinding, when unclearly reported in randomized trials, as “probably done” or “probably not done.”Study Design and SettingWe assessed blinding of patients, health care providers, data collectors, outcome adjudicators, and data analysts in 233 randomized trials in duplicate and independently using detailed instructions. The response options were “definitely yes,” “probably yes,” “probably no,” and “definitely no.” We contacted authors for data verification (46% response). For each of the five questions, we assessed reliability by calculating the agreement between the two reviewers and validity by calculating the agreement between reviewers’ consensus and verified data.ResultsThe percentage with unclear blinding status varied between 48.5% (patients) and 84.1% (data analysts). Reliability was moderate for blinding of outcome adjudicators (κ = 0.52) and data analysts (κ = 0.42) and substantial for blinding of patients (κ = 0.71), providers (κ = 0.68), and data collectors (κ = 0.65). The raw agreement between the consensus record and the author-verified record varied from 84.1% (blinding of data analysts) to 100% (blinding of health care providers).ConclusionWith the possible exception of blinding of data analysts, use of “probably yes” and “probably no” instead of “unclear” may enhance the assessment of blinding in trials.     

The risk of unblinding was infrequently and incompletely reported in 300 randomized clinical trial publications

ObjectivesTo assess the proportion of clinical trials explicitly reporting the risk of unblinding, to evaluate the completeness of reporting on unblinding risk, and to describe the reported procedures involved in assessing unblinding.Study Design and SettingWe sampled at random 300 blinded randomized clinical trials indexed in PubMed in 2010. Two authors read the trial publications and extracted data independently.ResultsTwenty-four trial publications, or 8% (95% confidence interval [CI], 5, 12%), explicitly reported the risk of unblinding, of which 16 publications, or 5% (95% CI, 3, 8%), reported compromised blinding; and 8 publications, or 3% (95% CI, 1, 5%), intact blinding. The reporting on risk of unblinding in the 24 trial publications was generally incomplete. The median proportion of assessments per trial affected by unblinding was 3% (range 1–30%). The most common mechanism for unblinding was perceptible physical properties of the treatments, for example, a difference in the taste and odor of a typhoid vaccine compared with its placebo.ConclusionPublished articles on randomized clinical trials infrequently reported risk of unblinding. This may reflect a tendency for avoiding reporting actual or suspected unblinding or a genuine low risk of unblinding.     

An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods

BACKGROUND AND OBJECTIVE: Readers of randomized controlled trials (RCTs) commonly assume that what was not reported did not occur. We undertook an observational study to determine whether concealment of randomization or blinding was used in RCTs that failed to report these bias-reducing strategies. METHODS: We recorded the reporting of concealment of randomization and blinding in 105 RCTs. We subsequently contacted the authors and determined if they had used these methodological safeguards. RESULTS: We successfully obtained data from 98 authors. The authors in the full-text publications of these 98 RCTs failed to report the presence or absence of concealment of randomization in 55%, and the blinding status of participants in 26%, health care providers in 64%, data collectors in 84%, outcome assessors in 83%, and data analysts in 96%. In direct contact, authors frequently reported concealing randomization (96%; 95% confidence interval CI=87-100%), blinding participants (20%; 95% CI=7-41%), blinding health care providers (65%; 95% CI=52-77%), blinding data collectors (65%; 95% CI=53-75%), blinding outcome assessors (79%; 95% CI=69-87%), and blinding data analysts (50%; 95% CI=40-60%), despite not reporting the use of these methodological safeguards in their publications. CONCLUSIONS: Readers should not assume that bias-reducing procedures not reported in an RCT did not occur.     

Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols

ObjectivesTo assess whether the reported methodological quality of randomized controlled trials (RCTs) reflects the actual methodological quality and to evaluate the association of effect size (ES) and sample size with methodological quality.Study Design and SettingSystematic review. This is a retrospective analysis of all consecutive phase III RCTs published by eight National Cancer Institute Cooperative Groups up to 2006. Data were extracted from protocols (actual quality) and publications (reported quality) for each study.ResultsFour hundred twenty-nine RCTs met the inclusion criteria. Overall reporting of methodological quality was poor and did not reflect the actual high methodological quality of RCTs. The results showed no association between sample size and actual methodological quality of a trial. Poor reporting of allocation concealment and blinding exaggerated the ES by 6% (ratio of hazard ratio [RHR]: 0.94; 95% confidence interval [CI]: 0.88, 0.99) and 24% (RHR: 1.24; 95% CI: 1.05, 1.43), respectively. However, actual quality assessment showed no association between ES and methodological quality.ConclusionThe largest study to date shows that poor quality of reporting does not reflect the actual high methodological quality. Assessment of the impact of quality on the ES based on reported quality can produce misleading results.     

Qualitative “trial-sibling” studies and “unrelated” qualitative studies contributed to complex intervention reviews

ObjectiveTo compare the contribution of “trial-sibling” and “unrelated” qualitative studies in complex intervention reviews.Study Design and SettingResearchers are using qualitative “trial-sibling” studies undertaken alongside trials to provide explanations to understand complex interventions. In the absence of qualitative “trial-sibling” studies, it is not known if qualitative studies “unrelated” to trials are helpful. Trials, “trial-sibling,” and “unrelated” qualitative studies looking at three health system interventions were identified. We looked for similarities and differences between the two types of qualitative studies, such as participants, intervention delivery, context, study quality and reporting, and contribution to understanding trial results.ResultsReporting was generally poor in both qualitative study types. We detected no substantial differences in participant characteristics. Interventions in qualitative “trial-sibling” studies were delivered using standardized protocols, whereas interventions in “unrelated” qualitative studies were delivered in routine care. Qualitative “trial-sibling” studies alone provided insufficient data to develop meaningful transferrable explanations beyond the trial context, and their limited focus on immediate implementation did not address all phenomena of interest. Together, “trial-sibling” and “unrelated” qualitative studies provided larger, richer data sets across contexts to better understand the phenomena of interest.ConclusionsFindings support inclusion of “trial-sibling” and “unrelated” qualitative studies to explore complexity in complex intervention reviews.     

Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding

BACKGROUND: Blinding can reduce bias in randomized clinical trials, but blinding procedures may be unsuccessful. Our aim was to assess how often randomized clinical trials test the success of blinding, the methods involved and how often blinding is reported as being successful. METHODS: We analysed a random sample of blinded randomized clinical trials indexed in the The Cochrane Central Register of Controlled Trials and published in 2001. We identified 1599 blinded trials, and noted if they had conducted any test for the success of blinding. We also selected 200 trials randomly that did not report any such test, and sent a questionnaire to the corresponding authors asking them if they had conducted any tests. RESULTS: Thirty-one out of 1599 trials (2%) reported tests for the success of blinding. Test methods varied, and reporting was generally incomplete. Blinding was considered successful in 14 out of the 31 trials (45%) and unclear in 10 (32%). Of the seven trials (23%) reporting unsuccessful blinding the risk of a biased trial result was either not addressed or was discounted in six cases. We received 130 questionnaires from trial authors (65%) of which 15 (12%) informed that they had conducted, but not published, tests. CONCLUSIONS: Blinding is rarely tested. Test methods vary, and the reporting of tests, and test results, is incomplete. There is a considerable methodological uncertainty how best to assess blinding, and an urgent need for improved methodology and improved reporting.     

Trial sample size,but not trial quality,is associated with positive study outcome

ObjectiveTo assess whether the reported trial quality or trial characteristics are associated with the trial outcome.Study Design and SettingWe identified all eligible randomized controlled trials (RCTs) of arthroplasty from 1997 and 2006. Trials were classified based on whether the main trial outcome was reported to be positive (n = 90) or negative (n = 94). Multivariable logistic regression analyses studied the association of reporting of trial-quality measures (blinding, placebo use, allocation procedure, overall quality) and trial characteristics (intervention type, number of patients/centers, funding) with positive trial outcome.ResultsRCTs that used placebo or blinded care providers, used pharmacological interventions, had higher Jadad quality scores or sample size of more than 100 patients were significantly more likely to report positive result in univariate analyses. Multivariable regression did not identify methodological quality of RCTs, but rather found that sample size was associated with trial outcome. Studies with more than 100 patients were 2.2 times more likely to report a positive result than smaller studies (P = 0.04).ConclusionsLack of association of reported trial quality with positive outcome in multivariable analyses suggests that previously observed association of reported study quality with study outcome in univariate analyses may be mediated by other study characteristics, such as study sample size.     

Radiographs of hip fractures were digitally altered to mask surgeons to the type of implant without compromising the reliability of quality ratings or making the rating process more difficult

ObjectiveTo devise and test techniques to blind outcome assessors in trials of hip fracture fixation.Study Design and SettingWe developed three techniques (Blackout, Subtraction, and Overlay) to mask radiographs of hip fractures fixated with cancellous screws or dynamic hip screws. Fifty orthopedic trauma surgeons each assessed 32 radiographs blinded with each technique.ResultsAll techniques achieved low rates of correct identification of screw type (14.9% for Blackout, 26.9% for Subtraction, 22.1% for Overlay) and high proportions of “don't know” responses (72.3%, 48.4%, 52.8%, respectively). The interrater reliability of reduction quality in the blinded images (intraclass correlation coefficient [ICC] = 0.55–0.57) was similar to the reliability of the unblinded radiographs (ICC = 0.60). Surgeons perceived 6.9% of the Overlay images as much more difficult to rate than unblinded radiographs, compared with 9.7% of Subtraction images (P = 0.25) and 28.0% of Blackout images (P < 0.001).ConclusionThree techniques of blinding radiographs of femoral neck fractures successfully mask surgeons to the type of implant fixated, do not compromise reliability of reduction ratings, and do not make rating most radiographs more difficult. Trialists should explore creative approaches to optimize blinding when designing trials, and should incorporate rigorous approaches to testing blinding success.     

A theoretical analysis showed that blinding cannot eliminate potential for bias associated with beliefs about allocation in randomized clinical trials

ObjectivesTo explore the theoretical justification for blinding in randomized trials and make recommendations concerning the implementation and interpretation of blinded randomized trials.Study Design and SettingA theoretical analysis was conducted of the potential for bias in randomized trials with successful blinding (ie, trials in which beliefs about allocation to treatment or control groups are independent of actual allocation). The analysis identified conditions that must be satisfied to ensure that blinding eliminates the potential for bias associated with beliefs about allocation.ResultsEven when beliefs about allocation are independent of actual allocation, they can still cause bias. The potential for bias is eliminated when the belief is uniformly one of complete ambivalence about allocation.ConclusionEven when blinding succeeds in making beliefs about allocation independent of actual allocation, beliefs about allocation may still cause bias. It is difficult to determine the extent of bias in any particular trial. Bias could be eliminated by establishing a state of complete ambivalence about the allocation of every trial participant, but universal ambivalence may be difficult to achieve and may reduce the generalizability of the trial's findings.     

Inadequate planning and reporting of adjudication committees in clinical trials: Recommendation proposal

ObjectivesAdjudication committees (ACs) are recommended in randomized controlled trials (RCTs) to standardize the assessment of events. We aimed to assess the reporting and functioning of ACs (synonyms: clinical event committees, endpoint committees) in clinical trials.Study Design and SettingWe searched five high-impact-factor medical journals for reports of RCTs with clinical event endpoints published between January 1, 2004 and December 31, 2005.ResultsACs were reported in 33.4% of the 314 reports of RCTs. ACs were reported in 29.6% of trials with low risk of misclassification (i.e., “hard” main outcome), in 47.5% of trials with medium risk of misclassification (i.e., subjective main outcome and intervention delivered in a blinded fashion) and in 31% of trials with high risk of misclassification (i.e., subjective main outcome without intervention delivered in a blinded fashion). Selected cases to be adjudicated consisted largely of events identified by site investigators (93.3%). Data provided to the AC were reported for 47.4% of ACs.ConclusionReporting of ACs is not fitted to the risk of biased misclassification. Important aspects of the functioning of ACs are insufficiently reported or raise methodological issues. We propose some recommendations for planning and reporting ACs in clinical trials.     

Reporting of methods was better in the Clinical Trials Registry-India than in Indian journal publications

ObjectiveWe sought to evaluate if editorial policies and the reporting quality of randomized controlled trials (RCTs) had improved since our 2004–05 survey of 151 RCTs in 65 Indian journals, and to compare reporting quality of protocols in the Clinical Trials Registry-India (CTRI).Study Design and SettingAn observational study of endorsement of Consolidated Standards for the Reporting of Trials (CONSORT) and International Committee of Medical Journal Editors (ICMJE) requirements in the instructions to authors in Indian journals, and compliance with selected requirements in all RCTs published during 2007–08 vs. our previous survey and between all RCT protocols in the CTRI on August 31, 2010 and published RCTs from both surveys.ResultsJournal policies endorsing the CONSORT statement (22/67, 33%) and ICMJE requirements (35/67, 52%) remained suboptimal, and only 4 of 13 CONSORT items were reported in more than 50% of the 145 RCTs assessed. Reporting of ethical issues had improved significantly, and that of methods addressing internal validity had not improved. Adequate methods were reported significantly more frequently in 768 protocols in the CTRI, than in the 296 published trials.ConclusionThe CTRI template facilitates the reporting of valid methods in registered trial protocols. The suboptimal compliance with CONSORT and ICMJE requirements in RCTs published in Indian journals reduces credibility in the reliability of their results.     

Changes to registration elements and results in a cohort of Clinicaltrials.gov trials were not reflected in published articles

ObjectivesTo assess effectiveness of legislative initiatives to stimulate public registration of trial results, we assessed adherence to protocol and results reporting, changes to registry, and publication data for randomized controlled trials (RCTs) after introduction of Food and Drug Administration Amendment Act (FDAAA).Study Design and SettingObservational study of a cohort of ClinicalTrials.gov registered FDAAA-covered RCTs found through ClinicalTrials.gov between 2009 and 2012 and data from corresponding publications. WHO Minimum Data Set items were abstracted by one author and verified by the other author.ResultsAmong 81 eligible trials, most were industry-funded, with a drug intervention in parallel assignment. Secondary outcomes at the initial and last registration were omitted for 17% and 19.7% of RCTs, respectively. RCT registration changes mostly involved scientific title (18.8%). Inclusion criteria omission was most common (88%) in publications. Inferential statistical methods for primary and secondary outcomes matched between registry and publication for 53.4% and 28.6% of RCTs, respectively. Serious and other adverse events (AEs) that were absent for 23.8% and 4.8% of RCTs, respectively, were published as nonoccurring.ConclusionDiscrepancies remain relatively high between registered and published outcomes, particularly regarding registered omissions in publications and concomitant reporting, nature of statistical method used, and reporting of AEs. This seriously undermines transparency of clinical trials and needs immediate attention of all stakeholders in health research.     

A pragmatic–explanatory continuum indicator summary (PRECIS): a tool to help trial designers

ObjectiveTo propose a tool to assist trialists in making design decisions that are consistent with their trial's stated purpose.Study Design and SettingRandomized trials have been broadly categorized as either having a pragmatic or explanatory attitude. Pragmatic trials seek to answer the question, “Does this intervention work under usual conditions?,” whereas explanatory trials are focused on the question, “Can this intervention work under ideal conditions?” Design decisions make a trial more (or less) pragmatic or explanatory, but no tool currently exists to help researchers make the best decisions possible in accordance with their trial's primary goal. During the course of two international meetings, participants with experience in clinical care, research commissioning, health care financing, trial methodology, and reporting defined and refined aspects of trial design that distinguish pragmatic attitudes from explanatory.ResultsWe have developed a tool (called PRECIS) with 10 key domains and which identifies criteria to help researchers determine how pragmatic or explanatory their trial is. The assessment is summarized graphically.ConclusionWe believe that PRECIS is a useful first step toward a tool that can help trialists to ensure that their design decisions are consistent with the stated purpose of the trial.     

A review of blinding in randomized controlled trials found results inconsistent and questionable

BACKGROUND AND OBJECTIVE: To determine methods to assess the success of blinding in randomized controlled trials (RCTs). METHODS: We searched MEDLINE, the Cochrane Controlled Trials Register, and the Cochrane Method Register and performed a manual search to target studies that attempt to assess blinding and describe the methods used in those studies. RESULTS: A total of 90 reports were selected. Reports assessed the success of blinding participants (n = 58), care providers (n = 36), and outcome assessors (n = 15). Of the 58 reports assessing the success of blinding participants, 54 (93%) reported asking participants to guess their treatment assignment. There was no consistency in timing of assessment (e.g., once at the end of the trial, 57%, or several times during the trial, 26%) or modalities of answering (e.g., "do not know" answers, 43%, or participants forced to guess, 31%). A statistical analysis was performed in 57% of reports. The statistical analysis mainly compared the proportion of correct guesses to those produced by chance (32%) or checked for a relation between participants' guesses and treatment assignment (23%). CONCLUSIONS: Methods of assessing the success of blinding, analysis and reporting the results were inconsistent and questionable.     

Research and Policy Impact of Trials Published by the UK National Institute of Health Research (2006-2015)

ObjectivesHealth technology assessment aims to inform and support healthcare decision making, and trials are part of that process. The purpose of this study was to measure the impact of a sample of trials in a meaningful but robust fashion.MethodsAll randomized controlled trials funded and published by the UK National Institute of Health Research in the Health Technology Assessment journal series and other peer-reviewed journals were identified for 2006 to 2015. Citation analysis was performed for all trials, and quantitative content analysis was undertaken on a purposive sample to determine whether impact could be categorized as “instrumental” (ie, having a clear influence on key research and policy publications).ResultsThe search identified 133 relevant trials. The citation rate per trial was 102.97. Of the 133 trials, 129 (98%) were cited in 1 or more systematic reviews or meta-analyses (mean per trial = 7.18, range = 0-44). Where they were cited, the trials were used in some form of synthesis 63% of the time. Ninety-one of the 133 (68%) trials were found to be cited in 1 or more guidance or policy document (mean per trial = 2.75, range = 0-26) and had an instrumental influence 41% of the time. The publication of these trials’ results in journals other than the Health Technology Assessment journal appears to enhance the discoverability of the trial data. Altmetric.com proved to be very useful in identifying unique policy and guidance documents.ConclusionThese trials have impressive citation rates, and a sizeable proportion are certainly being used in key publications in a genuinely instrumental manner.     

Randomized comparison of the safety of Flublok® versus licensed inactivated influenza vaccine in healthy,medically stable adults ≥50 years of age

BackgroundThe safety and tolerability of Flublok^®, a purified recombinant hemagglutinin seasonal influenza vaccine, was compared to AFLURIA^® in a randomized, blinded clinical trial in adults ≥50 years of age with attention to hypersensitivity reactions.MethodsThis blinded, randomized trial of healthy adults ≥50 years of age compared safety of Flublok vs. AFLURIA with respect to pre-specified possible hypersensitivity: “rash,” “urticaria,” “swelling” and “non-dependent edema;” solicited reactogenicity and unsolicited adverse events. Subject-reported outcomes were collected for 30 days after vaccination. All adverse event terms were reviewed by physicians blinded to vaccine group, who added other terms possibly reflecting hypersensitivity. Case records of subjects with possible hypersensitivity were adjudicated by independent experts blinded to treatment assignment to identify likely hypersensitivity reactions. Non-inferiority of the incidence of hypersensitivity in the two vaccine groups was pre-defined as an absolute difference with an upper bound of 2-sided 95% confidence limits ≤0.015.ResultsA total of 2640 subjects were enrolled, evenly split in age cohorts of 50–64 and ≥65 years. Fifty-two subjects reported at least one term possibly representing hypersensitivity, with a slight imbalance of 31 on Flublok and 21 on AFLURIA. The adjudicators determined that six and four subjects on Flublok and AFLURIA, respectively, likely met clinical criteria for hypersensitivity, yielding a difference in incidence between the two vaccine groups of 0.15% (upper bound of 2-sided 95% CI = 0.9%). Reactogenicity and overall adverse event profiles were similar across both vaccines.ConclusionsFlublok was non-inferior to AFLURIA in adults ≥50 years of age with respect to expert-adjudicated events of likely hypersensitivity during 30 days following vaccination (Sponsored by Protein Sciences Corporation; ClinicalTrials.gov number NCT01825200).     

Influence of trial registration on reporting quality of randomized trials: Study from highest ranked journals

ObjectiveTo evaluate the reporting quality of key methodological items of randomized control trials (RCTs) in 55 of the highest ranked journals.Study Design and SettingA list of the highest top ranked journals was identified, and a search for detecting RCTs in those journals was made. Two hundred sixty four journals were screened and 55 of them were identified having at least one RCT. Three RCTs were randomly selected a priori from each journal; 148 RCTs were finally included. RCTs were assessed by two reviewers using the Consolidated Standards of Reporting Trials (CONSORT) statement.ResultsOnly 11 (8%) RCTs had all items adequately reported. In addition, 36% of RCTs reported that the study was registered in any trial registry. We found a significant difference in the quality of reporting for baseline characteristics, recruitment, participant's flow, and randomization implementation between those studies having reported the registration of their RCT in a trial registry and those that have not. Adherence to key methodological items of the CONSORT statement was as follows: sample size determination (60%), sequence generation (49%), allocation concealment (40%), and blinding (25%).ConclusionsReporting of varied CONSORT items remains suboptimal. Registration in a trial registry was associated with improved reporting. Further efforts to enhance RCT registration could contribute to this improvement.     

Searching for unpublished trials in Cochrane reviews may not be worth the effort

ObjectiveTo assess the value of searching for unpublished data by exploring the extent to which Cochrane reviews include unpublished data and by evaluating the quality of unpublished trials.Study Design and SettingWe screened all 2,462 completed Cochrane reviews published since 2000 in the Cochrane Database of Systematic Reviews Issue 3, 2006. In a random sample (n = 61) of 292 reviews, including unpublished trials, we studied all 116 references.ResultsUnpublished trials make up 8.8% of all included trials in our sample. Thirty-eight percent of the “unpublished” trials have in fact been published. Allocation concealment was “unclear” or not adequate in 54.3% and 61.3% reported blinding. In 47.2% reported withdrawal rates were >20%. Trials that were eventually published had larger mean population sizes (P-value, 0.02). Of the reported sponsors, 87.3% were drug companies. Methodological quality and publication bias are mentioned in half of the reviews and explored in a third. Quality ratings did not have consequences for pooling, because 82.8% was included in the forest plots.ConclusionsA minority of Cochrane reviews include “unpublished trials” and many of these are eventually published. Truly unpublished studies have poor or unclear methodological quality. Therefore, it may be better to invest in regular updating of reviews, rather than in extensive searching for unpublished data.