Pseudo cluster randomization dealt with selection bias and contamination in clinical trials

BACKGROUND AND OBJECTIVES: When contamination is present, randomization on a patient level leads to dilution of the treatment effect. The usual solution is to randomize on a cluster level, but at the cost of efficiency and more importantly, this may introduce selection bias. Furthermore, it may slow down recruitment in the clusters that are randomized to the "less interesting" treatment. We discuss an alternative randomization procedure to approach these problems. METHODS: Pseudo cluster randomization is a two-stage randomization procedure that balances between individual randomization and cluster randomization. For common scenarios, the design factors needed to calculate the appropriate sample size are tabulated. RESULTS: A pseudo cluster randomized design can reduce selection bias and contamination, while maintaining good efficiency and possibly improving enrollment. To make a well-informed choice of randomization procedure, we discuss the advantages of each method and provide a decision flow chart. CONCLUSION: When contamination is thought to be substantial in an individually randomized setting and a cluster randomized design would suffer from selection bias and/or slow recruitment, pseudo cluster randomization can be considered.     

Multivariate network meta‐analysis to mitigate the effects of outcome reporting bias

Outcome reporting bias (ORB) is recognized as a threat to the validity of both pairwise and network meta‐analysis (NMA). In recent years, multivariate meta‐analytic methods have been proposed to reduce the impact of ORB in the pairwise setting. These methods have shown that multivariate meta‐analysis can reduce bias and increase efficiency of pooled effect sizes. However, it is unknown whether multivariate NMA (MNMA) can similarly reduce the impact of ORB. Additionally, it is quite challenging to implement MNMA due to the fact that correlation between treatments and outcomes must be modeled; thus, the dimension of the covariance matrix and number of components to estimate grows quickly with the number of treatments and number of outcomes. To determine whether MNMA can reduce the effects of ORB on pooled treatment effect sizes, we present an extensive simulation study of Bayesian MNMA. Via simulation studies, we show that MNMA reduces the bias of pooled effect sizes under a variety of outcome missingness scenarios, including missing at random and missing not at random. Further, MNMA improves the precision of estimates, producing narrower credible intervals. We demonstrate the applicability of the approach via application of MNMA to a multi‐treatment systematic review of randomized controlled trials of anti‐depressants for the treatment of depression in older adults.     

A Markov chain approach for ranking treatments in network meta‐analysis

When interpreting the relative effects from a network meta‐analysis (NMA), researchers are usually aware of the potential limitations that may render the results for some comparisons less useful or meaningless. In the presence of sufficient and appropriate data, some of these limitations (eg, risk of bias, small‐study effects, publication bias) can be taken into account in the statistical analysis. Very often, though, the necessary data for applying these methods are missing and data limitations cannot be formally integrated into ranking. In addition, there are other important characteristics of the treatment comparisons that cannot be addressed within a statistical model but only through qualitative judgments; for example, the relevance of data to the research question, the plausibility of the assumptions, and so on. Here, we propose a new measure for treatment ranking called the Probability of Selecting a Treatment to Recommend (POST‐R). We suggest that the order of treatments should represent the process of considering treatments for selection in clinical practice and we assign to each treatment a probability of being selected. This process can be considered as a Markov chain model that allows the end‐users of NMA to select the most appropriate treatments based not only on the NMA results but also to information external to the NMA. In this way, we obtain rankings that can inform decision‐making more efficiently as they represent not only the relative effects but also their potential limitations. We illustrate our approach using a NMA comparing treatments for chronic plaque psoriasis and we provide the Stata commands.     

Estimating causal contrasts involving intermediate variables in the presence of selection bias

An important goal across the biomedical and social sciences is the quantification of the role of intermediate factors in explaining how an exposure exerts an effect on an outcome. Selection bias has the potential to severely undermine the validity of inferences on direct and indirect causal effects in observational as well as in randomized studies. The phenomenon of selection may arise through several mechanisms, and we here focus on instances of missing data. We study the sign and magnitude of selection bias in the estimates of direct and indirect effects when data on any of the factors involved in the analysis is either missing at random or not missing at random. Under some simplifying assumptions, the bias formulae can lead to nonparametric sensitivity analyses. These sensitivity analyses can be applied to causal effects on the risk difference and risk‐ratio scales irrespectively of the estimation approach employed. To incorporate parametric assumptions, we also develop a sensitivity analysis for selection bias in mediation analysis in the spirit of the expectation–maximization algorithm. The approaches are applied to data from a health disparities study investigating the role of stage at diagnosis on racial disparities in colorectal cancer survival. Copyright © 2016 John Wiley & Sons, Ltd.     

Assessing the impact of selection bias on test decisions in trials with a time‐to‐event outcome

If past treatment assignments are unmasked, selection bias may arise even in randomized controlled trials. The impact of such bias can be measured by considering the type I error probability. In case of a normally distributed outcome, there already exists a model accounting for selection bias that permits calculating the corresponding type I error probabilities. To model selection bias for trials with a time‐to‐event outcome, we introduce a new biasing policy for exponentially distributed data. Using this biasing policy, we derive an exact formula to compute type I error probabilities whenever an F‐test is performed and no observations are censored. Two exemplary settings, with and without random censoring, are considered in order to illustrate how our results can be applied to compare distinct randomization procedures with respect to their performance in the presence of selection bias. © 2017 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

Investigating the Impact of Selection Bias in Dose-Response Analyses of Preventive Interventions

This paper focuses on the impact of selection bias in the context of extended, community-based prevention trials that attempt to “unpack” intervention effects and analyze mechanisms of change. Relying on dose-response analyses as the most general form of such efforts, this study provides two examples of how selection bias can affect the estimation of treatment effects. In Example 1, we describe an actual intervention in which selection bias was believed to influence the dose-response relation of an adaptive component in a preventive intervention for young children with severe behavior problems. In Example 2, we conduct a series of Monte Carlo simulations to illustrate just how severely selection bias can affect estimates in a dose-response analysis when the factors that affect dose are not recorded. We also assess the extent to which selection bias is ameliorated by the use of pretreatment covariates. We examine the implications of these examples and review trial design, data collection, and data analysis factors that can reduce selection bias in efforts to understand how preventive interventions have the effects they do.     

Selection and bias—Two hostile brothers

We consider the situation where in a first stage of a clinical trial several treatments are compared with a single control and the ‘best’ treatment(s) are selected in an interim analysis to be carried on to the second stage. We quantify the mean bias and mean square error of the conventional estimates after selection depending on the number of treatments and the selection time during the trial. The cases without or with reshuffling the planned sample size of the dropped treatments to the selected ones are investigated. The mean bias shows very different patterns depending on the selection rule and the unknown parameter values. We stress the fact that the quantification of the bias is possible only in designs with planned adaptivity where the design allows reacting to new evidence, but the decision rules are laid down in advance. Finally, we calculate the mean bias which arises in a simple but influential regulatory selection rule, to register a new medical therapy only when two pivotal trials have both proven an effect by a statistical test. Copyright © 2009 John Wiley & Sons, Ltd.     

Studywise minimization: A treatment allocation method that improves balance among treatment groups and makes allocation unpredictable

ObjectivesIn randomized controlled trials with many potential prognostic factors, serious imbalance among treatment groups regarding these factors can occur. Minimization methods can improve balance but increase the possibility of selection bias. We described and evaluated the performance of a new method of treatment allocation, called studywise minimization, that can avoid imbalance by chance and reduce selection bias.Study Design and SettingThe studywise minimization algorithm consists of three steps: (1) calculate the imbalance for all possible allocations, (2) list all allocations with minimum imbalance, and (3) randomly select one of the allocations with minimum imbalance. We carried out a simulation study to compare the performance of studywise minimization with three other allocation methods: randomization, biased-coin minimization, and deterministic minimization. Performance was measured, calculating maximal and average imbalance as a percentage of the group size.ResultsIndependent of trial size and number of prognostic factors, the risk of serious imbalance was the highest in randomization and absent in studywise minimization. The largest differences among the allocation methods regarding the risk of imbalance were found in small trials.ConclusionStudywise minimization is particularly useful in small trials, where it eliminates the risk of serious imbalances without generating the occurrence of selection bias.     

Adjusting for observable selection bias in block randomized trials

In this paper, we propose a model-based approach to detect and adjust for observable selection bias in a randomized clinical trial with two treatments and binary outcomes. The proposed method was evaluated using simulations of a randomized block design in which the investigator favoured the experimental treatment by attempting to enroll stronger patients (with greater probability of treatment success) if the probability of the next treatment being experimental was high, and enroll weak patients (with less probability of treatment success) if the probability of the next treatment being experimental was low. The method allows not only testing for the presence of observable selection bias, but also testing for a difference in treatment effects, adjusting for possible selection bias.     

INCREMENTAL EXPENDITURE OF BIOLOGIC DISEASE MODIFYING ANTIRHEUMATIC TREATMENT USING INSTRUMENTAL VARIABLES IN PANEL DATA

In health care, decision makers are generally interested in simultaneous comparisons among multiple treatments or interventions available as treatment choices in real‐world clinical setting. The lack of random assignment to treatment in real‐world clinical settings leads to selection‐bias issues when evaluating the marginal benefits of treatment. The application of instrumental variables (IV) estimation to mitigate selection bias has traditionally been limited to comparing only two treatments/interventions concurrently. Using the case of biologic treatment in rheumatoid arthritis, we describe a generalized method of moments (GMM)–based panel data IV (IV‐GMM) framework, to simultaneously estimate multiple treatment effects in the presence of time‐varying selection bias and time‐invariant heterogeneity. To satisfy the order and rank conditions for identification with multiple endogeneity, we propose lagged values of each treatment as excluded instruments. We evaluate the validity of the IV estimation assumptions on instrument relevance and exogeneity. Results indicate that the IV‐GMM model offers enhanced control over selection bias and heterogeneity, and more importantly the panel data framework can provide valid excluded instruments that satisfy the order and rank conditions for identification when dealing with multiple endogenous variables. The approach outlined in this article has broad application for comparative effectiveness and health technology assessment involving multiple treatments/interventions using real‐world nonexperimental data. Copyright © 2012 John Wiley & Sons, Ltd.     

A theoretical analysis showed that blinding cannot eliminate potential for bias associated with beliefs about allocation in randomized clinical trials

ObjectivesTo explore the theoretical justification for blinding in randomized trials and make recommendations concerning the implementation and interpretation of blinded randomized trials.Study Design and SettingA theoretical analysis was conducted of the potential for bias in randomized trials with successful blinding (ie, trials in which beliefs about allocation to treatment or control groups are independent of actual allocation). The analysis identified conditions that must be satisfied to ensure that blinding eliminates the potential for bias associated with beliefs about allocation.ResultsEven when beliefs about allocation are independent of actual allocation, they can still cause bias. The potential for bias is eliminated when the belief is uniformly one of complete ambivalence about allocation.ConclusionEven when blinding succeeds in making beliefs about allocation independent of actual allocation, beliefs about allocation may still cause bias. It is difficult to determine the extent of bias in any particular trial. Bias could be eliminated by establishing a state of complete ambivalence about the allocation of every trial participant, but universal ambivalence may be difficult to achieve and may reduce the generalizability of the trial's findings.     

Empirical evaluation showed that the Copas selection model provided a useful summary in 80% of meta-analyses

ObjectiveAlthough using meta-analysis to combine evidence from a number of studies should reduce both bias and uncertainty, it is sometimes not the case, because published studies represent a biased selection of the evidence. Copas proposed a selection model to assess the sensitivity of meta-analysis conclusions to possible selection bias. However, this relatively complex model awaits both reliable software and an empirical evaluation. This article reports work addressing both these issues.Study Design and SettingWe took 157 meta-analyses with binary outcomes, analyzed each one using the Copas selection model, and evaluated each analysis using a prespecified protocol. The evaluation aimed to assess the usefulness of the Copas selection model to a typical Cochrane reviewer.ResultsIn approximately 80% of meta-analyses, the overall interpretation of the Copas selection model was clear, with better results among the 22 with evidence of selection bias. However, as with the “Trim and Fill” method, allowing for selection bias can result in smaller standard errors for the treatment estimate.ConclusionWhen a reliable test for selection bias is significant, we recommend systematic reviewers to try the Copas selection model, although the results should be interpreted cautiously.     

Need for high-quality studies in health technology assessments: the case of a systematic review of treatment for retinoblastoma

OBJECTIVES: The aim of this study was to conduct a systematic review of the evidence for treatments for retinoblastoma in children. METHODS: Seventeen electronic databases were searched. Two reviewers independently selected studies. Studies of participants diagnosed with childhood retinoblastoma, any interventions, and all clinical outcomes were eligible. Randomized and nonrandomized controlled trials and cohort studies with clear comparisons between treatment groups were included. Methodological quality was assessed. RESULTS: Thirty-one observational comparative studies were included, of which twenty-seven were retrospective. The methodological quality was generally poor, with a high risk of selection bias in all studies. Although there were high levels of treatment success in many of the studies, due to the limitations of the evidence identified, it was not possible to make meaningful and robust conclusions about the relative effectiveness of different treatment approaches for retinoblastoma in children. CONCLUSIONS: Good quality randomized controlled trials are required. Where controlled trials are not feasible, only high quality prospective, nonrandomized studies should be given consideration, due to the generally higher risk of bias in retrospective studies.     

MM4 Reconciling Decision Models with the Real World: The Cost-Effectiveness of Erythropoietin for Esrd-Related Anemia

The choice of data used in decision modeling of health care interventions divides analysis into two groups: those who favor randomized clinical trial (RCT) data and those who prefer "real world" data. This decision may have serious consequences if the end result is to inform health care policy. This workshop employs a case study to (1) show how differences in the reality of clinical practice and the rigor of RCTs can lead to biases when decision models use RCT data to evaluate policy issues and (2) provide a method of updating decision models with claims/outcomes data to overcome this bias. We highlight three specific problems associated with the use of RCT data which may create misleading results: randomization and sample selection bias, clinically appropriate comparator groups, and indirect treatment effects. These issues are illustrated with a decision model analyzing Medicare's coverage of erythropoietin (EPO) for patients with End-Stage Renal Disease (ESRD). We show how logistic and multiple regression can be used to estimate branch probabilities and payoffs for each treatment group. The incorporation of additional data from the United States Renal Data System into the model enables us to update probabilities and payoffs when patients are not randomly assigned to treatment modalities. To highlight the potential bias that exists when models rely solely on RCT data, we compare our results to a previous study in which the authors employed a computerized decision model to estimate the net costs to Medicare of EPO coverage at 1 and 5 years. This exercise will offer policy analysts and others a method of updating RCT-based decision models to more accurately reflect clinical practice and predict policy effects.     

New algorithm for treatment allocation reduced selection bias and loss of power in small trials

OBJECTIVE: In clinical trials, patients become available for treatment sequentially. Especially in trials with a small number of patients, loss of power may become an important issue, if treatments are not allocated equally or if prognostic factors differ between the treatment groups. We present a new algorithm for sequential allocation of two treatments in small clinical trials, which is concerned with the reduction of both selection bias and imbalance. STUDY DESIGN AND SETTING: With the algorithm, an element of chance is added to the treatment as allocated by minimization. The amount of chance depends on the actual amount of imbalance of treatment allocations of the patients already enrolled. The sensitivity to imbalance may be tuned. We performed trial simulations with different numbers of patients and prognostic factors, in which we quantified loss of power and selection bias. RESULTS: With our method, selection bias is smaller than with minimization, and loss of power is lower than with pure randomization or treatment allocation according to a biased coin principle. CONCLUSION: Our method combines the conflicting aims of reduction of bias by predictability and reduction of loss of power, as a result of imbalance. The method may be of use in small trials.     

Comparing longitudinal binary outcomes in an observational oral health study

Observational studies continue to be recognized as viable alternatives to randomized trials when making treatment group comparisons, in spite of drawbacks due mainly to selection bias. Sample selection models have been proposed in the economics literature, and more recently in the medical literature, as a method to adjust for selection bias due to observed and unobserved confounders in observational studies. Application of these models has been limited to cross-sectional observational data and to outcomes that are continuous in nature. In this paper we extend application of these models to include longitudinal studies and binary outcomes. We apply a two-stage probit model using GEE to account for correlated longitudinal binary chewing difficulty outcomes. Chewing difficulty was measured every six months during a 24-month period between two groups of subjects: those either receiving or not receiving dental care. Dental care use was measured at six-month intervals as well. Results from our proposed model are compared to results using a standard GEE model that ignores the potential selection bias introduced by unobserved confounders. In this application, accounting for selection bias made a major difference in the substantive conclusions about the outcomes of interest. This is due in part to an adverse selection phenomenon in which those most in need of treatment (and consequently most likely to benefit from it) are actually the ones least likely to seek treatment. Our application of sample selection models to binary longitudinal observational outcome data should serve as impetus for increased utilization of this promising set of models to other health outcomes studies.     

On estimating efficacy from clinical trials

We define 'biologic efficacy' as the effect of treatment for all persons who receive the therapeutic agent to which they were assigned. It measures the biologic action of treatment among compliant persons. In a randomized trial with one treatment and one placebo control, one can theoretically estimate efficacy by comparing persons who complete the treatment regimen with controls who similarly complete the control regimen. In practice, however, we make this comparison with reservation because a control protocol often presents a different challenge for compliance than does the treatment, so that the compliant subgroups are not comparable. Standard practice employs intent-to-treat comparisons in which one compares those randomized to treatment and control without reference to whether they actually received the treatment. Intent-to-treat comparisons estimate the 'programmatic effectiveness' of a treatment rather than its biologic efficacy. This paper introduces and derives the statistical properties of an alternative estimator of biologic efficacy that avoids the potential selection bias inherent in a comparison of compliant subgroups. The method applies to randomized trials with a dichotomous outcome measure, whether or not a placebo is given to the control group. The idea is to compare the compliers in the treatment group to an inferred control subgroup chosen to eliminate selection bias. The methodology was motivated by and is illustrated in the analysis of a randomized community trial of the impact of vitamin A supplementation on children's mortality.     

Quantification methods were developed for selection bias by predictability of allocations with unequal block randomization

BACKGROUND AND OBJECTIVE: A selection of patients for a controlled clinical trial may be biased because of prior knowledge of the treatment. With randomized blocks of known or guessed lengths, some allocations can be predicted with certainty. Previously described methods determine the proportion of predictable cases for blocks of equal lengths. It may be useful to make a calculation for unequal blocks as well to find a method that reduces this predictability. STUDY DESIGN AND SETTING: Quantification methods are developed for series of two and three unequal blocks, using the probability of identifying a long block when it comes before a short one if it starts with a sequence incompatible with the content of a short block. Results are compared with the recently described maximal allocation procedure. RESULTS: Predictability is not always reduced by unequal blocks and is even worse in some cases, compared to equal blocks. Predictability is not necessarily decreased with the maximal allocation procedure. CONCLUSIONS: Before choosing an allocation method, it is important to quantify the predictability of possible options to reduce selection bias. Several practical recommendations are formulated for choosing methods, taking this risk of bias into account.     

Adjusting for treatment selection in phase II/III clinical trials with time to event data

Phase II/III clinical trials are efficient two-stage designs that test multiple experimental treatments. In stage 1, patients are allocated to the control and all experimental treatments, with the data collected from them used to select experimental treatments to continue to stage 2. Patients recruited in stage 2 are allocated to the selected treatments and the control. Combined data of stage 1 and stage 2 are used for a confirmatory phase III analysis. Appropriate analysis needs to adjust for selection bias of the stage 1 data. Point estimators exist for normally distributed outcome data. Extending these estimators to time to event data is not straightforward because treatment selection is based on correlated treatment effects and stage 1 patients who do not get events in stage 1 are followed-up in stage 2. We have derived an approximately uniformly minimum variance conditional unbiased estimator (UMVCUE) and compared its biases and mean squared errors to existing bias adjusted estimators. In simulations, one existing bias adjusted estimator has similar properties as the practically unbiased UMVCUE while the others can have noticeable biases but they are less variable than the UMVCUE. For confirmatory phase II/III clinical trials where unbiased estimators are desired, we recommend the UMVCUE or the existing estimator with which it has similar properties.     

Comparing Standard Regression, Propensity Score Matching, and Instrumental Variables Methods for Determining the Influence of Mammography on Stage of Diagnosis

In situations where randomized trials are not feasible, analysis of observational data must be used instead. However, when using observational data, there is often selection bias for which we must account in order to adjust for pre-treatment differences between groups in their baseline characteristics. As an example of this, we used the Linked Medicare-Tumor Registry Database created by the National Cancer Institute and the Centers for Medicare and Medicaid Services to look at screening with mammography in older women to determine its effectiveness in detecting cancer at an earlier stage. The standard regression method and two methods of adjusting for selection bias are compared. We start with the standard analysis, a logistic regression predicting stage at diagnosis that includes as independent variables a set of covariates to adjust for differences in baseline risk plus an indicator variable for whether the woman used screening. Next, we employ propensity score matching, which evens out the distribution of measured baseline characteristics across groups, and is more robust to model mis-specification than the standard analysis. Lastly, we conduct an instrumental variable analysis, which addresses unmeasured differences between the users and non-users. This article compares these methods and discusses issues of which researchers and analysts should be aware. It is important to look beyond the standard analysis and to consider propensity score matching when there is concern about group differences in measured covariates and instrumental variable analysis when there is concern about differences in unmeasured covariates.