A dynamic adaptation of the propensity score adjustment for effectiveness analyses of ordinal doses of treatment

The propensity score adjustment is a method to reduce bias in observational studies. We propose a strategy that involves a novel combination of three data analytic techniques, which adapts the propensity adjustment for additional perturbations of longitudinal, observational studies. First, ordinal logistic regression examines propensity for ordinal doses of treatment. Second, a mixed-model approach incorporates the multiple treatment trials and multiple episodes that are characteristic of chronically ill subjects. Finally, a mixed-effects grouped-time survival model incorporates the propensity score in treatment effectiveness analyses. The strategy that is applied here to an observational study of affective illness can also be used to evaluate the effectiveness of treatments for other chronic illnesses.     

Bias reduction in effectiveness analyses of longitudinal ordinal doses with a mixed-effects propensity adjustment

A mixed-effects propensity adjustment is described that can reduce bias in longitudinal studies involving non-equivalent comparison groups. There are two stages in this data analytic strategy. First, a model of propensity for treatment intensity examines variables that distinguish among subjects who receive various ordered doses of treatment across time using mixed-effects ordinal logistic regression. Second, the effectiveness model examines multiple times until recurrence to compare the ordered doses using a mixed-effects grouped-time survival model. Effectiveness analyses are initially stratified by propensity quintile. Then the quintile-specific results are pooled, assuming that there is not a propensity x treatment interaction. A Monte Carlo simulation study compares bias reduction in fully specified propensity model relative to misspecified models. In addition, type I error rate and statistical power are examined. The approach is illustrated by applying it to a longitudinal, observational study of maintenance treatment of major depression.     

A mixed-effects quintile-stratified propensity adjustment for effectiveness analyses of ordered categorical doses

Observational studies can be used to evaluate treatment effectiveness among patients with a broader range of illness severity than typically seen in randomized controlled clinical trials. However, there are several difficulties with observational evaluations including non-equivalent comparison groups, treatment doses and durations that vary widely, and, in longitudinal studies, multiple courses of treatment per subject. A mixed-effects approach to the propensity adjustment for non-equivalent comparison groups is described that can account for each of these perturbations. The strategy involves two stages. First, characteristics that distinguish among subjects who receive various levels of treatment are examined in a model of propensity for treatment intensity using mixed-effects ordinal logistic regression. Second, the propensity-stratified effectiveness of ordered categorical doses is compared in a mixed-effects grouped time survival model of time until recovery. The model is applied in a longitudinal, observational study of antidepressant effectiveness. Then a Monte Carlo simulation study indicates that the strategy has acceptable type I error rates and minimal bias in the estimates of treatment effectiveness. Statistical power exceeds 0.90 for an odds ratio of 1.5 with N = 250 and 500, and is acceptable for an odds ratio of 2.0 with N = 100. Nevertheless, with N = 100, the models that had high intraclass correlation coefficients had greater tendency towards non-convergence. This approach is a useful strategy for observational studies of treatment effectiveness. It is capable of adjusting for selection bias, incorporating multiple observations per subject, and comparing effectiveness of ordinal doses.     

The performance of different propensity score methods for estimating marginal odds ratios

The propensity score which is the probability of exposure to a specific treatment conditional on observed variables. Conditioning on the propensity score results in unbiased estimation of the expected difference in observed responses to two treatments. In the medical literature, propensity score methods are frequently used for estimating odds ratios. The performance of propensity score methods for estimating marginal odds ratios has not been studied. We performed a series of Monte Carlo simulations to assess the performance of propensity score matching, stratifying on the propensity score, and covariate adjustment using the propensity score to estimate marginal odds ratios. We assessed bias, precision, and mean-squared error (MSE) of the propensity score estimators, in addition to the proportion of bias eliminated due to conditioning on the propensity score. When the true marginal odds ratio was one, then matching on the propensity score and covariate adjustment using the propensity score resulted in unbiased estimation of the true treatment effect, whereas stratification on the propensity score resulted in minor bias in estimating the true marginal odds ratio. When the true marginal odds ratio ranged from 2 to 10, then matching on the propensity score resulted in the least bias, with a relative biases ranging from 2.3 to 13.3 per cent. Stratifying on the propensity score resulted in moderate bias, with relative biases ranging from 15.8 to 59.2 per cent. For both methods, relative bias was proportional to the true odds ratio. Finally, matching on the propensity score tended to result in estimators with the lowest MSE.     

A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003

Propensity-score methods are increasingly being used to reduce the impact of treatment-selection bias in the estimation of treatment effects using observational data. Commonly used propensity-score methods include covariate adjustment using the propensity score, stratification on the propensity score, and propensity-score matching. Empirical and theoretical research has demonstrated that matching on the propensity score eliminates a greater proportion of baseline differences between treated and untreated subjects than does stratification on the propensity score. However, the analysis of propensity-score-matched samples requires statistical methods appropriate for matched-pairs data. We critically evaluated 47 articles that were published between 1996 and 2003 in the medical literature and that employed propensity-score matching. We found that only two of the articles reported the balance of baseline characteristics between treated and untreated subjects in the matched sample and used correct statistical methods to assess the degree of imbalance. Thirteen (28 per cent) of the articles explicitly used statistical methods appropriate for the analysis of matched data when estimating the treatment effect and its statistical significance. Common errors included using the log-rank test to compare Kaplan-Meier survival curves in the matched sample, using Cox regression, logistic regression, chi-squared tests, t-tests, and Wilcoxon rank sum tests in the matched sample, thereby failing to account for the matched nature of the data. We provide guidelines for the analysis and reporting of studies that employ propensity-score matching.     

Performance of a propensity score adjustment in longitudinal studies with covariate-dependent representation

Longitudinal observational studies provide rich opportunities to examine treatment effectiveness during the course of a chronic illness. However, there are threats to the validity of observational inferences. For instance, clinician judgment and self‐selection play key roles in treatment assignment. To account for this, an adjustment such as the propensity score can be used if certain assumptions are fulfilled. Here, we consider a problem that could surface in a longitudinal observational study and has been largely overlooked. It can occur when subjects have a varying number of distinct periods of therapeutic intervention. We evaluate the implications of baseline variables in the propensity model being associated with the number of post baseline observations per subject and refer to it as ‘covariate‐dependent representation’. An observational study of antidepressant treatment effectiveness serves as a motivating example. The analyses examine the first 20 years of follow‐up data from the National Institute of Mental Health Collaborative Depression Study, a longitudinal, observational study. A simulation study evaluates the consequences of covariate‐dependent representation in longitudinal observational studies of treatment effectiveness under a range of data specifications.The simulations found that estimates were adversely affected by underrepresentation when there was lower ICC among repeated doses and among repeated outcomes. Copyright © 2012 John Wiley & Sons, Ltd.     

Comparing treatments via the propensity score: stratification or modeling?

In observational studies of treatments or interventions, propensity score (PS) adjustment is often useful for controlling bias in estimation of treatment effects. Regression on PS is used most often and can be highly efficient, but it can lead to biased results when model assumptions are violated. The validity of stratification on PS depends on fewer model assumptions, but this approach is less efficient than regression adjustment when the regression assumptions hold. To investigate these issues, we compare stratification and regression adjustments in a Monte Carlo simulation study. We consider two stratification approaches: equal frequency strata and an approach that attempts to choose strata that minimize the mean squared error (MSE) of the treatment effect estimate. The regression approach that we consider is a generalized additive model (GAM) that estimates treatment effect controlling for a potentially nonlinear association between PS and outcome. We find that under a wide range of plausible data generating distributions the GAM approach outperforms stratification in treatment effect estimation with respect to bias, variance, and thereby MSE. We illustrate each approach in an analysis of insurance plan choice and its relation to satisfaction with asthma care.     

Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study

Propensity score methods are increasingly being used to estimate causal treatment effects in the medical literature. Conditioning on the propensity score results in unbiased estimation of the expected difference in observed responses to two treatments. The degree to which conditioning on the propensity score introduces bias into the estimation of the conditional odds ratio or conditional hazard ratio, which are frequently used as measures of treatment effect in observational studies, has not been extensively studied. We conducted Monte Carlo simulations to determine the degree to which propensity score matching, stratification on the quintiles of the propensity score, and covariate adjustment using the propensity score result in biased estimation of conditional odds ratios, hazard ratios, and rate ratios. We found that conditioning on the propensity score resulted in biased estimation of the true conditional odds ratio and the true conditional hazard ratio. In all scenarios examined, treatment effects were biased towards the null treatment effect. However, conditioning on the propensity score did not result in biased estimation of the true conditional rate ratio. In contrast, conventional regression methods allowed unbiased estimation of the true conditional treatment effect when all variables associated with the outcome were included in the regression model. The observed bias in propensity score methods is due to the fact that regression models allow one to estimate conditional treatment effects, whereas propensity score methods allow one to estimate marginal treatment effects. In several settings with non-linear treatment effects, marginal and conditional treatment effects do not coincide.     

Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group

In observational studies, investigators have no control over the treatment assignment. The treated and non-treated (that is, control) groups may have large differences on their observed covariates, and these differences can lead to biased estimates of treatment effects. Even traditional covariance analysis adjustments may be inadequate to eliminate this bias. The propensity score, defined as the conditional probability of being treated given the covariates, can be used to balance the covariates in the two groups, and therefore reduce this bias. In order to estimate the propensity score, one must model the distribution of the treatment indicator variable given the observed covariates. Once estimated the propensity score can be used to reduce bias through matching, stratification (subclassification), regression adjustment, or some combination of all three. In this tutorial we discuss the uses of propensity score methods for bias reduction, give references to the literature and illustrate the uses through applied examples. © 1998 John Wiley & Sons, Ltd.     

A comparison of propensity score methods: a case-study estimating the effectiveness of post-AMI statin use

There is an increasing interest in the use of propensity score methods to estimate causal effects in observational studies. However, recent systematic reviews have demonstrated that propensity score methods are inconsistently used and frequently poorly applied in the medical literature. In this study, we compared the following propensity score methods for estimating the reduction in all-cause mortality due to statin therapy for patients hospitalized with acute myocardial infarction: propensity-score matching, stratification using the propensity score, covariate adjustment using the propensity score, and weighting using the propensity score. We used propensity score methods to estimate both adjusted treated effects and the absolute and relative risk reduction in all-cause mortality. We also examined the use of statistical hypothesis testing, standardized differences, box plots, non-parametric density estimates, and quantile-quantile plots to assess residual confounding that remained after stratification or matching on the propensity score. Estimates of the absolute reduction in 3-year mortality ranged from 2.1 to 4.5 per cent, while estimates of the relative risk reduction ranged from 13.3 to 17.0 per cent. Adjusted estimates of the reduction in the odds of 3-year death varied from 15 to 24 per cent across the different propensity score methods.     

Bias associated with using the estimated propensity score as a regression covariate

The use of propensity score methods to adjust for selection bias in observational studies has become increasingly popular in public health and medical research. A substantial portion of studies using propensity score adjustment treat the propensity score as a conventional regression predictor. Through a Monte Carlo simulation study, Austin and colleagues. investigated the bias associated with treatment effect estimation when the propensity score is used as a covariate in nonlinear regression models, such as logistic regression and Cox proportional hazards models. We show that the bias exists even in a linear regression model when the estimated propensity score is used and derive the explicit form of the bias. We also conduct an extensive simulation study to compare the performance of such covariate adjustment with propensity score stratification, propensity score matching, inverse probability of treatment weighted method, and nonparametric functional estimation using splines. The simulation scenarios are designed to reflect real data analysis practice. Instead of specifying a known parametric propensity score model, we generate the data by considering various degrees of overlap of the covariate distributions between treated and control groups. Propensity score matching excels when the treated group is contained within a larger control pool, while the model‐based adjustment may have an edge when treated and control groups do not have too much overlap. Overall, adjusting for the propensity score through stratification or matching followed by regression or using splines, appears to be a good practical strategy. Copyright © 2013 John Wiley & Sons, Ltd.     

The performance of different propensity-score methods for estimating relative risks

OBJECTIVES: The propensity score is the probability of treatment conditional on observed variables. Conditioning on the propensity-score results in unbiased estimation of the expected difference in observed responses to two treatments. The performance of propensity-score methods for estimating relative risks has not been studied. STUDY DESIGN AND SETTING: Monte Carlo simulations were used to assess the performance of matching, stratification, and covariate adjustment using the propensity score to estimate relative risks. RESULTS: Matching on the propensity score and stratification on the quintiles of the propensity score resulted in estimates of relative risk with similar mean squared error (MSE). Propensity-score matching resulted in estimates with less bias, whereas stratification on the propensity score resulted in estimates of with greater precision. Including only variables associated with the outcome or including only the true confounders in the propensity-score model resulted in estimates with lower MSE than did including all variables associated with treatment or all measured variables in the propensity-score model. CONCLUSIONS: When estimating relative risks, propensity-score matching resulted in estimates with less bias than did stratification on the quintiles of the propensity score, but stratification on the quintiles of the propensity score resulted in estimates with greater precision.     

Covariate balance in a Bayesian propensity score analysis of beta blocker therapy in heart failure patients

Regression adjustment for the propensity score is a statistical method that reduces confounding from measured variables in observational data. A Bayesian propensity score analysis extends this idea by using simultaneous estimation of the propensity scores and the treatment effect. In this article, we conduct an empirical investigation of the performance of Bayesian propensity scores in the context of an observational study of the effectiveness of beta-blocker therapy in heart failure patients. We study the balancing properties of the estimated propensity scores. Traditional Frequentist propensity scores focus attention on balancing covariates that are strongly associated with treatment. In contrast, we demonstrate that Bayesian propensity scores can be used to balance the association between covariates and the outcome. This balancing property has the effect of reducing confounding bias because it reduces the degree to which covariates are outcome risk factors.     

Comparing Standard Regression, Propensity Score Matching, and Instrumental Variables Methods for Determining the Influence of Mammography on Stage of Diagnosis

In situations where randomized trials are not feasible, analysis of observational data must be used instead. However, when using observational data, there is often selection bias for which we must account in order to adjust for pre-treatment differences between groups in their baseline characteristics. As an example of this, we used the Linked Medicare-Tumor Registry Database created by the National Cancer Institute and the Centers for Medicare and Medicaid Services to look at screening with mammography in older women to determine its effectiveness in detecting cancer at an earlier stage. The standard regression method and two methods of adjusting for selection bias are compared. We start with the standard analysis, a logistic regression predicting stage at diagnosis that includes as independent variables a set of covariates to adjust for differences in baseline risk plus an indicator variable for whether the woman used screening. Next, we employ propensity score matching, which evens out the distribution of measured baseline characteristics across groups, and is more robust to model mis-specification than the standard analysis. Lastly, we conduct an instrumental variable analysis, which addresses unmeasured differences between the users and non-users. This article compares these methods and discusses issues of which researchers and analysts should be aware. It is important to look beyond the standard analysis and to consider propensity score matching when there is concern about group differences in measured covariates and instrumental variable analysis when there is concern about differences in unmeasured covariates.     

The performance of different propensity score methods for estimating marginal hazard ratios

Propensity score methods are increasingly being used to reduce or minimize the effects of confounding when estimating the effects of treatments, exposures, or interventions when using observational or non‐randomized data. Under the assumption of no unmeasured confounders, previous research has shown that propensity score methods allow for unbiased estimation of linear treatment effects (e.g., differences in means or proportions). However, in biomedical research, time‐to‐event outcomes occur frequently. There is a paucity of research into the performance of different propensity score methods for estimating the effect of treatment on time‐to‐event outcomes. Furthermore, propensity score methods allow for the estimation of marginal or population‐average treatment effects. We conducted an extensive series of Monte Carlo simulations to examine the performance of propensity score matching (1:1 greedy nearest‐neighbor matching within propensity score calipers), stratification on the propensity score, inverse probability of treatment weighting (IPTW) using the propensity score, and covariate adjustment using the propensity score to estimate marginal hazard ratios. We found that both propensity score matching and IPTW using the propensity score allow for the estimation of marginal hazard ratios with minimal bias. Of these two approaches, IPTW using the propensity score resulted in estimates with lower mean squared error when estimating the effect of treatment in the treated. Stratification on the propensity score and covariate adjustment using the propensity score result in biased estimation of both marginal and conditional hazard ratios. Applied researchers are encouraged to use propensity score matching and IPTW using the propensity score when estimating the relative effect of treatment on time‐to‐event outcomes. Copyright © 2012 John Wiley & Sons, Ltd.     

Covariance adjustment on propensity parameters for continuous treatment in linear models

Propensity scores are widely used to control for confounding when estimating the effect of a binary treatment in observational studies. They have been generalized to ordinal and continuous treatments in the recent literature. Following the definition of propensity function and its parameterizations (called the propensity parameter in this paper) proposed by Imai and van Dyk, we explore sufficient conditions for selecting propensity parameters to control for confounding for continuous treatments in the context of regression‐based adjustment in linear models. Typically, investigators make parametric assumptions about the form of the dose–response function for a continuous treatment. Such assumptions often allow the analyst to use only a subset of the propensity parameters to control confounding. When the treatment is the only predictor in the structural, that is, causal model, it is sufficient to adjust only for the propensity parameters that characterize the expectation of the treatment variable or its functional form. When the structural model includes selected baseline covariates other than the treatment variable, those baseline covariates, in addition to the propensity parameters, must also be adjusted in the model. We demonstrate these points with an example estimating the dose–response relationship for the effect of erythropoietin on hematocrit level in patients with end‐stage renal disease. Copyright © 2014 John Wiley & Sons, Ltd.     

The performance of different propensity‐score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies

Propensity score methods are increasingly being used to estimate the effects of treatments on health outcomes using observational data. There are four methods for using the propensity score to estimate treatment effects: covariate adjustment using the propensity score, stratification on the propensity score, propensity‐score matching, and inverse probability of treatment weighting (IPTW) using the propensity score. When outcomes are binary, the effect of treatment on the outcome can be described using odds ratios, relative risks, risk differences, or the number needed to treat. Several clinical commentators suggested that risk differences and numbers needed to treat are more meaningful for clinical decision making than are odds ratios or relative risks. However, there is a paucity of information about the relative performance of the different propensity‐score methods for estimating risk differences. We conducted a series of Monte Carlo simulations to examine this issue. We examined bias, variance estimation, coverage of confidence intervals, mean‐squared error (MSE), and type I error rates. A doubly robust version of IPTW had superior performance compared with the other propensity‐score methods. It resulted in unbiased estimation of risk differences, treatment effects with the lowest standard errors, confidence intervals with the correct coverage rates, and correct type I error rates. Stratification, matching on the propensity score, and covariate adjustment using the propensity score resulted in minor to modest bias in estimating risk differences. Estimators based on IPTW had lower MSE compared with other propensity‐score methods. Differences between IPTW and propensity‐score matching may reflect that these two methods estimate the average treatment effect and the average treatment effect for the treated, respectively. Copyright © 2010 John Wiley & Sons, Ltd.     

Methods to assess intended effects of drug treatment in observational studies are reviewed

BACKGROUND AND OBJECTIVE: To review methods that seek to adjust for confounding in observational studies when assessing intended drug effects. METHODS: We reviewed the statistical, economical and medical literature on the development, comparison and use of methods adjusting for confounding. RESULTS: In addition to standard statistical techniques of (logistic) regression and Cox proportional hazards regression, alternative methods have been proposed to adjust for confounding in observational studies. A first group of methods focus on the main problem of nonrandomization by balancing treatment groups on observed covariates: selection, matching, stratification, multivariate confounder score, and propensity score methods, of which the latter can be combined with stratification or various matching methods. Another group of methods look for variables to be used like randomization in order to adjust also for unobserved covariates: instrumental variable methods, two-stage least squares, and grouped-treatment approach. Identifying these variables is difficult, however, and assumptions are strong. Sensitivity analyses are useful tools in assessing the robustness and plausibility of the estimated treatment effects to variations in assumptions about unmeasured confounders. CONCLUSION: In most studies regression-like techniques are routinely used for adjustment for confounding, although alternative methods are available. More complete empirical evaluations comparing these methods in different situations are needed.     

New strategies are needed to improve the accuracy of influenza vaccine effectiveness estimates among seniors

ObjectiveThe magnitude of the benefit of influenza vaccine among elderly individuals has been recently debated. Existing vaccine effectiveness estimates derive primarily from observational studies, which may be biased. In this paper, we provide a methodological examination of the potential sources of bias in observational studies of influenza vaccine effectiveness in seniors and propose design and analysis strategies to reduce bias in future studies.Study Design and SettingWe draw parallels to bias documented in observational studies of therapies in other areas of medical research including pharmacoepidemiology, discuss reasons why existing adjustment methods in influenza studies may not adequately control for the bias, and evaluate statistical approaches that may yield more accurate estimation of influenza vaccine effectiveness.ResultsThere is strong evidence for the presence of bias in existing observational estimates of influenza vaccine effectiveness in the elderly and the failure of current adjustment methods to reduce bias.ConclusionPromising approaches for reducing bias include obtaining more accurate information on confounders, such as functional status, avoiding all-cause death in favor of outcomes, such as pneumonia or influenza-related pneumonia, and evaluating the extent to which bias is reduced by these and other methods using the ‘control’ period before influenza season.     

Mitigating bias in observational vaccine effectiveness studies using simulated comparator populations: Application to rotavirus vaccination in the UK

BackgroundMeasuring vaccine effectiveness (VE) relies on the use of observational study designs. However, achieving robust estimates of direct and indirect VE is frequently compromised by bias, particularly when using syndromic diagnoses of low-specificity.MethodsIn order to mitigate confounding between the measured outcome and vaccine uptake, we developed a method to balance comparator populations using individual-level propensity scoring derived from the vaccine-exposed population, and applied it to the unexposed comparator population. Indirect VE was estimated by comparing the unvaccinated vaccine-exposed group with a propensity score-simulated unvaccinated, unexposed group. Direct VE was derived by removing indirect VE from the overall VE.We applied this method to an evaluation of the effectiveness of infant rotavirus vaccination in the UK. Using a general practice cohort of 45,259 live births between May 2010 and December 2015, we calculated indirect and direct VE against consultations for acute gastroenteritis using conventional and vaccination-propensity adjustment comparator populations.ResultsThe overall VE during the rotavirus-season (January-May) calculated using mixed-effects Cox regression was 30% [95% confidence intervals (95% CI: 25,35%)]. Use of conventional comparator populations resulted in implausible VE estimates −14% (95% CI: −41,7%) for direct and 29% (95% CI: 14,42%) for indirect effects. Applying our alternative method, direct VE was 26% (95% CI: 1,45%) and indirect VE was 8% (95% CI: −19,29%).ConclusionsEstimating VE using propensity score simulated comparator populations, particularly for studies using routine health data with syndromic, low-specificity endpoints will aid accurate measurement of the broader public health impact of a vaccine programme.