The performance of different propensity score methods for estimating marginal hazard ratios

Propensity score methods are increasingly being used to reduce or minimize the effects of confounding when estimating the effects of treatments, exposures, or interventions when using observational or non‐randomized data. Under the assumption of no unmeasured confounders, previous research has shown that propensity score methods allow for unbiased estimation of linear treatment effects (e.g., differences in means or proportions). However, in biomedical research, time‐to‐event outcomes occur frequently. There is a paucity of research into the performance of different propensity score methods for estimating the effect of treatment on time‐to‐event outcomes. Furthermore, propensity score methods allow for the estimation of marginal or population‐average treatment effects. We conducted an extensive series of Monte Carlo simulations to examine the performance of propensity score matching (1:1 greedy nearest‐neighbor matching within propensity score calipers), stratification on the propensity score, inverse probability of treatment weighting (IPTW) using the propensity score, and covariate adjustment using the propensity score to estimate marginal hazard ratios. We found that both propensity score matching and IPTW using the propensity score allow for the estimation of marginal hazard ratios with minimal bias. Of these two approaches, IPTW using the propensity score resulted in estimates with lower mean squared error when estimating the effect of treatment in the treated. Stratification on the propensity score and covariate adjustment using the propensity score result in biased estimation of both marginal and conditional hazard ratios. Applied researchers are encouraged to use propensity score matching and IPTW using the propensity score when estimating the relative effect of treatment on time‐to‐event outcomes. Copyright © 2012 John Wiley & Sons, Ltd.     

Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies

The propensity score is defined as a subject's probability of treatment selection, conditional on observed baseline covariates. Weighting subjects by the inverse probability of treatment received creates a synthetic sample in which treatment assignment is independent of measured baseline covariates. Inverse probability of treatment weighting (IPTW) using the propensity score allows one to obtain unbiased estimates of average treatment effects. However, these estimates are only valid if there are no residual systematic differences in observed baseline characteristics between treated and control subjects in the sample weighted by the estimated inverse probability of treatment. We report on a systematic literature review, in which we found that the use of IPTW has increased rapidly in recent years, but that in the most recent year, a majority of studies did not formally examine whether weighting balanced measured covariates between treatment groups. We then proceed to describe a suite of quantitative and qualitative methods that allow one to assess whether measured baseline covariates are balanced between treatment groups in the weighted sample. The quantitative methods use the weighted standardized difference to compare means, prevalences, higher‐order moments, and interactions. The qualitative methods employ graphical methods to compare the distribution of continuous baseline covariates between treated and control subjects in the weighted sample. Finally, we illustrate the application of these methods in an empirical case study. We propose a formal set of balance diagnostics that contribute towards an evolving concept of ‘best practice’ when using IPTW to estimate causal treatment effects using observational data. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

The use of propensity score methods with survival or time‐to‐event outcomes: reporting measures of effect similar to those used in randomized experiments

Propensity score methods are increasingly being used to estimate causal treatment effects in observational studies. In medical and epidemiological studies, outcomes are frequently time‐to‐event in nature. Propensity‐score methods are often applied incorrectly when estimating the effect of treatment on time‐to‐event outcomes. This article describes how two different propensity score methods (matching and inverse probability of treatment weighting) can be used to estimate the measures of effect that are frequently reported in randomized controlled trials: (i) marginal survival curves, which describe survival in the population if all subjects were treated or if all subjects were untreated; and (ii) marginal hazard ratios. The use of these propensity score methods allows one to replicate the measures of effect that are commonly reported in randomized controlled trials with time‐to‐event outcomes: both absolute and relative reductions in the probability of an event occurring can be determined. We also provide guidance on variable selection for the propensity score model, highlight methods for assessing the balance of baseline covariates between treated and untreated subjects, and describe the implementation of a sensitivity analysis to assess the effect of unmeasured confounding variables on the estimated treatment effect when outcomes are time‐to‐event in nature. The methods in the paper are illustrated by estimating the effect of discharge statin prescribing on the risk of death in a sample of patients hospitalized with acute myocardial infarction. In this tutorial article, we describe and illustrate all the steps necessary to conduct a comprehensive analysis of the effect of treatment on time‐to‐event outcomes. © 2013 The authors. Statistics in Medicine published by John Wiley & Sons, Ltd.     

Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity‐score matched samples

The propensity score is a subject's probability of treatment, conditional on observed baseline covariates. Conditional on the true propensity score, treated and untreated subjects have similar distributions of observed baseline covariates. Propensity‐score matching is a popular method of using the propensity score in the medical literature. Using this approach, matched sets of treated and untreated subjects with similar values of the propensity score are formed. Inferences about treatment effect made using propensity‐score matching are valid only if, in the matched sample, treated and untreated subjects have similar distributions of measured baseline covariates. In this paper we discuss the following methods for assessing whether the propensity score model has been correctly specified: comparing means and prevalences of baseline characteristics using standardized differences; ratios comparing the variance of continuous covariates between treated and untreated subjects; comparison of higher order moments and interactions; five‐number summaries; and graphical methods such as quantile–quantile plots, side‐by‐side boxplots, and non‐parametric density plots for comparing the distribution of baseline covariates between treatment groups. We describe methods to determine the sampling distribution of the standardized difference when the true standardized difference is equal to zero, thereby allowing one to determine the range of standardized differences that are plausible with the propensity score model having been correctly specified. We highlight the limitations of some previously used methods for assessing the adequacy of the specification of the propensity‐score model. In particular, methods based on comparing the distribution of the estimated propensity score between treated and untreated subjects are uninformative. Copyright © 2009 John Wiley & Sons, Ltd.     

The performance of different propensity score methods for estimating marginal odds ratios

The propensity score which is the probability of exposure to a specific treatment conditional on observed variables. Conditioning on the propensity score results in unbiased estimation of the expected difference in observed responses to two treatments. In the medical literature, propensity score methods are frequently used for estimating odds ratios. The performance of propensity score methods for estimating marginal odds ratios has not been studied. We performed a series of Monte Carlo simulations to assess the performance of propensity score matching, stratifying on the propensity score, and covariate adjustment using the propensity score to estimate marginal odds ratios. We assessed bias, precision, and mean-squared error (MSE) of the propensity score estimators, in addition to the proportion of bias eliminated due to conditioning on the propensity score. When the true marginal odds ratio was one, then matching on the propensity score and covariate adjustment using the propensity score resulted in unbiased estimation of the true treatment effect, whereas stratification on the propensity score resulted in minor bias in estimating the true marginal odds ratio. When the true marginal odds ratio ranged from 2 to 10, then matching on the propensity score resulted in the least bias, with a relative biases ranging from 2.3 to 13.3 per cent. Stratifying on the propensity score resulted in moderate bias, with relative biases ranging from 15.8 to 59.2 per cent. For both methods, relative bias was proportional to the true odds ratio. Finally, matching on the propensity score tended to result in estimators with the lowest MSE.     

基于倾向指数匹配法的肝癌患者生存分析

  目的  基于倾向性评分逆概率加权法（IPTW）评价手术、放疗和联合治疗3种方式治疗胃癌患者的疗效, 为胃癌的治疗提供参考依据。  方法  收集2004年1月 — 2013年12月美国国家癌症研究所监测、流行病学与预后项目（SEER）数据库中经胃镜病理诊断确诊的7 005例胃癌患者数据,其中接受手术治疗者3 983例、接受放疗者795例、接受联合治疗者2 227例;采用倾向性评分IPTW法以生存时间和结局为效应指标,分析不同的治疗方法对胃癌患者生存率的影响。  结果  手术组、放疗组和联合治疗组胃癌患者倾向性评分IPTW法加权前中位生存期分别为30、9和38个月,加权后分别为25、11和38个月,加权前、后3组胃癌患者生存曲线间差异均有统计学意义（均P < 0.001）,联合治疗组患者加权前、后的预后均优于手术组和放疗组患者;加权后大部分基线特征绝对标准化平均差异（ASMD）减小,且均 < 0.2,3组患者均衡效果较好;在控制了性别、年龄、种族、婚姻状况、肿瘤大小、病理分化、肿瘤分期、T分期、N分期和远处转移情况等混杂因素后,多因素Cox回归分析结果显示,与手术组胃癌患者相比,放疗组胃癌患者的预后较差（HR = 2.044,95 % CI = 1.770～2.361）,联合治疗组胃癌患者的预后较好（HR = 0.630,95 % CI = 0.573～0.694）。  结论  联合治疗方法疗效优于手术和放疗治疗方法。     

Variance estimation when using inverse probability of treatment weighting (IPTW) with survival analysis

Propensity score methods are used to reduce the effects of observed confounding when using observational data to estimate the effects of treatments or exposures. A popular method of using the propensity score is inverse probability of treatment weighting (IPTW). When using this method, a weight is calculated for each subject that is equal to the inverse of the probability of receiving the treatment that was actually received. These weights are then incorporated into the analyses to minimize the effects of observed confounding. Previous research has found that these methods result in unbiased estimation when estimating the effect of treatment on survival outcomes. However, conventional methods of variance estimation were shown to result in biased estimates of standard error. In this study, we conducted an extensive set of Monte Carlo simulations to examine different methods of variance estimation when using a weighted Cox proportional hazards model to estimate the effect of treatment. We considered three variance estimation methods: (i) a naïve model‐based variance estimator; (ii) a robust sandwich‐type variance estimator; and (iii) a bootstrap variance estimator. We considered estimation of both the average treatment effect and the average treatment effect in the treated. We found that the use of a bootstrap estimator resulted in approximately correct estimates of standard errors and confidence intervals with the correct coverage rates. The other estimators resulted in biased estimates of standard errors and confidence intervals with incorrect coverage rates. Our simulations were informed by a case study examining the effect of statin prescribing on mortality. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.     

Empirical use of causal inference methods to evaluate survival differences in a real-world registry vs those found in randomized clinical trials

With heighted interest in causal inference based on real-world evidence, this empirical study sought to understand differences between the results of observational analyses and long-term randomized clinical trials. We hypothesized that patients deemed “eligible” for clinical trials would follow a different survival trajectory from those deemed “ineligible” and that this factor could partially explain results. In a large observational registry dataset, we estimated separate survival trajectories for hypothetically trial-eligible vs ineligible patients under both coronary artery bypass surgery (CABG) and percutaneous coronary intervention (PCI). We also explored whether results would depend on the causal inference method (inverse probability of treatment weighting vs optimal full propensity matching) or the approach to combine propensity scores from multiple imputations (the “across” vs “within” approaches). We found that, in this registry population of PCI/CABG multivessel patients, 32.5% would have been eligible for contemporaneous RCTs, suggesting that RCTs enroll selected populations. Additionally, we found treatment selection bias with different distributions of propensity scores between PCI and CABG patients. The different methodological approaches did not result in different conclusions. Overall, trial-eligible patients appeared to demonstrate at least marginally better survival than ineligible patients. Treatment comparisons by eligibility depended on disease severity. Among trial-eligible three-vessel diseased and trial-ineligible two-vessel diseased patients, CABG appeared to have at least a slight advantage with no treatment difference otherwise. In conclusion, our analyses suggest that RCTs enroll highly selected populations, and our findings are generally consistent with RCTs but less pronounced than major registry findings.     

Comparing paired vs non-paired statistical methods of analyses when making inferences about absolute risk reductions in propensity-score matched samples

Propensity-score matching allows one to reduce the effects of treatment-selection bias or confounding when estimating the effects of treatments when using observational data. Some authors have suggested that methods of inference appropriate for independent samples can be used for assessing the statistical significance of treatment effects when using propensity-score matching. Indeed, many authors in the applied medical literature use methods for independent samples when making inferences about treatment effects using propensity-score matched samples. Dichotomous outcomes are common in healthcare research. In this study, we used Monte Carlo simulations to examine the effect on inferences about risk differences (or absolute risk reductions) when statistical methods for independent samples are used compared with when statistical methods for paired samples are used in propensity-score matched samples. We found that compared with using methods for independent samples, the use of methods for paired samples resulted in: (i) empirical type I error rates that were closer to the advertised rate; (ii) empirical coverage rates of 95 per cent confidence intervals that were closer to the advertised rate; (iii) narrower 95 per cent confidence intervals; and (iv) estimated standard errors that more closely reflected the sampling variability of the estimated risk difference. Differences between the empirical and advertised performance of methods for independent samples were greater when the treatment-selection process was stronger compared with when treatment-selection process was weaker. We recommend using statistical methods for paired samples when using propensity-score matched samples for making inferences on the effect of treatment on the reduction in the probability of an event occurring.     

The effect of a constraint on the maximum number of controls matched to each treated subject on the performance of full matching on the propensity score when estimating risk differences

Many observational studies estimate causal effects using methods based on matching on the propensity score. Full matching on the propensity score is an effective and flexible method for utilizing all available data and for creating well‐balanced treatment and control groups. An important component of the full matching algorithm is the decision about whether to impose a restriction on the maximum ratio of controls matched to each treated subject. Despite the possible effect of this restriction on subsequent inferences, this issue has not been examined. We used a series of Monte Carlo simulations to evaluate the effect of imposing a restriction on the maximum ratio of controls matched to each treated subject when estimating risk differences. We considered full matching both with and without a caliper restriction. When using full matching with a caliper restriction, the imposition of a subsequent constraint on the maximum ratio of the number of controls matched to each treated subject had no effect on the quality of inferences. However, when using full matching without a caliper restriction, the imposition of a constraint on the maximum ratio of the number of controls matched to each treated subject tended to result in an increase in bias in the estimated risk difference. However, this increase in bias tended to be accompanied by a corresponding decrease in the sampling variability of the estimated risk difference. We illustrate the consequences of these restrictions using observational data to estimate the effect of medication prescribing on survival following hospitalization for a heart attack.     

Estimating long‐term treatment effects in observational data: A comparison of the performance of different methods under real‐world uncertainty

In the presence of time‐dependent confounding, there are several methods available to estimate treatment effects. With correctly specified models and appropriate structural assumptions, any of these methods could provide consistent effect estimates, but with real‐world data, all models will be misspecified and it is difficult to know if assumptions are violated. In this paper, we investigate five methods: inverse probability weighting of marginal structural models, history‐adjusted marginal structural models, sequential conditional mean models, g‐computation formula, and g‐estimation of structural nested models. This work is motivated by an investigation of the effects of treatments in cystic fibrosis using the UK Cystic Fibrosis Registry data focussing on two outcomes: lung function (continuous outcome) and annual number of days receiving intravenous antibiotics (count outcome). We identified five features of this data that may affect the performance of the methods: misspecification of the causal null, long‐term treatment effects, effect modification by time‐varying covariates, misspecification of the direction of causal pathways, and censoring. In simulation studies, under ideal settings, all five methods provide consistent estimates of the treatment effect with little difference between methods. However, all methods performed poorly under some settings, highlighting the importance of using appropriate methods based on the data available. Furthermore, with the count outcome, the issue of non‐collapsibility makes comparison between methods delivering marginal and conditional effects difficult. In many situations, we would recommend using more than one of the available methods for analysis, as if the effect estimates are very different, this would indicate potential issues with the analyses.     

The added value of propensity score matching when using health‐related quality of life reference data

Direct comparisons of health‐related quality of life (HRQoL) outcomes between non‐randomized groups might be biased, as outcomes are confounded by imbalance in pre‐treatment patient characteristics. Such bias can be reduced by adjusting on observed covariates. This is the setting of HRQoL comparisons with reference data, where age and gender adjustment is commonly used for this purpose. However, other observed covariates can be used to lessen this bias and yield more precise estimates. The objective of this study is to show that more accurate HRQoL comparisons with reference data can be obtained, accounting for few covariates in addition to age and gender by a propensity score matching approach. Copyright © 2013 John Wiley & Sons, Ltd.     

A flexible parametric approach for estimating continuous‐time inverse probability of treatment and censoring weights

Marginal structural Cox models are used for quantifying marginal treatment effects on outcome event hazard function. Such models are estimated using inverse probability of treatment and censoring (IPTC) weighting, which properly accounts for the impact of time‐dependent confounders, avoiding conditioning on factors on the causal pathway. To estimate the IPTC weights, the treatment assignment mechanism is conventionally modeled in discrete time. While this is natural in situations where treatment information is recorded at scheduled follow‐up visits, in other contexts, the events specifying the treatment history can be modeled in continuous time using the tools of event history analysis. This is particularly the case for treatment procedures, such as surgeries. In this paper, we propose a novel approach for flexible parametric estimation of continuous‐time IPTC weights and illustrate it in assessing the relationship between metastasectomy and mortality in metastatic renal cell carcinoma patients. Copyright © 2016 John Wiley & Sons, Ltd.     

Likelihood ratio and score tests to test the non‐inferiority (or equivalence) of the odds ratio in a crossover study with binary outcomes

We consider the non‐inferiority (or equivalence) test of the odds ratio (OR) in a crossover study with binary outcomes to evaluate the treatment effects of two drugs. To solve this problem, Lui and Chang (2011) proposed both an asymptotic method and a conditional method based on a random effects logit model. Kenward and Jones (1987) proposed a likelihood ratio test (LRT_M) based on a log linear model. These existing methods are all subject to model misspecification. In this paper, we propose a likelihood ratio test (LRT) and a score test that are independent of model specification. Monte Carlo simulation studies show that, in scenarios considered in this paper, both the LRT and the score test have higher power than the asymptotic and conditional methods for the non‐inferiority test; the LRT, score, and asymptotic methods have similar power, and they all have higher power than the conditional method for the equivalence test. When data can be well described by a log linear model, the LRT_M has the highest power among all the five methods (LRT_M, LRT, score, asymptotic, and conditional) for both non‐inferiority and equivalence tests. However, in scenarios for which a log linear model does not describe the data well, the LRT_M has the lowest power for the non‐inferiority test and has inflated type I error rates for the equivalence test. We provide an example from a clinical trial that illustrates our methods. Copyright © 2016 John Wiley & Sons, Ltd.     

Reduced risk of serious pneumococcal infections up to 10 years after a dose of pneumococcal conjugate vaccine in established arthritis

BackgroundTo examine rates of serious pneumococcal infections up to 10 years after vaccination with 7-valent conjugated pneumococcal vaccine (PCV7) in patients with arthritis compared to non-vaccinated arthritis patients.MethodsIn total, 595 adult arthritis patients (rheumatoid arthritis; RA = 342, 80 % women and spondylarthropathy; SpA = 253, 45 % women) received one dose of PCV7. Mean age/disease duration were 62/16 and 51/14 years, respectively. For each patient, 4 matched reference subjects were identified.At vaccination, 420 patients received bDMARDs (anti-TNF = 330, tocilizumab = 15, abatacept = 18, anakinra = 1, rituximab = 56). Methotrexate was given as monotherapy (n = 86) or in combination with bDMARD (n = 220). 89 SpA patients received NSAIDs without DMARD.The Skåne Healthcare Register was searched for ICD-10 diagnostic codes for pneumococcal infections (pneumonia, lower respiratory tract infection, septicemia, meningitis, septic arthritis) between January 2000 and December 2018. Frequency of infections after vs before vaccination were calculated (relative risks). Relative risk ratio (RRR) and relative risk reduction (1-RRR) were calculated comparing patients vs non-vaccinated references. Kaplan-Meier and Cox regression were used to investigate time to first event and predictors of infections.ResultsAmong vaccinated RA and SpA patients, there was a significant relative risk reduction of pneumonia and all serious infections; 53% and 46%, respectively. There was no significant difference in time to first pneumonia or all serious infections after vaccination between patients and references. Higher age, RA diagnosis and concomitant prednisolone were associated with infections.ConclusionOne dose of pneumococcal conjugate vaccine may decrease risk of serious pneumococcal infection up to 10 years in patients with arthritis receiving immunomodulating treatment.