An intuitive review of methods for observational studies of comparative effectiveness

I use diagrams to illustrate the sources of potential selection bias in observational studies of comparative effectiveness. I adapt these diagrams for three hypothetical scenarios that clarify the strengths and weaknesses of two prominent methods used to account for potential selection bias: propensity scores and instrumental variables. After reviewing the fundamentals of how to apply each method, including new developments that make implementation easier, I refer to some recent studies that illustrate how choice of method can affect estimates. I conclude by emphasizing that many studies with apparently rich sources of data are nevertheless unlikely to produce unbiased estimates and that conceptual modeling can help identify these problems in advance.     

Marginal structural models might overcome confounding when analyzing multiple treatment effects in observational studies

OBJECTIVE: We review marginal structural models (MSMs) and show how they are useful when comparing the effects of multiple treatments on outcomes in observational studies. Until now, MSMs have not been used to compare the effects of more than two treatments. STUDY DESIGN AND SETTING: To illustrate the application of MSMs when patients may receive several treatments, we have reanalyzed the effects of antipsychotic medication on achieving remission in schizophrenia using data from the SOHO study, a 3-year observational study of health outcomes associated with the treatment of schizophrenia. RESULTS: The MSM results were, in general, consistent with but less statistically significant than those obtained using conventional methods. The MSM also showed qualitative differences in some comparisons in which the conventional analysis obtained results that were not consistent with previous knowledge. CONCLUSION: MSMs can be used to analyze multiple treatment effects. MSMs, by using inverse-probability of treatment weights, might provide a better control for confounding than conventional methods by improving the adjustment for treatment group differences in observational studies, which may approximate their results to those of randomized controlled trials.     

A cautionary note concerning the use of stabilized weights in marginal structural models

Marginal structural models are commonly used to estimate the causal effect of a time‐varying treatment in presence of time‐dependent confounding. When fitting an MSM to data, the analyst must specify both the structural model for the outcome and the treatment models for the inverse‐probability‐of‐treatment weights. The use of stabilized weights is recommended because they are generally less variable than the standard weights. In this paper, we are concerned with the use of the common stabilized weights when the structural model is specified to only consider partial treatment history, such as the current or most recent treatments. We present various examples of settings where these stabilized weights yield biased inferences while the standard weights do not. These issues are first investigated on the basis of simulated data and subsequently exemplified using data from the Honolulu Heart Program. Unlike common stabilized weights, we find that basic stabilized weights offer some protection against bias in structural models designed to estimate current or most recent treatment effects. Copyright © 2014 John Wiley & Sons, Ltd.     

群组对照试验中应用倾向得分匹配法分析数据对干预效果评价的影响

目的 探讨应用倾向评分匹配法（propensity score matching,PSM）处理群组平行对照试验中横断面调查数据对干预效果评估的影响。方法 以某"十二五"国家科技重大专项的子课题"低档暗娼减少性病艾滋病感染干预研究"数据为例,应用PSM法对干预前后两次横断面调查的低档暗娼人群数据进行匹配,对匹配后样本开展结局变量χ2检验并拟合广义线性混合模型（generalized linear mixed models,GLMM）,讨论PSM法对评估结果的影响。结果 以存在显著差别的关键特征变量作为匹配因素进行PSM后,样本量为537,干预前后的两个人群完全可比。GLMM分析结果表明,干预是梅毒感染率降低的主要因素。PSM后数据拟合模型获得的OR值为0.33,与原始数据得到的OR值（0.51）相比降低了0.18,而且前者获得的95%CI（0.16~0.70）比后者（0.27~0.96）更窄,更远离1,将干预措施降低梅毒感染风险的效果从49%提高到了67%。结论 PSM法用于系列横断面调查的群组对照干预试验,可以有效提高不同调查人群之间的可比性,降低人群差异对效果评估的影响,从而提高研究结果的准确性。     

倾向评分配比在流行病学设计中的应用

介绍倾向评分配比法(PSM)的基本原理、具体方法,并结合实例探讨其在流行病学设计过程中的应用.PSM通过某些观察性研究某些混杂变量与研究因素的关系计算倾向评分,然后从对照组中为处理组每个个体寻找一个或多个倾向评分值相同或非常接近的个体做对照,最终使选取观察对象的混杂变量在处理组和对照组趋于均衡可比.实例分析表明.利用PSM筛选后的研究对象,主要混杂因素在两组中的偏差下降在55%以上.结论 :PSM可有效降低观察性研究的混杂偏倚,在流行病学设计阶段使用PSM可使某些观察性研究得到类似随机对照研究的效果.     

Observational studies using propensity score analysis underestimated the effect sizes in critical care medicine

Background and ObjectivePropensity score (PS) analysis has been increasingly used in critical care medicine; however, its validation has not been systematically investigated. The present study aimed to compare effect sizes in PS-based observational studies vs. randomized controlled trials (RCTs) (or meta-analysis of RCTs).MethodsCritical care observational studies using PS were systematically searched in PubMed from inception to April 2013. Identified PS-based studies were matched to one or more RCTs in terms of population, intervention, comparison, and outcome. The effect sizes of experimental treatments were compared for PS-based studies vs. RCTs (or meta-analysis of RCTs) with sign test. Furthermore, ratio of odds ratio (ROR) was calculated from the interaction term of treatment × study type in a logistic regression model. A ROR < 1 indicates greater benefit for experimental treatment in RCTs compared with PS-based studies. RORs of each comparison were pooled by using meta-analytic approach with random-effects model.ResultsA total of 20 PS-based studies were identified and matched to RCTs. Twelve of the 20 comparisons showed greater beneficial effect for experimental treatment in RCTs than that in PS-based studies (sign test P = 0.503). The difference was statistically significant in four comparisons. ROR can be calculated from 13 comparisons, of which four showed significantly greater beneficial effect for experimental treatment in RCTs. The pooled ROR was 0.71 (95% CI: 0.63, 0.79; P = 0.002), suggesting that RCTs (or meta-analysis of RCTs) were more likely to report beneficial effect for the experimental treatment than PS-based studies. The result remained unchanged in sensitivity analysis and meta-regression.ConclusionIn critical care literature, PS-based observational study is likely to report less beneficial effect of experimental treatment compared with RCTs (or meta-analysis of RCTs).     

The value of audit and feedback reports in improving nutrition therapy in the intensive care unit: a multicenter observational study

Background: The objective of this study was to determine whether auditing practice and providing feedback in the form of benchmarked site reports is an effective strategy to improve adherence to nutrition guidelines. Methods: The authors conducted a multicenter observational study in Canadian intensive care units (ICUs). In January 2007, an audit of daily nutrition information was collected (type and amount of nutrition received and strategies to improve nutrition delivery). Each ICU was e‐mailed individualized benchmarked performance reports documenting their performance compared with the Canadian Critical Care Nutrition guidelines and in relation to the other ICUs. Nutrition practice was reaudited in May 2008 to evaluate changes in practice. Results: Twenty‐six ICUs in Canada participated, with 473 and 486 patients accrued in 2007 and 2008, respectively. The authors observed a significant increase in enteral nutrition (EN) adequacy (from 45.1% to 51.9% for calories, and from 44.8% to 51.5% for protein) and an increase in the percentage of patients receiving EN without parenteral nutrition (from 71.9% to 81.3%). They also observed trends toward improvements in the percentage of patients who had EN started within 48 hours (from 60.3% to 66.8%). There were no significant differences in the use of motility agents or small bowel feeding in patients who had high gastric residual volumes. Conclusion: Audit and feedback reports are associated with improvement in some nutrition practices in many ICUs; however, the magnitude of these effects is quite modest. More research is needed to determine the optimal methods of using audit and feedback to improve quality of nutrition care.     

Predictors of follow-up and assessment of selection bias from dropouts using inverse probability weighting in a cohort of university graduates

Dropouts in cohort studies can introduce selection bias. In this paper, we aimed (i) to assess predictors of retention in a cohort study (the SUN Project) where participants are followed-up through biennial mailed questionnaires, and (ii) to evaluate whether differential follow-up introduced selection bias in rate ratio (RR) estimates. The SUN Study recruited 9907 participants from December 1999 to January 2002. Among them, 8647 (87%) participants answered the 2-year follow-up questionnaire. The presence of missing information in key variables at baseline, being younger, smoker, a marital status different of married, being obese/overweight and a history of motor vehicle injury were associated with being lost to follow-up, while a self-reported history of cardiovascular disease predicted a higher retention proportion. To assess whether differential follow-up affected RR estimates, we studied the association between body mass index and the risk of hypertension, using inverse probability weighting (IPW) to adjust for␣confounding and selection bias. Obese individuals had a higher crude rate of hypertension compared with␣normoweight participants (RR = 6.4, 95% confidence interval (CI): 3.9–10.5). Adjustment for confounding using IPW attenuated the risk of hypertension associated to obesity (RR = 2.4, 95% CI: 1.1–5.3). Additional adjustment for selection bias did not modify the estimations. In conclusion, we show that the follow-up through mailed questionnaires of a geographically disperse cohort in Spain is possible. Furthermore, we show that despite existing differences between retained or lost to follow-up participants this may not necessarily have an important impact on the RR estimates of hypertension associated to obesity.     

Effect of propensity of seeking medical care on the bias of the estimated effectiveness of rotavirus vaccines from studies using a test-negative case-control design

BackgroundRotavirus is the leading cause of severe diarrhea among children worldwide, and vaccines can reduce morbidity and mortality by 50–98%. The test-negative control (TNC) study design is increasingly used for evaluating the effectiveness of vaccines against rotavirus and other vaccine-preventable diseases. In this study design, symptomatic patients who seek medical care are tested for the pathogen of interest. Those who test positive (negative) are classified as cases (controls).MethodsWe use a probability model to evaluate the bias of estimates of rotavirus vaccine effectiveness (VE) against rotavirus diarrhea resulting in hospitalization in the presence of possible confounding and selection biases due to differences in the propensity of seeking medical care (PSMC) between vaccinated and unvaccinated children.ResultsThe TNC-based VE estimate corrects for confounding bias when the confounder’s effects on the probabilities of rotavirus and non-rotavirus related hospitalizations are equal. If this condition is not met, then the estimated VE may be substantially biased. The bias is more severe in low-income countries, where VE is known to be lower. Under our model, differences in PSMC between vaccinated and unvaccinated children do not result in selection bias when the TNC study design is used.ConclusionsIn practice, one can expect the association of PSMC (or other potential confounders) with the probabilities of rotavirus and non-rotavirus related hospitalization to be similar, in which case the confounding effects will only result in small bias in the VE estimate from TNC studies. The results of this work, along with those of our previous paper, confirm the TNC design can be expected to provide reliable estimates of rotavirus VE in both high- and low-income countries.     

Methods to assess intended effects of drug treatment in observational studies are reviewed

BACKGROUND AND OBJECTIVE: To review methods that seek to adjust for confounding in observational studies when assessing intended drug effects. METHODS: We reviewed the statistical, economical and medical literature on the development, comparison and use of methods adjusting for confounding. RESULTS: In addition to standard statistical techniques of (logistic) regression and Cox proportional hazards regression, alternative methods have been proposed to adjust for confounding in observational studies. A first group of methods focus on the main problem of nonrandomization by balancing treatment groups on observed covariates: selection, matching, stratification, multivariate confounder score, and propensity score methods, of which the latter can be combined with stratification or various matching methods. Another group of methods look for variables to be used like randomization in order to adjust also for unobserved covariates: instrumental variable methods, two-stage least squares, and grouped-treatment approach. Identifying these variables is difficult, however, and assumptions are strong. Sensitivity analyses are useful tools in assessing the robustness and plausibility of the estimated treatment effects to variations in assumptions about unmeasured confounders. CONCLUSION: In most studies regression-like techniques are routinely used for adjustment for confounding, although alternative methods are available. More complete empirical evaluations comparing these methods in different situations are needed.     

A prospective observational study of the effect of penicillin skin testing on antibiotic use in the intensive care unit.

BACKGROUND: Patients with penicillin allergy admitted to the intensive care unit (ICU) frequently receive non-beta-lactam antimicrobials for the treatment of infection. The use of these antimicrobials, more commonly vancomycin and fluoroquinolones, is associated with the emergence of multidrug-resistant infections. The penicillin skin test (PST) can help detect patients at risk of developing an immediate allergic reaction to penicillin and those patients with a negative PST may be able to use a penicillin antibiotic safely. METHODS: We determined the incidence of true penicillin allergy, the percentage of patients changed to a beta-lactam antimicrobial when the test was negative, the safety of the test, and the safety of administration of beta-lactam antimicrobials in patients with a negative test. Skin testing was performed using standard methodology. RESULTS: One hundred patients admitted to 4 ICUs were prospectively studied; 58 of them were male. The mean age was 63 years. Ninety-six patients had the PST: one was positive (1.04%), 10 (10.4%) were nondiagnostic, and 85 (88.5%) were negative. Of the 38 patients who received antimicrobials for therapeutic reasons, 31(81.5%) had the antibiotic changed to a beta-lactam antimicrobial after a negative reading versus 7 patients of the 57 (12%) who had received a prophylactic antimicrobial (P < .001). No adverse effects were reported after the PST or after antimicrobial administration. CONCLUSIONS: The PST is a safe, reliable, and effective strategy to reduce the use of non-beta-lactam antimicrobials in patients who are labeled as penicillin allergic and admitted to the ICU.     

儿童重症监护病房收治的脓毒症患儿血清乳酸脱氢酶水平对病情及预后的预测价值

目的探讨脓毒症(sepsis)患儿转入儿童重症监护病房(PICU)24 h内血清乳酸脱氢酶(LDH)水平(以下简称为24 h-LDH水平),对其疾病程度及预后的预测价值。 方法收集2015年5月4日至2016年12月31日,于四川大学华西第二医院PICU住院治疗的326例脓毒症患儿为研究对象。回顾性分析其年龄、住院时间、24 h-LDH水平、感染性原发性疾病的受累器官和(或)系统、器官损伤、出院转归等临床病例资料。采用Wilcoxon秩和检验或Kruskal-Wallis H秩和检验,对患儿年龄、住院时间、24 h-LDH水平等进行比较分析。采用Spearman秩相关分析,对24 h-LDH水平分别与脓毒症严重程度、是否发生多器官功能障碍(MODS)和出院转归进行相关性分析。绘制脓毒症患儿24 h-LDH水平预测严重脓毒症、脓毒性休克(SS)、MODS及死亡的受试者工作特征(ROC)曲线,并计算ROC曲线下面积(AUC),根据约登指数最大原则,确定24 h-LDH水平,对预测严重脓毒症、SS、MODS及死亡的最佳临界值,并计算其预测上述指标的敏感度和特异度。本研究遵循的程序符合2013年修订的《世界医学协会赫尔辛基宣言》的要求。 结果①脓毒症、严重脓毒症、SS患儿发生MODS发生率分别为27.5%(30/109)、68.8%(121/176)和78.0%(32/41),死亡率分别为0、8.0%(14/176)和22.0%(9/41)。②脓毒症、严重脓毒症、SS患儿24 h-LDH水平中位数分别为718 U/L(560～976 U/L),871 U/L(666～1 972 U/L)和1 588 U/L(799～2 150 U/L),3者比较,差异有统计学意义(χ²＝27.445,P＝0.008)。发生MODS患儿的24 h-LDH水平中位数为713 U/L(571～967 U/L),显著低于未发生者的1 002 U/L(706～2 150 U/L),二者比较,差异有统计学意义(Z＝—5.745,P<0.001)。痊愈、好转、放弃和死亡患儿24 h-LDH水平中位数分别为712 U/L(584～945 U/L)、824 U/L(605～1 200 U/L)、1 311 U/L(716～2 150 U/L)和2 150 U/L(1 380～2 150 U/L),差异亦有统计学意义(χ²＝38.105,P<0.001)。③Spearman秩相关性分析结果显示,24 h-LDH水平分别与脓毒症严重程度、是否发生MODS呈正相关关系(r_s＝0.290、0.319,P<0.001),24 h-LDH水平与脓毒症患儿出院转归呈负相关关系(r_s＝ —0.387,P<0.001)。④ROC曲线分析结果显示,24 h-LDH水平预测患儿发生严重脓毒症、SS、MODS及死亡的ROC-AUC分别为0.658(95%CI：0.598～0.719,P<0.001),0.671(95%CI：0.583～0.759,P<0.001),0.685(95%CI：0.628～0.742,P<0.001)和0.788(95%CI：0.698～0.879,P<0.001)。根据约登指数最大原则,患儿发生上述4种情况时,血清LDH的最佳临界值分别为1 294.0 U/L、1 307.0 U/L、1 318.5 U/L和1 323.0 U/L(血清LDH水平正常参考值上限为618 U/L),此时对于预测患儿发生严重脓毒症、脓毒症休克、MODS和死亡的敏感度、特异度分别为39.2%、63.4%、43.7%和82.6%,87.2%、74.7%、78.2%和73.9%。⑤24 h-LDH水平>1 236 U/L的脓毒症患儿的MODS发生率,是24 h-LDH水平≤1 236 U/L者的4.565倍(OR＝4.565,95% CI：2.662～7.830,P<0.001)。⑥患儿出院转归情况为死亡、放弃、好转的与痊愈发生可能性比较,24 h-LDH水平>1 236 U/L患儿,分别较24 h-LDH水平≤1 236 U/L者增高25.175倍(OR＝25.175,95%CI：7.741～81.875,P<0.001),增高5.979倍(OR＝5.979,95%CI：3.142～11.380,P<0.001),增高1.619倍(OR＝1.619,95%CI：0.824～3.182,P＝0.162)。 结论脓毒症患儿24 h-LDH水平与脓毒症严重程度、是否发生MODS及出院转归情况均有关,尤其是24 h-LDH水平>1 236 U/L,可考虑作为预测脓毒症患儿发生MODS风险及判断预后的潜在生物标志物。     

Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study

Propensity score methods are increasingly being used to estimate causal treatment effects in the medical literature. Conditioning on the propensity score results in unbiased estimation of the expected difference in observed responses to two treatments. The degree to which conditioning on the propensity score introduces bias into the estimation of the conditional odds ratio or conditional hazard ratio, which are frequently used as measures of treatment effect in observational studies, has not been extensively studied. We conducted Monte Carlo simulations to determine the degree to which propensity score matching, stratification on the quintiles of the propensity score, and covariate adjustment using the propensity score result in biased estimation of conditional odds ratios, hazard ratios, and rate ratios. We found that conditioning on the propensity score resulted in biased estimation of the true conditional odds ratio and the true conditional hazard ratio. In all scenarios examined, treatment effects were biased towards the null treatment effect. However, conditioning on the propensity score did not result in biased estimation of the true conditional rate ratio. In contrast, conventional regression methods allowed unbiased estimation of the true conditional treatment effect when all variables associated with the outcome were included in the regression model. The observed bias in propensity score methods is due to the fact that regression models allow one to estimate conditional treatment effects, whereas propensity score methods allow one to estimate marginal treatment effects. In several settings with non-linear treatment effects, marginal and conditional treatment effects do not coincide.     

Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies

The propensity score is defined as a subject's probability of treatment selection, conditional on observed baseline covariates. Weighting subjects by the inverse probability of treatment received creates a synthetic sample in which treatment assignment is independent of measured baseline covariates. Inverse probability of treatment weighting (IPTW) using the propensity score allows one to obtain unbiased estimates of average treatment effects. However, these estimates are only valid if there are no residual systematic differences in observed baseline characteristics between treated and control subjects in the sample weighted by the estimated inverse probability of treatment. We report on a systematic literature review, in which we found that the use of IPTW has increased rapidly in recent years, but that in the most recent year, a majority of studies did not formally examine whether weighting balanced measured covariates between treatment groups. We then proceed to describe a suite of quantitative and qualitative methods that allow one to assess whether measured baseline covariates are balanced between treatment groups in the weighted sample. The quantitative methods use the weighted standardized difference to compare means, prevalences, higher‐order moments, and interactions. The qualitative methods employ graphical methods to compare the distribution of continuous baseline covariates between treated and control subjects in the weighted sample. Finally, we illustrate the application of these methods in an empirical case study. We propose a formal set of balance diagnostics that contribute towards an evolving concept of ‘best practice’ when using IPTW to estimate causal treatment effects using observational data. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

Strategies for the prevention of hospital-acquired infections in the neonatal intensive care unit

Nosocomial infections are among the leading causes of mortality and morbidity in neonatal intensive care units. Prevention of healthcare-associated infections is based on strategies that aim to limit susceptibility to infections by enhancing host defences, interrupting transmission of organisms by healthcare workers and by promoting the judicious use of antimicrobials. Several strategies are available and include: hand hygiene practices; prevention of central venous catheter-related bloodstream infections; judicious use of antimicrobials for therapy and prophylaxis; enhancement of host defences; skin care; and early enteral feeding with human milk.     

Outbreak of burkholderia cepacia in the adult intensive care unit traced to contaminated indigo-carmine dye.

We report an unusual cluster of Burkholderia cepacia in patients. Environmental cultures identified indigo-carmine dye used in enteral feeding as the reservoir. Compared with the controls, the cases were significantly more likely to have received tube feedings tinted with this dye. This outbreak was terminated with the removal of the dye from hospital inventory.     

2007-2009年重症监护病房病原菌分布及耐药性分析

目的了解医院重症监护病房(ICU)病原菌的分布特点及耐药性,为临床合理使用抗菌药物提供理论依据。方法回顾性分析2007年1月-2009年12月医院ICU分离的514株病原菌的耐药性,细菌药物敏感性测定采用K-B纸片扩散法。结果 514株病原菌中革兰阴性杆菌377株,占73.35%,革兰阳性球菌75株,占14.59%,真菌62株,占12.06%;革兰阴性杆菌以鲍氏不动杆菌、铜绿假单胞菌、肺炎克雷伯菌、嗜麦芽寡养单胞菌、大肠埃希菌为主,革兰阳性球菌以耐甲氧西林金黄色葡萄球菌(MRSA)为主;各类病原菌对常用抗菌药物表现为严重耐药和多药耐药。结论 ICU分离病原菌耐药情况严重,病原菌分布及耐药性的监测对临床合理用药、降低医院感染率有重大的指导意义。