Performance of disease risk scores, propensity scores, and traditional multivariable outcome regression in the presence of multiple confounders

Propensity scores are widely used in cohort studies to improve performance of regression models when considering large numbers of covariates. Another type of summary score, the disease risk score (DRS), which estimates disease probability conditional on nonexposure, has also been suggested. However, little is known about how it compares with propensity scores. Monte Carlo simulations were conducted comparing regression models using the DRS and the propensity score with models that directly adjust for all of the individual covariates. The DRS was calculated in 2 ways: from the unexposed population and from the full cohort. Compared with traditional multivariable outcome regression models, all 3 summary scores had comparable performance for moderate correlation between exposure and covariates and, for strong correlation, the full-cohort DRS and propensity score had comparable performance. When traditional methods had model misspecification, propensity scores and the full-cohort DRS had superior performance. All 4 models were affected by the number of events per covariate, with propensity scores and traditional multivariable outcome regression least affected. These data suggest that, for cohort studies for which covariates are not highly correlated with exposure, the DRS, particularly that calculated from the full cohort, is a useful tool.     

The Association Between Short Interpregnancy Interval and Preterm Birth in Louisiana: A Comparison of Methods

There is growing interest in the application of propensity scores (PS) in epidemiologic studies, especially within the field of reproductive epidemiology. This retrospective cohort study assesses the impact of a short interpregnancy interval (IPI) on preterm birth and compares the results of the conventional logistic regression analysis with analyses utilizing a PS. The study included 96,378 singleton infants from Louisiana birth certificate data (1995–2007). Five regression models designed for methods comparison are presented. Ten percent (10.17 %) of all births were preterm; 26.83 % of births were from a short IPI. The PS-adjusted model produced a more conservative estimate of the exposure variable compared to the conventional logistic regression method (β-coefficient: 0.21 vs. 0.43), as well as a smaller standard error (0.024 vs. 0.028), odds ratio and 95 % confidence intervals [1.15 (1.09, 1.20) vs. 1.23 (1.17, 1.30)]. The inclusion of more covariate and interaction terms in the PS did not change the estimates of the exposure variable. This analysis indicates that PS-adjusted regression may be appropriate for validation of conventional methods in a large dataset with a fairly common outcome. PS’s may be beneficial in producing more precise estimates, especially for models with many confounders and effect modifiers and where conventional adjustment with logistic regression is unsatisfactory. Short intervals between pregnancies are associated with preterm birth in this population, according to either technique. Birth spacing is an issue that women have some control over. Educational interventions, including birth control, should be applied during prenatal visits and following delivery.     

Evaluating bias from birth-cohort effects in the age-based cox proportional hazards model

The nested case-control design is frequently used to evaluate exposures and health outcomes within the confines of a cohort study. When incidence-density sampling is used to identify controls, the resulting data can be analyzed using conditional logistic regression (equivalent to stratified Cox proportional hazards regression). In these studies, exposure lagging is often used to account for disease latency. In light of recent criticism of incidence-density sampling, we used simulated occupational cohorts to evaluate age-based incidence-density sampling for lagged exposures in the presence of birth-cohort effects and associations among time-related variables. Effect estimates were unbiased when adjusted for birth cohort; however, unadjusted effect estimates were biased, particularly when age at hire and year of hire were correlated. When the analysis included an adjustment for birth cohort, the inclusion of lagged-out cases and controls (assigned a lagged exposure of zero) did not introduce bias.     

Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders

The aim of this study was to use Monte Carlo simulations to compare logistic regression with propensity scores in terms of bias, precision, empirical coverage probability, empirical power, and robustness when the number of events is low relative to the number of confounders. The authors simulated a cohort study and performed 252,480 trials. In the logistic regression, the bias decreased as the number of events per confounder increased. In the propensity score, the bias decreased as the strength of the association of the exposure with the outcome increased. Propensity scores produced estimates that were less biased, more robust, and more precise than the logistic regression estimates when there were seven or fewer events per confounder. The logistic regression empirical coverage probability increased as the number of events per confounder increased. The propensity score empirical coverage probability decreased after eight or more events per confounder. Overall, the propensity score exhibited more empirical power than logistic regression. Propensity scores are a good alternative to control for imbalances when there are seven or fewer events per confounder; however, empirical power could range from 35% to 60%. Logistic regression is the technique of choice when there are at least eight events per confounder.     

Propensity score and doubly robust methods for estimating the effect of treatment on censored cost

The estimation of treatment effects on medical costs is complicated by the need to account for informative censoring, skewness, and the effects of confounders. Because medical costs are often collected from observational claims data, we investigate propensity score (PS) methods such as covariate adjustment, stratification, and inverse probability weighting taking into account informative censoring of the cost outcome. We compare these more commonly used methods with doubly robust (DR) estimation. We then use a machine learning approach called super learner (SL) to choose among conventional cost models to estimate regression parameters in the DR approach and to choose among various model specifications for PS estimation. Our simulation studies show that when the PS model is correctly specified, weighting and DR perform well. When the PS model is misspecified, the combined approach of DR with SL can still provide unbiased estimates. SL is especially useful when the underlying cost distribution comes from a mixture of different distributions or when the true PS model is unknown. We apply these approaches to a cost analysis of two bladder cancer treatments, cystectomy versus bladder preservation therapy, using SEER‐Medicare data. Copyright © 2015 John Wiley & Sons, Ltd.     

Adjustment for multiple cardiovascular risk factors using a summary risk score

BACKGROUND: To simultaneously adjust for confounding by multiple cardiovascular risk factors, recently published pharmacoepidemiologic studies have used an index of risk of cardiovascular disease (a cardiovascular risk score). This summary measure is a multivariate confounder score created from regression models relating these risk factors to the outcome. The score is then used in regression models to adjust for potential confounding of the exposure of interest. Although this summary score has a number of advantages, there is concern that it may result in underestimation of the standard error of the exposure estimate and thus inflate the number of statistically significant results. METHODS: We conducted simulation studies comparing regression models adjusting for all risk factors directly to models using this summary risk score for large cohort studies. RESULTS: Results indicated that, as long as there was not a high degree of intercorrelation between the potential confounders and the exposure, estimated standard errors from the regression models using this summary risk score approximate their empirical standard errors well and are similar to the standard errors from the regression models directly adjusting for all risk factors. CONCLUSIONS: Based on these simulation results, using this summary risk score can be a reasonable approach for summarizing many risk factors in large cohort studies.     

Case-crossover analyses of air pollution exposure data: referent selection strategies and their implications for bias

The case-crossover design has been widely used to study the association between short-term air pollution exposure and the risk of an acute adverse health event. The design uses cases only; for each individual case, exposure just before the event is compared with exposure at other control (or "referent") times. Time-invariant confounders are controlled by making within-subject comparisons. Even more important in the air pollution setting is that time-varying confounders can also be controlled by design by matching referents to the index time. The referent selection strategy is important for reasons in addition to control of confounding. The case-crossover design makes the implicit assumption that there is no trend in exposure across the referent times. In addition, the statistical method that is used-conditional logistic regression-is unbiased only with certain referent strategies. We review here the case-crossover literature in the air pollution context, focusing on key issues regarding referent selection. We conclude with a set of recommendations for choosing a referent strategy with air pollution exposure data. Specifically, we advocate the time-stratified approach to referent selection because it ensures unbiased conditional logistic regression estimates, avoids bias resulting from time trend in the exposure series, and can be tailored to match on specific time-varying confounders.     

A matching method for improving covariate balance in cost-effectiveness analyses

In cost-effectiveness analyses (CEA) that use randomized controlled trials (RCTs), covariates of prognostic importance may be imbalanced and warrant adjustment. In CEA that use non-randomized studies (NRS), the selection on observables assumption must hold for regression and matching methods to be unbiased. Even in restricted circumstances when this assumption is plausible, a key concern is how to adjust for imbalances in observed confounders. If the propensity score is misspecified, the covariates in the matched sample will be imbalanced, which can lead to conditional bias. To address covariate imbalance in CEA based on RCTs and NRS, this paper considers Genetic Matching. This matching method uses a search algorithm to directly maximize covariate balance. We compare Genetic and propensity score matching in Monte Carlo simulations and two case studies, CEA of pulmonary artery catheterization, based on an RCT and an NRS. The simulations show that Genetic Matching reduces the conditional bias and root mean squared error compared with propensity score matching. Genetic Matching achieves better covariate balance than the unadjusted analyses of the RCT data. In the NRS, Genetic Matching improves on the balance obtained from propensity score matching and gives substantively different estimates of incremental cost-effectiveness. We conclude that Genetic Matching can improve balance on measured covariates in CEA that use RCTs and NRS, but with NRS, this will be insufficient to reduce bias; the selection on observables assumption must also hold.     

Confidence intervals for multinomial logistic regression in sparse data

Logistic regression is one of the most widely used regression models in practice, but alternatives to conventional maximum likelihood estimation methods may be more appropriate for small or sparse samples. Modification of the logistic regression score function to remove first-order bias is equivalent to penalizing the likelihood by the Jeffreys prior, and yields penalized maximum likelihood estimates (PLEs) that always exist, even in samples in which maximum likelihood estimates (MLEs) are infinite. PLEs are an attractive alternative in small-to-moderate-sized samples, and are preferred to exact conditional MLEs when there are continuous covariates. We present methods to construct confidence intervals (CI) in the penalized multinomial logistic regression model, and compare CI coverage and length for the PLE-based methods to that of conventional MLE-based methods in trinomial logistic regressions with both binary and continuous covariates. Based on simulation studies in sparse data sets, we recommend profile CIs over asymptotic Wald-type intervals for the PLEs in all cases. Furthermore, when finite sample bias and data separation are likely to occur, we prefer PLE profile CIs over MLE methods.     

Correction of logistic regression relative risk estimates and confidence intervals for systematic within-person measurement error

Errors in the measurement of exposure that are independent of disease status tend to bias relative risk estimates and other measures of effect in epidemiologic studies toward the null value. Two methods are provided to correct relative risk estimates obtained from logistic regression models for measurement errors in continuous exposures within cohort studies that may be due to either random (unbiased) within-person variation or to systematic errors for individual subjects. These methods require a separate validation study to estimate the regression coefficient lambda relating the surrogate measure to true exposure. In the linear approximation method, the true logistic regression coefficient beta* is estimated by beta/lambda, where beta is the observed logistic regression coefficient based on the surrogate measure. In the likelihood approximation method, a second-order Taylor series expansion is used to approximate the logistic function, enabling closed-form likelihood estimation of beta*. Confidence intervals for the corrected relative risks are provided that include a component representing error in the estimation of lambda. Based on simulation studies, both methods perform well for true odds ratios up to 3.0; for higher odds ratios the likelihood approximation method was superior with respect to both bias and coverage probability. An example is provided based on data from a prospective study of dietary fat intake and risk of breast cancer and a validation study of the questionnaire used to assess dietary fat intake.     

Residential exposure to petrochemicals and the risk of leukemia: using geographic information system tools to estimate individual-level residential exposure

The authors conducted a population-based, case-control study in Kaohsiung, southern Taiwan, Republic of China, to investigate the association between residential petrochemical exposure and leukemia risk among subjects 29 years of age and younger. Between November 1997 and June 2003, 171 cases and 410 controls matched for age and sex were recruited. Since assessment of petrochemical impacts depends on accurate exposure estimates, the authors developed a procedure using geographic information system tools to assign subjects' exposure. The resulting individual-level exposure estimates (the exposure opportunity score) are an integrated exposure measure that accounts for subjects' mobility, length of stay at each residence, distance to petrochemical plant(s), monthly prevailing wind direction, and multiple petrochemical pollution sources. Different conditional logistic regression models were fitted for subjects aged 0-19 and 20-29 years to evaluate separately childhood versus adulthood leukemia. No overall association was observed for the younger age group. However, residential petrochemical exposure was a significant risk factor for leukemia for the older age group. For one unit of increase in the log-transformed exposure opportunity score, the adjusted odds ratio was 1.54 (95 percent confidence interval: 1.14, 2.09). This study illustrates the utility of geographic information system tools for providing refined exposure estimates for residential exposure to petrochemical pollution.     

Comparison of the missing-indicator method and conditional logistic regression in 1:m matched case-control studies with missing exposure values

The missing-indicator method and conditional logistic regression have been recommended as alternative approaches for data analysis in matched case-control studies with missing exposure values. The authors evaluated the performance of the two methods using Monte Carlo simulation. Data were generated from a 1:m matched design based on McNemar's 2 x 2 tables with four scenarios for missing values: completely-at-random, case-dependent, exposure-dependent, and case/exposure-dependent. In their analysis, the authors used conditional logistic regression for complete pairs and the missing-indicator method for all pairs. For 1:1 matched studies, given no confounding between exposure and disease, the two methods yielded unbiased estimates. Otherwise, conditional logistic regression produced unbiased estimates with empirical confidence interval coverage similar to nominal coverage under the first three missing-value scenarios, whereas the missing-indicator method produced slightly more bias and lower confidence interval coverage. An increased number of matched controls was associated with slightly more bias and lower confidence interval coverage. Under the case/exposure-dependent missing-value scenario, neither method performed satisfactorily; this indicates the need for more sophisticated statistical methods for handling such missing values. Overall, compared with the missing-indicator method, conditional logistic regression provided a slight advantage in terms of bias and coverage probability, at the cost of slightly reduced statistical power and efficiency.     

Performance of principal scores to estimate the marginal compliers causal effect of an intervention

We examine the properties of principal scores methods to estimate the causal marginal odds ratio of an intervention for compliers in the context of a randomized controlled trial with non‐compliers. The two‐stage estimation approach has been proposed for a linear model by Jo and Stuart (Statistics in Medicine 2009; 28 :2857–2875) under a principal ignorability (PI) assumption. Using a Monte Carlo simulation study, we compared the performance of several strategies to build and use principal score models and the robustness of the method to violations of underlying assumptions, in particular PI. Results showed that the principal score approach yielded unbiased estimates of the causal marginal log odds ratio under PI but that the method was sensitive to violations of PI, which occurs in particular when confounders are omitted from the analysis. For principal score analysis, probability weighting performed slightly better than full matching or 1:1 matching. Concerning the variables to be included in principal score models, the lowest mean squared error was generally obtained when using the true confounders. Using variables associated with the outcome only but not compliance however yielded very similar performance. Copyright © 2015 John Wiley & Sons, Ltd.     

Unconditional analyses can increase efficiency in assessing gene-environment interaction of the case-combined-control design

BACKGROUND: A design combining both related and unrelated controls, named the case-combined-control design, was recently proposed to increase the power for detecting gene-environment (GxE) interaction. Under a conditional analytic approach, the case-combined-control design appeared to be more efficient and feasible than a classical case-control study for detecting interaction involving rare events. METHODS: We now propose an unconditional analytic strategy to further increase the power for detecting gene-environment (GxE) interactions. This strategy allows the estimation of GxE interaction and exposure (E) main effects under certain assumptions (e.g. no correlation in E between siblings and the same exposure frequency in both control groups). Only the genetic (G) main effect cannot be estimated because it is biased. RESULTS: Using simulations, we show that unconditional logistic regression analysis is often more efficient than conditional analysis for detecting GxE interaction, particularly for a rare gene and strong effects. The unconditional analysis is also at least as efficient as the conditional analysis when the gene is common and the main and joint effects of E and G are small. CONCLUSIONS: Under the required assumptions, the unconditional analysis retains more information than does the conditional analysis for which only discordant case-control pairs are informative leading to more precise estimates of the odds ratios.     

Doubly robust conditional logistic regression

Epidemiologic research often aims to estimate the association between a binary exposure and a binary outcome, while adjusting for a set of covariates (eg, confounders). When data are clustered, as in, for instance, matched case-control studies and co-twin-control studies, it is common to use conditional logistic regression. In this model, all cluster-constant covariates are absorbed into a cluster-specific intercept, whereas cluster-varying covariates are adjusted for by explicitly adding these as explanatory variables to the model. In this paper, we propose a doubly robust estimator of the exposure-outcome odds ratio in conditional logistic regression models. This estimator protects against bias in the odds ratio estimator due to misspecification of the part of the model that contains the cluster-varying covariates. The doubly robust estimator uses two conditional logistic regression models for the odds ratio, one prospective and one retrospective, and is consistent for the exposure-outcome odds ratio if at least one of these models is correctly specified, not necessarily both. We demonstrate the properties of the proposed method by simulations and by re-analyzing a publicly available dataset from a matched case-control study on induced abortion and infertility.     

Confounder-adjusted estimates of the risk difference using propensity score-based weighting

Confounder-adjusted estimates of the risk difference are often difficult to obtain by direct regression adjustment. Estimates can be obtained from a propensity score-based method using inverse probability-of-exposure weights to balance groups defined by exposure status with respect to confounders. Simulation was used to evaluate the performance of this method. The simulation model incorporated a binary confounder and a normally distributed confounder into logistic models of exposure status, and disease status conditional on exposure status. Data were generated for combinations of values of several design parameters, including the odds ratio relating each of the confounders to exposure status, the odds ratio relating each of the confounders to disease status and the total sample size. For most design parameter combinations (474 of 486), the absolute bias in the estimated risk difference was less than 1 percentage point, and it was never greater than 3 percentage points. The confidence interval generally had close to nominal 95 per cent coverage, but was prone to poor coverage levels (as low as 78.5 per cent) when both the confounder-to-exposure and confounder-to-outcome odds ratios were 5, consistent with strong confounding. The simulation results showed that the conditions that are favourable for good performance of the weighting method are: reasonable overlap in the propensity score distributions of the exposed and non-exposed groups and a large sample size.     

What's the Risk? A Simple Approach for Estimating Adjusted Risk Measures from Nonlinear Models Including Logistic Regression

Objective. To develop and validate a general method (called regression risk analysis) to estimate adjusted risk measures from logistic and other nonlinear multiple regression models. We show how to estimate standard errors for these estimates. These measures could supplant various approximations (e.g., adjusted odds ratio [AOR]) that may diverge, especially when outcomes are common.
Study Design. Regression risk analysis estimates were compared with internal standards as well as with Mantel–Haenszel estimates, Poisson and log-binomial regressions, and a widely used (but flawed) equation to calculate adjusted risk ratios (ARR) from AOR.
Data Collection. Data sets produced using Monte Carlo simulations.
Principal Findings. Regression risk analysis accurately estimates ARR and differences directly from multiple regression models, even when confounders are continuous, distributions are skewed, outcomes are common, and effect size is large. It is statistically sound and intuitive, and has properties favoring it over other methods in many cases.
Conclusions. Regression risk analysis should be the new standard for presenting findings from multiple regression analysis of dichotomous outcomes for cross-sectional, cohort, and population-based case–control studies, particularly when outcomes are common or effect size is large.     

Propensity score model overfitting led to inflated variance of estimated odds ratios

ObjectiveSimulation studies suggest that the ratio of the number of events to the number of estimated parameters in a logistic regression model should be not less than 10 or 20 to 1 to achieve reliable effect estimates. Applications of propensity score approaches for confounding control in practice, however, do often not consider these recommendations.Study Design and SettingWe conducted extensive Monte Carlo and plasmode simulation studies to investigate the impact of propensity score model overfitting on the performance in estimating conditional and marginal odds ratios using different established propensity score inference approaches. We assessed estimate accuracy and precision as well as associated type I error and type II error rates in testing the null hypothesis of no exposure effect.ResultsFor all inference approaches considered, our simulation study revealed considerably inflated standard errors of effect estimates when using overfitted propensity score models. Overfitting did not considerably affect type I error rates for most inference approaches. However, because of residual confounding, estimation performance and type I error probabilities were unsatisfactory when using propensity score quintile adjustment.ConclusionOverfitting of propensity score models should be avoided to obtain reliable estimates of treatment or exposure effects in individual studies.     

The performance of different propensity-score methods for estimating relative risks

OBJECTIVES: The propensity score is the probability of treatment conditional on observed variables. Conditioning on the propensity-score results in unbiased estimation of the expected difference in observed responses to two treatments. The performance of propensity-score methods for estimating relative risks has not been studied. STUDY DESIGN AND SETTING: Monte Carlo simulations were used to assess the performance of matching, stratification, and covariate adjustment using the propensity score to estimate relative risks. RESULTS: Matching on the propensity score and stratification on the quintiles of the propensity score resulted in estimates of relative risk with similar mean squared error (MSE). Propensity-score matching resulted in estimates with less bias, whereas stratification on the propensity score resulted in estimates of with greater precision. Including only variables associated with the outcome or including only the true confounders in the propensity-score model resulted in estimates with lower MSE than did including all variables associated with treatment or all measured variables in the propensity-score model. CONCLUSIONS: When estimating relative risks, propensity-score matching resulted in estimates with less bias than did stratification on the quintiles of the propensity score, but stratification on the quintiles of the propensity score resulted in estimates with greater precision.     

Regression standardization with the <Emphasis FontCategory="NonProportional">R</Emphasis> package <Emphasis FontCategory="NonProportional">stdReg</Emphasis>

When studying the association between an exposure and an outcome, it is common to use regression models to adjust for measured confounders. The most common models in epidemiologic research are logistic regression and Cox regression, which estimate conditional (on the confounders) odds ratios and hazard ratios. When the model has been fitted, one can use regression standardization to estimate marginal measures of association. If the measured confounders are sufficient for confounding control, then the marginal association measures can be interpreted as poulation causal effects. In this paper we describe a new R package, stdReg, that carries out regression standardization with generalized linear models (e.g. logistic regression) and Cox regression models. We illustrate the package with several examples, using real data that are publicly available.