Bias in estimates of confidence intervals for health outcome report cards

Health researchers commonly use logistic regression when profiling health providers. Data from the patients treated by the providers are used to construct models predicting the expected number of outcomes for providers and the ratio of observed to expected outcomes (O/E ratio) used as a risk-adjusted measure of provider performance. Typically, when calculating the standard deviation (SD) of O/E ratios, only O is treated as a random variable. We used the propagation of errors (Pe) to derive a SD estimate that accounted for variability in O and the estimate of E. Using data previously used to profile Canadian cardiac surgery providers, we compared Pe-SD estimates with typical SD (SDT) estimates. The SDT estimates and confidence intervals were always larger than the Pe estimates, most notably when one or more providers treated a large proportion of the patients. This was confirmed using computer simulations. SDT estimates should be abandoned in favor of more sophisticated estimates.