Repeated split sample validation to assess logistic regression and recursive partitioning: an application to the prediction of cognitive impairment

Screening strategies play an important part in the identification and diagnosis of illness. Testing of such strategies in a clinical trial can have important implications for the treatment of such illnesses. Before the clinical trial, however, it is important to develop a practical screening/classification procedure that accurately predicts the presence of the illness in question. Recent published studies have shown a growing preference for classification tree/recursive partitioning procedures.This paper compares the application of logistic regression and recursive partitioning to a neuropsychological data set of 252 patients recruited from four Veterans Affairs Medical Centers. Logistic regression and recursive partitioning was used to predict cognitive impairment in 12 randomly selected exploratory/validation samples. We assessed the effect of sampling on variable selection and predictive accuracy.Predictive accuracy of the logistic regression and recursive partitioning procedures was comparable across the exploratory data samples but varied across the validation samples. Based on shrinkage, both classification procedures performed equally well for the prediction of cognitive impairment across the twelve samples. While logistic regression provided an estimated probability of outcome for each patient, it required several mathematical calculations to do so. However, logistic regression selected one or two less predictors than recursive partitioning with comparable predictive accuracy. Recursive partitioning, on the other hand, readily identified patient characteristics and variable interactions, was easy to interpret clinically and required no mathematical calculations. There was a high degree of overlap of the predictor variables between the two procedures.In the context of neuropsychological screening, logistic regression and recursive partitioning performed equally well and were quite stable in the selection of predictors for the identification of patients with cognitive impairment, although recursive partitioning may be easier to use in a clinical setting because it is based on a simple decision tree. Copyright (c) 2005 John Wiley & Sons, Ltd.     

The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models

Assessing the calibration of methods for estimating the probability of the occurrence of a binary outcome is an important aspect of validating the performance of risk-prediction algorithms. Calibration commonly refers to the agreement between predicted and observed probabilities of the outcome. Graphical methods are an attractive approach to assess calibration, in which observed and predicted probabilities are compared using loess-based smoothing functions. We describe the Integrated Calibration Index (ICI) that is motivated by Harrell's E_max index, which is the maximum absolute difference between a smooth calibration curve and the diagonal line of perfect calibration. The ICI can be interpreted as weighted difference between observed and predicted probabilities, in which observations are weighted by the empirical density function of the predicted probabilities. As such, the ICI is a measure of calibration that explicitly incorporates the distribution of predicted probabilities. We also discuss two related measures of calibration, E50 and E90, which represent the median and 90th percentile of the absolute difference between observed and predicted probabilities. We illustrate the utility of the ICI, E50, and E90 by using them to compare the calibration of logistic regression with that of random forests and boosted regression trees for predicting mortality in patients hospitalized with a heart attack. The use of these numeric metrics permitted for a greater differentiation in calibration than was permissible by visual inspection of graphical calibration curves.     

Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models

The change in c‐statistic is frequently used to summarize the change in predictive accuracy when a novel risk factor is added to an existing logistic regression model. We explored the relationship between the absolute change in the c‐statistic, Brier score, generalized R², and the discrimination slope when a risk factor was added to an existing model in an extensive set of Monte Carlo simulations. The increase in model accuracy due to the inclusion of a novel marker was proportional to both the prevalence of the marker and to the odds ratio relating the marker to the outcome but inversely proportional to the accuracy of the logistic regression model with the marker omitted. We observed greater improvements in model accuracy when the novel risk factor or marker was uncorrelated with the existing predictor variable compared with when the risk factor has a positive correlation with the existing predictor variable. We illustrated these findings by using a study on mortality prediction in patients hospitalized with heart failure. In conclusion, the increase in predictive accuracy by adding a marker should be considered in the context of the accuracy of the initial model. Copyright © 2012 John Wiley & Sons, Ltd.     

Selecting risk factors: A comparison of discriminant analysis,logistic regression and cox's regression model using data from the tromsø heart study

For comparative evaluation, discriminant analysis, logistic regression and Cox's model were used to select risk factors for total and coronary deaths among 6595 men aged 20-49 followed for 9 years. Groups with mortality between 5 and 93 per 1000 were considered. Discriminant analysis selected variable sets only marginally different from the logistic and Cox methods which always selected the same sets. A time-saving option, offered for both the logistic and Cox selection, showed no advantage compared with discriminant analysis. Analysing more than 3800 subjects, the logistic and Cox methods consumed, respectively, 80 and 10 times more computer time than discriminant analysis. When including the same set of variables in non-stepwise analyses, all methods estimated coefficients that in most cases were almost identical. In conclusion, discriminant analysis is advocated for preliminary or stepwise analysis, otherwise Cox's method should be used.     

A comparison of methods for combining quality and efficiency performance measures: profiling the value of hospital care following acute myocardial infarction

Health plans have begun to combine data on the quality and cost of medical providers in an attempt to identify and reward those that offer the greatest 'value.' The analytical methods used to combine these measures in the context of provider profiling have not been rigorously studied. We propose three methods to measure and compare the value of hospital care following acute myocardial infarction by combining a single measure of quality, in-hospital survival, and the cost of an episode of acute care. To illustrate these methods, we use administrative data for heart attack patients treated at 69 acute care hospitals in Massachusetts in fiscal year 2003. In the first method we reproduce a common approach to value profiling by modeling the two case mix-standardized outcomes independently. In the second approach, survival is regressed on patient risk factors and the average cost of care at each hospital. The third method models survival and cost for each hospital jointly and combines the outcomes on a common scale using a cost-effectiveness framework. For each method we use the resulting parameter estimates or functions of the estimates to compute posterior tail probabilities, representing the probability of being classified in the upper or lower quartile of the statewide distribution. Hospitals estimated to have the highest and lowest value according to each method are compared for consistency, and the advantages and disadvantages of each approach are discussed.