Effect of mortality rate on the performance of the Acute Physiology and Chronic Health Evaluation II: a simulation study

OBJECTIVE: To evaluate the impact of case mix variation on the performance of the Acute Physiology and Chronic Health Evaluation (APACHE) II using measures of calibration and discrimination. DESIGN: APACHE II data were collected prospectively at the surgical intensive care unit of the University of Vermont on all adult admissions over an 8-yr period (excluding cardiac surgical patients, burn patients, and patients < 16 yrs of age). The original case mix was systematically varied to create 2,000 different case mixes ranging in mortality between 5% and 18% using a computer-intensive resampling algorithm. The area under the receiver operating characteristic curve and the Hosmer-Lemeshow C statistic were derived for each of the simulated case mixes with bootstrapping. SETTING: The surgical intensive care unit at a 450-bed teaching hospital. PATIENTS: A group of 6,806 adult surgical patients excluding cardiac surgical patients and burn patients. MEASUREMENTS AND RESULTS: Simulated data sets were created from a database of patients treated at a single institution to test the hypothesis that the performance of APACHE II is stable across a clinically reasonable range of mortality rates. The discrimination and calibration of APACHE II varied with case mix. CONCLUSION: The discrimination of APACHE II is not independent of case mix. However, the variability of the Hosmer-Lemeshow statistic as a function of the case mix may simply reflect the limitations of this goodness of fit statistic to assess model calibration. Because the discrimination of APACHE II is a function of case mix, caution should be exercised when using APACHE II-based adjusted mortality rates to compare intensive care units with widely divergent case mixes.     

External validation of severity scoring systems for acute renal failure using a multinational database

OBJECTIVE: Several different severity scoring systems specific to acute renal failure have been proposed. However, most validation studies of these scoring systems were conducted in a single center or in a small number of centers, often the same ones used for their development. Therefore, it is not known whether such severity scoring systems may be widely applied. DESIGN: Prospective clinical investigation. SETTING: Intensive care units. PATIENTS: One thousand seven hundred and forty-two intensive care unit patients with acute renal failure who were either treated with renal replacement therapy or fulfilled predefined criteria. INTERVENTIONS: Demographic and clinical information and outcomes were measured. MEASUREMENTS AND MAIN RESULTS: Scores for four acute renal failure-specific scoring systems and two general scoring systems (Simplified Acute Physiology Score II and Sequential Organ Failure Assessment) were calculated, and their discrimination and calibration were tested with receiver operating characteristic curves and Hosmer-Lemeshow goodness-of fit-tests. For the receiver operating characteristic curves, blood lactate levels were also used as a reference. All scores had an area under the receiver operating characteristic curve <0.7 (Mehta 0.670, Liano 0.698, Chertow 0.610, Paganini 0.643, Simplified Acute Physiology Score II 0.645, Sequential Organ Failure Assessment 0.675, lactate 0.639). For scores that can calculate predicted mortality, the Hosmer-Lemeshow goodness-of-fit test showed poor calibration. CONCLUSIONS: None of the scoring systems tested had a high level of discrimination or calibration to predict mortality for patients with acute renal failure when tested in a broad cohort of patients from multiple countries. A large, multiple-center database might be needed to improve the discrimination and calibration of acute renal failure scoring system.     

Toxic epidermal necrolysis: does immunoglobulin make a difference?

Experimental evidence implicates Fas ligand-mediated keratinocyte apoptosis as an underlying mechanism of toxic epidermal necrolysis syndrome (TEN). In vitro studies indicate a potential role for immunoglobulin (Ig) therapy in blocking Fas ligand signaling, thus reducing the severity of TEN. Anecdotal reports have described successful treatment of TEN patients with Ig; however, no study to date has analyzed outcome data in a large series of patients treated with Ig using institutional controls. The SCORTEN severity-of-illness score ranks severity and predicts prognosis in TEN patients using age, heart rate, TBSA slough, history of malignancy, and admission blood urea nitrogen, serum bicarbonate, and glucose levels. A retrospective chart review was performed that included all patients treated for TEN at our burn center since 1997. Ig therapy was instituted for all patients with biopsy-proven TEN beginning in January 2000. Twenty-one TEN patients were treated before Ig (no-Ig group), and 24 patients have been treated with Ig. SCORTEN data were collected, as well as length of stay (LOS) and status upon discharge. Each patient was given a SCORTEN of 0 to 6, with 1 point each for age greater than 40, TBSA slough greater than 10%, history of malignancy, admission BUN greater than 28 mg/dl, HCO3 less than 20 mg/dl, and glucose greater then 252 mg/dl. Outcome was compared between patients treated with Ig and without Ig. Overall mortality for patients treated before Ig was 28.6% (6/21), and with Ig, mortality was 41.7%% (10/24). There was no significant difference in age or TBSA slough. The average SCORTEN between the groups was equivalent (2.2 in no-Ig group vs 2.7 in Ig group, P = 0.3), and no group of patients with any SCORTEN score showed a significant benefit from Ig therapy. Overall LOS as well as LOS for survivors was longer in the Ig group. This series represents the largest single-institution analysis of TEN patient outcome after institution of Ig therapy. Our data do not show a significant improvement in mortality for TEN patients treated with Ig at any level of severity and may indicate a potential detriment in using Ig. Ig should not be given to TEN patients outside of a clinical trial. A multicenter, prospective, double-blinded randomized trial is necessary and urgently indicated to determine whether Ig therapy is beneficial or harmful in the care of TEN patients.     

Short-term and long-term outcome prediction with the Acute Physiology and Chronic Health Evaluation II system after orthotopic liver transplantation

OBJECTIVE: To evaluate the relationship between the postoperative Acute Physiology and Chronic Health Evaluation (APACHE) II score and mortality at hospital discharge and at 1 yr in liver transplant recipients. POPULATION: Adult orthotopic liver transplant (OLTX) recipients (n = 599) admitted to the intensive care unit postoperatively at a university hospital. METHODS: The cohort was split randomly into development and validation sets. Three models were compared for each end point: a) the original APACHE II slope with the original APACHE II postgastrointestinal surgery intercept; b) the original APACHE II slope with an OLTX-specific intercept generated from the development set; and c) an OLTX-specific slope and intercept generated from the development set. Goodness-of-fit and calibration were assessed by the Hosmer-Lemeshow C statistic (where p>.05 suggests good fit) and standardized mortality ratios. Discrimination was assessed by receiver operator characteristic area under the curve analysis. MEASUREMENTS AND MAIN RESULTS: Hospital and 1-yr mortality rates were 9.9% and 15.9%, respectively. The APACHE II score was strongly associated with mortality (chi-square, p<.0001), but when used with the original equation, it significantly overestimated hospital mortality (standardized mortality ratio, 0.73 [confidence interval, 0.58-0.99]). Using the OLTX-specific approaches, goodness-of-fit for both hospital and 1-yr mortality was good (p = .2-.57) but discrimination was only moderate (receiver operator characteristic area under the curve, 0.675-0.723). CONCLUSIONS: APACHE II is a good predictor of short- and long-term mortality after liver transplantation, especially when using OLTX-specific coefficients. Because fit and calibration were better than discrimination, APACHE II will be most useful in the prediction of risk for groups of patients (e.g., in clinical trials or institutional comparisons) rather than for individuals. This study raises the possibility that APACHE II may be useful for long-term mortality prediction in other critically ill populations. The overestimation of mortality using the original equation suggests that orthotopic liver transplantation, by reversing the underlying pathophysiology, may modify risk.     

Risk prediction of hospital mortality for adult patients admitted to Australian and New Zealand intensive care units: Development and validation of the Australian and New Zealand Risk of Death model

Purpose

The purpose of this study is to develop and validate a new mortality prediction model (Australian and New Zealand Risk of Death [ANZROD]) for Australian and New Zealand intensive care units (ICUs) and compare its performance with the existing Acute Physiology and Chronic Health Evaluation (APACHE) III-j.

Materials and Methods

All ICU admissions from 2004 to 2009 were extracted from the Australian and New Zealand Intensive Care Society Adult Patient Database. Hospital mortality was modeled using logistic regression with training (two third) and validation (one third) data sets. Predictor variables included APACHE III score components, source of admission to ICU and hospital, lead time, elective surgery, treatment limitation, ventilation status, and APACHE III diagnoses. Model performance was assessed by standardized mortality ratio, Hosmer-Lemeshow C and H statistics, Brier score, Cox calibration regression, area under the receiver operating characteristic curve, and calibration curves.

Results

There were 456 605 patients available for model development and validation. Observed mortality was 11.3%. Performance measures (standardized mortality ratio, Hosmer-Lemeshow C and H statistics, and receiver operating characteristic curve) for the ANZROD and APACHE III-j model in the validation data set were 1.01, 104.9 and 111.4, and 0.902; 0.84, 1596.6 and 2087.3, and 0.885, respectively.

Conclusions

The ANZROD has better calibration; discrimination compared with the APACHE III-j. Further research is required to validate performance over time and in specific subgroups of ICU population.     

Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients

OBJECTIVE: To improve the accuracy of the Acute Physiology and Chronic Health Evaluation (APACHE) method for predicting hospital mortality among critically ill adults and to evaluate changes in the accuracy of earlier APACHE models. DESIGN:: Observational cohort study. SETTING: A total of 104 intensive care units (ICUs) in 45 U.S. hospitals. PATIENTS: A total of 131,618 consecutive ICU admissions during 2002 and 2003, of which 110,558 met inclusion criteria and had complete data. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We developed APACHE IV using ICU day 1 information and a multivariate logistic regression procedure to estimate the probability of hospital death for randomly selected patients who comprised 60% of the database. Predictor variables were similar to those in APACHE III, but new variables were added and different statistical modeling used. We assessed the accuracy of APACHE IV predictions by comparing observed and predicted hospital mortality for the excluded patients (validation set). We tested discrimination and used multiple tests of calibration in aggregate and for patient subgroups. APACHE IV had good discrimination (area under the receiver operating characteristic curve = 0.88) and calibration (Hosmer-Lemeshow C statistic = 16.9, p = .08). For 90% of 116 ICU admission diagnoses, the ratio of observed to predicted mortality was not significantly different from 1.0. We also used the validation data set to compare the accuracy of APACHE IV predictions to those using APACHE III versions developed 7 and 14 yrs previously. There was little change in discrimination, but aggregate mortality was systematically overestimated as model age increased. When examined across disease, predictive accuracy was maintained for some diagnoses but for others seemed to reflect changes in practice or therapy. CONCLUSIONS: APACHE IV predictions of hospital mortality have good discrimination and calibration and should be useful for benchmarking performance in U.S. ICUs. The accuracy of predictive models is dynamic and should be periodically retested. When accuracy deteriorates they should be revised and updated.     

Prognostic accuracy of Acute Physiology and Chronic Health Evaluation II scores in critically ill cancer patients.

BACKGROUND: The predictive accuracy of scores on the Acute Physiology and Chronic Health Evaluation II (APACHE II) for in-hospital mortality among critically ill cancer patients varies. OBJECTIVE: To evaluate the predictive accuracy of APACHE II scores for severity of illness in critically ill cancer patients and to find clinical indicators to improve the accuracy. METHODS: Actual hospital mortality rates were compared with predicted rates. Data were collected prospectively from 1263 cancer patients admitted to the intensive care unit during a 5-year period in a cancer center in Taiwan. The APACHE II score for each patient was calculated at admission. Stepwise logistic regression was used to identify clinical predictors associated with increased mortality. RESULTS: The scores ranged from 2 to 54. The mortality rates were 19% overall, 45% for medical patients, and 1% for surgical patients. The fit of the scores was good for the medical patients (Hosmer-Lemeshow statistic 8.2, P = .41). The estimated odds ratios for mortality of presence of metastasis and respiratory failure were 4.18 (95% CI 2.65-6.59) and 2.03 (95% CI 1.22-3.38), respectively. When metastasis and respiratory failure were incorporated into the APACHE II model, the area under the receiver operating characteristic curve for medical patients increased from 0.82 to 0.86. The fit of the modified model was excellent (Hosmer and Lemeshow statistic 6.57, P=.58). CONCLUSIONS: APACHE II scores are predictive of hospital mortality in critically ill cancer patients. The presence of metastasis and respiratory failure at admission are also associated with outcome.     

The apache III prognostic system: customized mortality predictions for Spanish ICU patients

Objective: To customize the Acute Physiology and Chronic Health Evaluation (APACHE) III mortality equation for Spanish admissions to the intensive care unit (ICU) and evaluate its discrimination and calibration. Design: Prospective multicenter inception cohort study. Setting: 86 ICUs located in all regions of Spain. Patients: 10 929 adult patients selected by a systematic sampling method. All types of critical care patients were included, including coronary bypass patients, but excluding those with burn injury, those admitted for pacemaker implants, patients under 16 years of age, and patients with length of ICU stay < 6 h. Measurements and results: Data collection in the first 24 h after patient admission included: APACHE III score, treatment location prior to ICU admission, and main ICU admission diagnosis. Using these variables, a model for predicting hospital mortality was constructed, adapted to Spain, and its discriminating ability was assessed by the area below the ROC curve, which was 0.83. The model was validated using the jacknife method and the area below the receiver operating characterisitic (ROC) curve for the cross-validated predictions was 0.82. The percentage of patients correctly classified at 0.50 risk of death was 82.3 %. Model calibration was evaluated by analysis of the agreement between the observed and cross-validated predicted mortality using the Hosmer-Lemeshow test, which gave a value of (H) 12.27, with no statistical significance, i. e., good calibration. Conclusions: We have customized the APACHE III mortality prediction system for the Spanish population. This adapted model has demonstrated the requisite validation, calibration, and discrimination for its use among Spanish critical care patients. Received: 22 September 1997 Accepted: 19 February 1998     

Is the ACLS Score a Valid Prediction Rule for Survival after Cardiac Arrest?

The ACLS (advanced cardiac life support) Score was previously developed to predict survival from out-of-hospital cardiac arrest. Whether the arrest was witnessed, initial cardiac rhythm, performance of bystander cardiopulmonary resuscitation (CPR), and the response time of the paramedic unit were determined to be predictive of survival. However, the ACLS Score has not been validated in other emergency medical services systems. OBJECTIVES: The purpose of this study was to externally validate the ACLS Score in one patient population. METHODS: This was a retrospective cohort study performed at an urban county teaching hospital. The study population consisted of consecutive adult patients treated for out-of-hospital, nontraumatic cardiac arrest, and transported to the authors' institution between November 1, 1994, and September 30, 2001. Patient records for all cardiac arrests during the study period were reviewed. Study variables included witnessed arrest, initial arrest rhythm, bystander CPR, paramedic response time, and survival to hospital discharge. Predicted probability of survival to hospital discharge was calculated for each patient using the ACLS Score. The overall predicted and observed survival rates were compared using Flora's Z score. The Hosmer-Lemeshow test was used to evaluate the model's goodness-of-fit over a range of survival probabilities. RESULTS: Of 754 cardiac arrest patients enrolled in the study period, 575 (76%) patients had documentation that allowed scoring using the ACLS Score. Twenty-five (4%) patients survived to hospital discharge. The predicted number of survivors based on the ACLS Score was 104 (18%), yielding a Flora's Z statistic of -4.46 (p < 0.0001). After categorizing predicted survival probabilities into four categories, the resulting Hosmer-Lemeshow statistic was 210 (p < 10(-6)). Both goodness-of-fit statistics demonstrated extremely poor fit of the model. A receiver operating characteristic (ROC) curve was created, yielding an area under the ROC curve of 0.33 (95% CI = 0.19 to 0.47), signifying extremely poor discrimination. CONCLUSIONS: The previously published ACLS Score was not valid when applied to an external cohort of out-of-hospital cardiac arrest patients. An externally valid model is needed to predict survival to hospital discharge following out-of-hospital cardiac arrest.     

Severity scores in respiratory intensive care: APACHE II predicted mortality better than SAPS II

BACKGROUND: In recent years several scoring systems have been developed to describe the severity of illness, to establish the individual prognosis, and to group adult ICU patients by predicted risk of mortality. In addition, these scores can be used to measure and/or compare the quality of care in different ICUs. We compared the mortality predictions of the Acute Physiology and Chronic Health Evaluation (APACHE II) score and a new Simplified Acute Physiology Score (SAPS II) in patients with respiratory disease who require intensive care. PATIENTS & METHODS: We prospectively studied all 306 admissions from January 1, 1992 through December 31, 1994. McNemar and Hosmer-Lemeshow tests, and receiver operating characteristic (ROC) curves were used to describe and analyze our data. RESULTS: The average APACHE II score was 17.5 (SD 6.0), corresponding to a mean predicted death rate of 24.9% (SD 17.2%) as compared to an observed overall RICU mortality rate of 21.6%. The average SAPS II score was 39.1 (SD 11.1) corresponding to a mean predicted death rate of 26.0% (SD 18.4%). The ratio between the actual and predicted hospital mortality was 86% for APACHE II and 83% for SAPS II. Survivors had a significantly lower predicted risk of death than nonsurvivors (p < 0.0001) with both indices, and a higher Glasgow coma scale score (p < 0.0001). The ROC-curve analysis suggested the superior predictive ability of APACHE II in our patients. Area under the APACHE II ROC curve was 80.88% (standard error [SE] 2.89%), significantly larger (p < 0.01) than that found for SAPS II (73.52%, SE 3.61%). CONCLUSIONS: The APACHE II score was a good predictor of hospital outcome and better than SAPS II in our population.     

The predictive performance of the SAPS II and SAPS 3 scoring systems: A retrospective analysis

PurposeThe purpose was to analyze and compare the performance of Simplified Acute Physiology Score (SAPS) II and SAPS 3 (North Europe Logit) in an intensive care unit (ICU) for internal disorders at a German university hospital.Materials and methodsThis retrospective study was conducted at a single-center 12-bed ICU sector for Internal Medicine in Essen, Germany, within an 18-month period. Data for adult ICU patients (N = 548) were evaluated. SAPS II and SAPS 3 scores were assessed along with the predicted mortality rates. Discrimination was evaluated by calculating the area under the receiver operating characteristic curve, and calibration was evaluated using the Hosmer-Lemeshow goodness-of-fit C-test. The ratios of observed-to-expected deaths (standardized mortality ratio, SMR) were calculated along with the 95% confidence intervals (95% CIs).ResultsThe in-hospital mortality rate was 22.6%, which provided an SMR of 0.91 (95% CI, 0.77-0.99) for SAPS II and 0.62 (95% CI, 0.52-0.71) for SAPS 3. Both SAPS II and SAPS 3 exhibited acceptable discrimination, with an area under the receiver operating characteristic curve of 0.84 (95% CI, 0.79-0.89) and 0.73 (95% CI, 0.67-0.79), respectively. However, SAPS II demonstrated superior SMR-based discrimination, which was closer to the observed mortality rate, compared with SAPS 3. Calibration curves exhibited similar performance based on the Hosmer-Lemeshow goodness-of-fit C-test results: χ² = 7.10 with P = .525 for SAPS II and χ² = 3.10 with P = .876 for SAPS 3. Interestingly, both scores overpredicted mortality.ConclusionsIn this study, SAPS 3 overestimated mortality and therefore appears less suitable for risk evaluation in comparison to SAPS II.     

Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited

OBJECTIVE: To examine the Hosmer-Lemeshow test's sensitivity in evaluating the calibration of models predicting hospital mortality in large critical care populations. DESIGN: Simulation study. SETTING: Intensive care unit databases used for predictive modeling. PATIENTS: Data sets were simulated representing the approximate number of patients used in earlier versions of critical care predictive models (n = 5,000 and 10,000) and more recent predictive models (n = 50,000). Each patient had a hospital mortality probability generated as a function of 23 risk variables. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Data sets of 5,000, 10,000, and 50,000 patients were replicated 1,000 times. Logistic regression models were evaluated for each simulated data set. This process was initially carried out under conditions of perfect fit (observed mortality = predicted mortality; standardized mortality ratio = 1.000) and repeated with an observed mortality that differed slightly (0.4%) from predicted mortality. Under conditions of perfect fit, the Hosmer-Lemeshow test was not influenced by the number of patients in the data set. In situations where there was a slight deviation from perfect fit, the Hosmer-Lemeshow test was sensitive to sample size. For populations of 5,000 patients, 10% of the Hosmer-Lemeshow tests were significant at p < .05, whereas for 10,000 patients 34% of the Hosmer-Lemeshow tests were significant at p < .05. When the number of patients matched contemporary studies (i.e., 50,000 patients), the Hosmer-Lemeshow test was statistically significant in 100% of the models. CONCLUSIONS: Caution should be used in interpreting the calibration of predictive models developed using a smaller data set when applied to larger numbers of patients. A significant Hosmer-Lemeshow test does not necessarily mean that a predictive model is not useful or suspect. While decisions concerning a mortality model's suitability should include the Hosmer-Lemeshow test, additional information needs to be taken into consideration. This includes the overall number of patients, the observed and predicted probabilities within each decile, and adjunct measures of model calibration.     

Assessment of the performance of five intensive care scoring models within a large Scottish database

OBJECTIVE: To assess and compare the performance of five severity of illness scoring systems used commonly for intensive care unit (ICU) patients in the United Kingdom. The five models analyzed were versions II and III of the Acute Physiology and Chronic Health Evaluation (APACHE) system, a version of APACHE II using United Kingdom (UK)-derived coefficients (UK APACHE II), version II of the Simplified Acute Physiology Score (SAPS), and version II of the Mortality Probability Model, computed at admission (MPM0) and after 24 hrs in the ICU (MPM24). DESIGN: A 2-yr prospective cohort study of consecutive admissions to intensive care units. SETTING: A total of 22 general ICUs in Scotland PATIENTS: A total of 13,291 admissions to the study, which after prospectively agreed exclusions left a total of 10,393 patients for the analysis. OUTCOME MEASURES: Death or survival at hospital discharge. MEASUREMENTS AND MAIN RESULTS: All the models showed reasonable discrimination using the area under the receiver operating characteristic curve (APACHE III, 0.845; APACHE II, 0.805; UKAPACHE II, 0.809; SAPS II, 0.843; MPM0, 0.785; MPM24, 0.799). The levels of observed mortality were significantly different than that predicted by all models, using the Hosmer-Lemeshow goodness-of-fit C test (p < .001), with the results of the test being confirmed by calibration curves. When excluding patients discharged in the first 24 hrs to allow for comparisons using the same patient group, APACHE III, MPM24, and SAPS II (APACHE III, 0.795; MPM24, 0.791; SAPS II, 0.784) showed significantly better discrimination than APACHE II, UK APACHE II, and MPM0 (APACHE II, 0.763; UK APACHE II, 0.756; MPM0, 0.741). However, calibration changed little for all models with observed mortality still significantly different from that predicted by the scoring systems (p < .001). For equivalent data sets, APACHE II demonstrated superior calibration to all the models using the chi-squared value from the Hosmer-Lemeshow test for both populations (APACHE III, 366; APACHE II, 67; UKAPACHE II, 237; SAPS II, 142; MPM0, 452; MPM24, 101). CONCLUSIONS: SAPS II demonstrated the best overall performance, but the superior calibration of APACHE II makes it the most appropriate model for comparisons of mortality rates in different ICUs. The significance of the Hosmer-Lemeshow C test in all the models suggest that new logistic regression coefficients should be generated and the systems retested before they could be used with confidence in Scottish ICUs.     

Identifying quality outliers in a large,multiple-institution database by using customized versions of the Simplified Acute Physiology Score II and the Mortality Probability Model II0

OBJECTIVE: To assess whether customized versions of the Simplified Acute Physiology Score (SAPS) II and the Mortality Probability Model (MPM) II0 agree on the identity of intensive care unit quality outliers within a multiple-center database. DESIGN: Retrospective database analysis. SETTING AND PATIENTS: Patient subset of the Project IMPACT database consisting of 39,617 adult patients admitted to surgical, medical, and mixed surgical-medical intensive care units at 54 hospitals between 1995 and 1999 who met inclusion criteria for SAPS II and MPM II0. INTERVENTIONS: Customized versions of SAPS II and MPM II0 were obtained by fitting new logistic regressions to the data by using the risk score as the independent variable and outcome at hospital discharge as the dependent variable. The data set was divided randomly into a training set and a validation set. Each model was customized by using the training set; model performance was then assessed in the validation set by using the area under the receiver operating characteristic curve and the Hosmer-Lemeshow statistic. The final models were based on the entire data set. The level of agreement between the customized models on the identity of quality outliers was evaluated by using kappa analysis. MEASUREMENTS AND MAIN RESULTS: Both customized models exhibited good discrimination and good calibration in this database. The area under the receiver operating characteristic curve was 0.83 for MPM II0 and 0.872 for SAPS II following model customization. The Hosmer-Lemeshow statistic was 12.3 ( >.14) for MPM II0, and 8.17 (p >.42) for SAPS II, after customization. Kappa analysis showed only fair agreement between the two customized models with regard to the identity of the quality outliers: kappa = 0.44 (95% confidence interval, 0.24, 0.65). CONCLUSIONS: Customization of SAPS II and MPM II0 to the Project IMPACT database resulted in well-calibrated models. Despite this, the models exhibited only a moderate level of agreement in which hospitals were designated as quality outliers. Seventeen of the 54 hospitals were categorized differently depending on which of the two scoring systems was used. Therefore, the rating of quality of care appears, in part, to be a function of the prediction model used.     

Scoring systems in pediatric intensive care: PRISM III versus PIM

OBJECTIVE: To compare the performance of two different clinical scoring systems that were developed to assess mortality probability in pediatric intensive care. DESIGN AND METHODS: Prospective cohort study in a multidisciplinary tertiary pediatric intensive care unit. The Pediatric Risk of Mortality score (PRISM III) and the Pediatric Index of Mortality (PIM) were collected for each patient. Standardized mortality rate (SMR), discrimination and calibration of both scoring systems were compared by goodness-of-fit tests and receiver operating characteristic (ROC) curves. RESULTS: Data from 303 patients were collected over a 9-month period. Twenty patients (6.6%) died in the PICU. Expected mortality based on PRISM III (12 h) was 6.96% (SMR 0.95; 95% CI 0.68-1.23), based on PRISM III (24 h) was 6.95% (SMR 0.95; 0.67-1.22) and based on PIM was 7.5% (SMR 0.88; 0.55-1.20). Calibration by Hosmer-Lemeshow goodness-of-fit test showed for PRISM III (12 h) chi(2) (8) =10.8, p=0.21; for PRISM III (24 h) chi(2) (8) =13.3, p=0.21 and for the PIM score chi(2) (8) = 4.92, p=0.77. Discriminatory performance assessed by ROC curves showed an area under the curve of 0.78 (95% CI 0.67-0.89) for the PRISM III score both after 12 and 24 h and 0.74 (0.63-0.85) for the PIM score. CONCLUSION: PRISM III and PIM scores are both adequate indicators of mortality probability for heterogeneous patient groups in pediatric intensive care. Possibly in larger studies (equivalence trial) a significant and relevant difference between these scores would be demonstrated.     

A multicenter review of toxic epidermal necrolysis treated in U.S. burn centers at the end of the twentieth century

Toxic epidermal necrolysis (TEN) is a potentially fatal disorder that involves large areas of skin desquamation. Patients with TEN are often referred to burn centers for expert wound management and comprehensive care. The purpose of this study was to define the presenting characteristics and treatment of TEN before and after admission to regional burn centers and to evaluate the efficacy of burn center treatment for this disorder. A retrospective multicenter chart review was completed for patients admitted with TEN to 15 burn centers from 1995 to 2000. Charts were reviewed for patient characteristics, non-burn hospital and burn center treatment, and outcome. A total of 199 patients were admitted. Patients had a mean age of 47 years, mean 67.7% total body surface area skin slough, and mean Acute Physiology and Chronic Health Evaluation (APACHE II) score of 10. Sixty-four patients died, for a mortality rate of 32%. Mortality increased to 51% for patients transferred to a burn center more than one week after onset of disease. Burn centers and non-burn hospitals differed in their use of enteral nutrition (70 vs 12%, respectively, P < 0.05), prophylactic antibiotics (22 vs 37.9%, P < 0.05), corticosteroid use (22 vs 51%, P < 0.05), and wound management. Age, body surface area involvement, APACHE II score, complications, and parenteral nutrition before transfer correlated with increased mortality. The treatment of TEN differs markedly between burn centers and non-burn centers. Early transport to a burn unit is warranted to improve patient outcome.     

Intensive care unit support and Acute Physiology and Chronic Health Evaluation III performance in hematopoietic stem cell transplant recipients

OBJECTIVE: Hematopoietic stem cell transplant (HSCT) recipients admitted to the intensive care unit (ICU) have high mortality. The prognostic importance of peripheral blood stem cell source in critically ill HSCT recipients and the performance of Acute Physiology and Chronic Health Evaluation (APACHE) III have not been well studied. In a previous study, the hospital mortality rate of HSCT recipients admitted to our ICU was 77%. The objectives of this study were to describe the clinical course of HSCT recipients admitted to the ICU and to determine the performance of APACHE III in predicting their mortality. DESIGN: Retrospective cohort study. SETTING: Academic medical center. PATIENTS: HSCT recipients admitted to the ICU. MEASUREMENTS: Demographics, transplant type, stem cell source, APACHE II and III predicted mortality, development of sepsis and organ failure, use of mechanical ventilation, duration of hospital stay, and mortality. RESULTS: Ninety-four percent of the 112 HSCT recipients were white and 64% male. The mean APACHE II and III scores were 25 and 44, respectively. The APACHE II and III hospital predicted mortality rates were 44% and 42%, respectively. Mechanical ventilation was provided to 63%. Organ failure developed in 94% and sepsis in 62%. The ICU, hospital, and 30-day mortality rates were 33%, 46%, and 52%, respectively. Allogeneic transplant and higher APACHE III scores, but not bone marrow stem cell source, were associated with increased mortality. Invasive mechanical ventilation, vasoactive medication use, sepsis, and organ failure during patients' ICU course were also associated with increased mortality. The area under the receiver operating characteristic curve for APACHE III hospital mortality prediction was 0.704 (95% confidence interval, 0.610-0.786). For APACHE III hospital mortality prediction, the value of the Hosmer-Lemeshow statistic showed good model fit. CONCLUSIONS: Current mortality figures of HSCT recipients admitted to the ICU are better than previously reported. The APACHE III prognostic model has moderate discrimination and good calibration in predicting hospital mortality in these patients.     

Performance assessment of the Simplified Acute Physiology Score II,the Acute Physiology and Chronic Health Evaluation II score,and the Sequential Organ Failure Assessment score in predicting the outcomes of adult patients with hepatic portal venous gas in the ED

ObjectiveThis study aims to evaluate the performance of Simplified Acute Physiology Score II (SAPS II), the Acute Physiology and Chronic Health Evaluation II (APACHE II) score, and the Sequential Organ Failure Assessment (SOFA) score for predicting illness severity and the mortality of adult hepatic portal venous gas (HPVG) patients presenting to the emergency department (ED). This will assist emergency physicians in risk stratification.MethodsData for 48 adult HPVG patients who visited our ED between December 2009 and December 2013 were analyzed. The SAPS II, APACHE II score, and SOFA score were calculated based on the worst laboratory values in the ED. The probability of death was calculated for each patient based on these scores. The ability of the SAPS II, APACHE II score, and SOFA score to predict group mortality was assessed by using receiver operating characteristic curve analysis and calibration analysis.ResultsThe sensitivity, specificity, and accuracy were 92.6%,71.4%, and 83.3%, respectively, for the SAPS II method; 77.8%, 81%, and 79.2%, respectively, for the APACHE II scoring system, and 77.8%, 76.2%, and 79.2%, respectively, for the SOFA score. In the receiver operating characteristic curve analysis, the areas under the curve for the SAPS II, APACHE II scoring system, and SOFA score were 0.910, 0.878, and 0.809, respectively.ConclusionThis is one of the largest series performed in a population of adult HPVG patients in the ED. The results from the present study showed that SAPS II is easier and more quickly calculated than the APACHE II and more superior in predicting the mortality of ED adult HPVG patients than the SOFA. We recommend that the SAPS II be used for outcome prediction and risk stratification in adult HPVG patients in the ED.     

Should the pre-sedation Glasgow Coma Scale value be used when calculating Acute Physiology and Chronic Health Evaluation scores for sedated patients? Scottish Intensive Care Society Audit Group

OBJECTIVE: To assess the effect on the performance of Acute Physiology and Chronic Health Evaluation (APACHE) II and APACHE III of two different approaches to scoring the Glasgow Coma Scale (GCS) in sedated patients. The first approach was to assume that the GCS score was normal, and the second was to use the GCS value recorded before the patient was sedated. DESIGN: Prospective cohort study over 2 yrs. SETTING: Twenty-two general adult intensive care units in Scotland. PATIENTS: 13,291 consecutive admissions to the participating intensive care units. MEASUREMENTS AND MAIN RESULTS: After exclusion of patients according to standard, predefined criteria, the Acute Physiology and Chronic Health Evaluation II and III systems were used to calculate the probability of hospital mortality for patients included in the study. In patients whose GCS scores could not be assessed accurately during the first 24 hrs, the APACHE II and III predictions were calculated twice: first, assuming that the GCS score was normal; and second, substituting the GCS score recorded before sedation. This generated two different databases for each system, and the predictions for both were compared with the observed hospital mortality rate. The effect of the two different approaches to the GCS on the performance of both APACHE II and APACHE III was assessed using measures of discrimination (area under the receiver operating characteristic curve) and goodness of fit (calibration curves and the Hosmer-Lemeshow statistic). Analysis was undertaken for both the entire cohort and for the group of patients whose APACHE scores were altered. There was a wide variation in the number of patients who had their scores altered between participating units. There were also differences between diagnostic groups. Overall, however, 50% of the patients were sedated and 22% had their scores altered. Using the presedation GCS score increased the discrimination of both APACHE II and APACHE III. The calibration of APACHE III was also improved but that of APACHE II deteriorated. The calibration improved, however, in those patients with altered scores, suggesting that the overall deterioration is attributable to other limitations in the fit of the model to these data. Although changes had the greatest effect in patients with a neurologic or trauma diagnosis, the changes were important in most diagnostic groups. CONCLUSIONS: The GCS is an important component of both APACHE II and APACHE III. It should be assessed directly whenever possible. When patients are sedated, using the GCS score recorded before sedation is preferable to the assumption of normality. The variations between different units and different diagnostic groups highlight the possible effects of case mix on the performance of prognostic scoring systems.     

Application of mortality prediction systems to individual intensive care units

Objective: To evaluate the predictive accuracy of the severity of illness scoring systems in a single institution. Design: A prospective study conducted by collecting data on consecutive patients admitted to the medical intensive care unit over 20 months. Surgical and coronary care admissions were excluded. Setting: Veterans Affairs Medical Center at Buffalo, New York. Patients and participants: Data collected on 302 unique, consecutive patients admitted to the medical intensive care unit. Interventions: None. Measurements and results: Data required to calculate the patients' predicted mortality by the Mortality Probability Model (MPM) II, Acute Physiology and Chronic Health Evaluation (APACHE) II and Simplified Acute Physiology Score (SAPS) II scoring systems were collected. The probability of mortality for the cohort of patients was analyzed using confidence interval analyses, receiver operator characteristic (ROC) curves, two by two contingency tables and the Lemeshow–Hosmer chi-square statistic. Predicted mortality for all three scoring systems lay within the 95 % confidence interval for actual mortality. For the MPM II, SAPS II and APACHE II, the c-index (equivalent to the area under the ROC curve) was 0.695 ± 0.0307 SE, 0.702 ± 0.063 SE and 0.672 ± 0.0306 SE, respectively, which were not statistically different from each other but were lower than values obtained in previous studies. Conclusion: Although the overall mortality was consistent with the predicted mortality, the poor fit of the data to the model impairs the validity of the result. The observed outcoume could be due to erratic quality of care, or differences between the study population and the patient population in the original studies. The data cannot be used to distinguish between these possibilities. To increase predictive accuracy when studying individual intensive care units and enhance quality of care assessments it may be necessary to adapt the model to the patient population. Received: 21 December 1998 Final revision received: 26 May 1999 Accepted: 1 June 1999