Polytomous logistic regression analysis could be applied more often in diagnostic research

OBJECTIVE: Physicians commonly consider the presence of all differential diagnoses simultaneously. Polytomous logistic regression modeling allows for simultaneous estimation of the probability of multiple diagnoses. We discuss and (empirically) illustrate the value of this method for diagnostic research. STUDY DESIGN AND SETTING: We used data from a study on the diagnosis of residual retroperitoneal mass histology in patients presenting with nonseminomatous testicular germ cell tumor. The differential diagnoses include benign tissue, mature teratoma, and viable cancer. Probabilities of each diagnosis were estimated with a polytomous logistic regression model and compared with the probabilities estimated from two consecutive dichotomous logistic regression models. RESULTS: We provide interpretations of the odds ratios derived from the polytomous regression model and present a simple score chart to facilitate calculation of predicted probabilities from the polytomous model. For both modeling methods, we show the calibration plots and receiver operating characteristics curve (ROC) areas comparing each diagnostic outcome category with the other two. The ROC areas for benign tissue, mature teratoma, and viable cancer were similar for both modeling methods, 0.83 (95% confidence interval [CI]=0.80-0.85) vs. 0.83 (95% CI=0.80-0.85), 0.78 (95% CI=0.75-0.81) vs. 0.78 (95% CI=0.75-0.81), and 0.66 (95% CI=0.61-0.71) vs. 0.64 (95% CI=0.59-0.69), for polytomous and dichotomous regression models, respectively. CONCLUSION: Polytomous logistic regression is a useful technique to simultaneously model predicted probabilities of multiple diagnostic outcome categories. The performance of a polytomous prediction model can be assessed similarly to a dichotomous logistic regression model, and predictions by a polytomous model can be made with a user-friendly method. Because the simultaneous consideration of the presence of multiple (differential) conditions serves clinical practice better than consideration of the presence of only one target condition, polytomous logistic regression could be applied more often in diagnostic research.     

Assessing the discriminative ability of risk models for more than two outcome categories

The discriminative ability of risk models for dichotomous outcomes is often evaluated with the concordance index (c-index). However, many medical prediction problems are polytomous, meaning that more than two outcome categories need to be predicted. Unfortunately such problems are often dichotomized in prediction research. We present a perspective on the evaluation of discriminative ability of polytomous risk models, which may instigate researchers to consider polytomous prediction models more often. First, we suggest a “discrimination plot” as a tool to visualize the model’s discriminative ability. Second, we discuss the use of one overall polytomous c-index versus a set of dichotomous measures to summarize the performance of the model. Third, we address several aspects to consider when constructing a polytomous c-index. These involve the assessment of concordance in pairs versus sets of patients, weighting by outcome prevalence, the value related to models with random performance, the reduction to the dichotomous c-index for dichotomous problems, and interpretation. We illustrate these issues on case studies dealing with ovarian cancer (four outcome categories) and testicular cancer (three categories). We recommend the use of a discrimination plot together with an overall c-index such as the Polytomous Discrimination Index. If the overall c-index suggests that the model has relevant discriminative ability, pairwise c-indexes for each pair of outcome categories are informative. For pairwise c-indexes we recommend the ‘conditional-risk’ method which is consistent with the analytical approach of the multinomial logistic regression used to develop polytomous risk models.     

Dichotomous or categorical response? Analysing self-rated health and lifetime social class

BACKGROUND: Self-rated health is a commonly used measure of health status, usually having three to five categories. The measure is often collapsed into a dichotomous variable of good versus less than good health. This categorization has not yet been justified. METHODS: Using data from the 1958 British birth cohort, we examined the relationship between socioeconomic conditions, indicated by occupational class at four ages, and self-rated health. Results obtained for a dichotomous variable using logistic regression were compared with alternative methods for ordered categorical variables including polytomous regression, cumulative odds, continuation ratio and adjacent categories models. RESULTS AND CONCLUSIONS: Findings concerning the relationship between socioeconomic position and self-rated health yielded by a logistic regression model were confirmed by alternative statistical methods which incorporate the ordered nature of self-rated health. Similarity of results was found regarding size and significance of main effects, type of association and interactive effects.     

Simple dichotomous updating methods improved the validity of polytomous prediction models

ObjectivesPrediction models may perform poorly in a new setting. We aimed to determine which model updating methods should be applied for models predicting polytomous outcomes, which often suffer from one or more categories with low prevalence.Study Design and SettingWe used case studies on testicular and ovarian tumors. The original regression models were based on 502 and 2,037 patients and validated on 273 and 1,107 patients, respectively. The polytomous models combined dichotomous models for category A vs. B + C and B vs. C (sequential dichotomous modeling). Simple recalibration, revision, and redevelopment methods were considered. To assess discrimination (using dichotomous and polytomous c-statistics) and calibration (by comparing observed and expected prevalences) of these methods, the validation data were divided into updating and test parts. Five hundred such divisions were randomly generated, and the average test set results reported.ResultsNone of the updating methods could improve discrimination of the original models, but recalibration, revision, and redevelopment strongly improved calibration. Redevelopment was unstable with respect to overfitting and performance.ConclusionSimple dichotomous updating methods behaved well when applied to polytomous models. Our results suggest that recalibration is preferred, but larger validation sets may make revision or redevelopment a sensible alternative.     

Assessing calibration of multinomial risk prediction models

Calibration, that is, whether observed outcomes agree with predicted risks, is important when evaluating risk prediction models. For dichotomous outcomes, several tools exist to assess different aspects of model calibration, such as calibration‐in‐the‐large, logistic recalibration, and (non‐)parametric calibration plots. We aim to extend these tools to prediction models for polytomous outcomes. We focus on models developed using multinomial logistic regression (MLR): outcome Y with k categories is predicted using k ? 1 equations comparing each category i (i = 2, … ,k) with reference category 1 using a set of predictors, resulting in k ? 1 linear predictors. We propose a multinomial logistic recalibration framework that involves an MLR fit where Y is predicted using the k ? 1 linear predictors from the prediction model. A non‐parametric alternative may use vector splines for the effects of the linear predictors. The parametric and non‐parametric frameworks can be used to generate multinomial calibration plots. Further, the parametric framework can be used for the estimation and statistical testing of calibration intercepts and slopes. Two illustrative case studies are presented, one on the diagnosis of malignancy of ovarian tumors and one on residual mass diagnosis in testicular cancer patients treated with cisplatin‐based chemotherapy. The risk prediction models were developed on data from 2037 and 544 patients and externally validated on 1107 and 550 patients, respectively. We conclude that calibration tools can be extended to polytomous outcomes. The polytomous calibration plots are particularly informative through the visual summary of the calibration performance. Copyright © 2014 John Wiley & Sons, Ltd.     

Vital signs should be maintained as continuous variables when predicting bacterial infections in febrile children

ObjectiveTo determine how vital signs such as heart and respiratory rates should be included in prediction models for serious bacterial infections (SBIs) in febrile children.Study Design and SettingProspective observational study of 1,750 febrile children aged <16 years, visiting the emergency department of a university hospital; of them 13% (n = 222) had SBI. Common age-specific thresholds of heart and respiratory rates were used to define tachycardia and tachypnea. We compared seven strategies to handle vital signs as predictors of SBI (dichotomized or continuously in various ways).ResultsThe dichotomous predictors, namely tachycardia and tachypnea, containing information on the vital sign and age showed limited value to predict the presence of SBI (area under the receiver operating characteristic curve [AUC (ROC)]: 0.53 for heart rate and 0.55 for respiratory rate). In comparison, a model with age as a single continuous predictor resulted in an AUC of 0.58. Models with age and one of the vital signs included continuously showed the highest AUC (heart rate: 0.60 and respiratory rate: 0.63).ConclusionHeart and respiratory rates should be maintained as continuous variables in model development to predict SBI in febrile children, as dichotomization results in information loss and lower predictive ability.     

Absolute risk reductions,relative risks,relative risk reductions,and numbers needed to treat can be obtained from a logistic regression model

ObjectiveLogistic regression models are frequently used in cohort studies to determine the association between treatment and dichotomous outcomes in the presence of confounding variables. In a logistic regression model, the association between exposure and outcome is measured using the odds ratio (OR). The OR can be difficult to interpret and only approximates the relative risk (RR) in certain restrictive settings. Several authors have suggested that for dichotomous outcomes, RRs, RR reductions, absolute risk reductions, and the number needed to treat (NNT) are more clinically meaningful measures of treatment effect.Study Design and SettingWe describe a method for deriving clinically meaningful measures of treatment effect from a logistic regression model. This method involves determining the probability of the outcome if each subject in the cohort was treated and if each subject was untreated. These probabilities are then averaged across the study cohort to determine the average probability of the outcome in the population if all subjects were treated and if they were untreated.ResultsRisk differences, RRs, and NNTs were derived using a logistic regression model.ConclusionsClinically meaningful measures of effect can be derived from a logistic regression model in a cohort study. These methods can also be used in randomized controlled trials when logistic regression is used to adjust for possible imbalance in prognostically important baseline covariates.     

Using logistic regression in perinatal epidemiology: an introduction for clinical researchers. Part 2: The logistic regression equation

In Part 1 basic concepts were introduced as a preparation for an introductory explanation of logistic regression. Logistic regression is a statistical modelling technique, designed for the estimation of the simultaneous effects of predictors on the risk of a certain dichotomous outcome variable where each effect is estimated while adjusting for the effect of the other factors considered. The basic concepts--odds, odds ratio, confounding and interaction--were introduced in such a way that they naturally lead to the concept of logistic regression. In Part 2 the concepts are 'translated' into simple equations. By studying these equations the equivalence between such mathematical expressions and the underlying clinical assessment of risk will become clear.     

Using logistic regression in perinatal epidemiology: an introduction for clinical researchers. Part 2: the logistic regression equation

Summary. In Part 1 basic concepts were introduced as a preparation for an introductory explanation of logistic regression. Logistic regression is a statistical modelling technique, designed for the estimation of the simultaneous effects of predictors on the risk of a certain dichotomous outcome variable where each effect is estimated while adjusting for the effect of the other factors considered. The basic concepts - odds, odds ratio, confounding and interaction - were introduced in such a way that they naturally lead to the concept of logistic regression. In Part 2 the concepts are translated into simple equations. By studying these equations the equivalence between such mathematical expressions and the underlying clinical assessment of risk will become clear.     

Binary regression with continuous outcomes

Clinical research often involves continuous outcome measures, such as blood cholesterol, that are amenable to statistical techniques of analysis based on the mean, such as the t-test or multiple linear regression. Clinical interest, however, frequently focuses on the proportion of subjects who fall below or above a clinically relevant cut-off value, as a measure of the risk of disease. The customary approach to analyse such data is to dichotomize the continuous outcome measure and use statistical techniques based on binary data and the binomial distribution. In this paper, we use a parametric approach and the framework of generalized linear models to fit various regression models, including the logistic, on the basis of the original continuous outcome. We consider the Gaussian and the three-parameter log-normal distributions for the continuous outcome, assessing both precision and bias under various conditions. In simulation analyses, we find that we are unable to fit some of the samples with the ‘dichotomous’ approach, but we can with the ‘continuous’ approach, and that the latter yields estimates between 25 and 85 per cent more efficient than the former. We illustrate the method, programmed using GLIM macros, with data from clinical studies of the risk of hypoxaemia during open thoracic surgery and the risk of nocturnal hypoglycaemia among diabetic children.     

Comparison of prediction models for adverse outcome in pediatric meningococcal disease using artificial neural network and logistic regression analyses

The objective of this study was to compare artificial neural network (ANN) and multivariable logistic regression analyses for prediction modeling of adverse outcome in pediatric meningococcal disease. We analyzed a previously constructed database of children younger than 20 years of age with meningococcal disease at four pediatric referral hospitals from 1985-1996. Patients were randomly divided into derivation and validation datasets. Adverse outcome was defined as death or limb amputation. ANN and multivariable logistic regression models were developed using the derivation set, and were tested on the validation set. Eight variables associated with adverse outcome in previous studies of meningococcal disease were considered in both the ANN and logistic regression analyses. Accuracies of these models were then compared. There were 381 patients with meningococcal disease in the database, of whom 50 had adverse outcomes. When applied to the validation data set, the sensitivities for both the ANN and logistic regressions models were 75% and the specificities were both 91%. There were no significant differences in any of the performance parameters between the two models. ANN analysis is an effective tool for developing prediction models for adverse outcome of meningococcal disease in children, and has similar accuracy as logistic regression modeling. With larger, more complete databases, and with advanced ANN algorithms, this technology may become increasingly useful for real-time prediction of patient outcome.     

Estimating interaction on an additive scale between continuous determinants in a logistic regression model

BACKGROUND: To determine the presence of interaction in epidemiologic research, typically a product term is added to the regression model. In linear regression, the regression coefficient of the product term reflects interaction as departure from additivity. However, in logistic regression it refers to interaction as departure from multiplicativity. Rothman has argued that interaction estimated as departure from additivity better reflects biologic interaction. So far, literature on estimating interaction on an additive scale using logistic regression only focused on dichotomous determinants. The objective of the present study was to provide the methods to estimate interaction between continuous determinants and to illustrate these methods with a clinical example. METHODS: and results From the existing literature we derived the formulas to quantify interaction as departure from additivity between one continuous and one dichotomous determinant and between two continuous determinants using logistic regression. Bootstrapping was used to calculate the corresponding confidence intervals. To illustrate the theory with an empirical example, data from the Utrecht Health Project were used, with age and body mass index as risk factors for elevated diastolic blood pressure. CONCLUSIONS: The methods and formulas presented in this article are intended to assist epidemiologists to calculate interaction on an additive scale between two variables on a certain outcome. The proposed methods are included in a spreadsheet which is freely available at: http://www.juliuscenter.nl/additive-interaction.xls.     

Risk assessment for case-control subgroups by polychotomous logistic regression

Case-control studies involving more than two disease and referent categories may be analyzed by means of polychotomous logistic regression, an extension of the usual dichotomous logistic regression model. Although the standard method still may be used to compare the several disease subgroups in pairs, the polychotomous approach is advantageous in that it allows simultaneous estimation of the disease-specific parameters and direct hypothesis testing involving multiple disease categories. This is especially useful for assessing whether different disease types have different risk factors. The method is applied to a large case-control study of breast cancer involving three disease categories for which both categoric and continuous risk factors are considered. Substantive epidemiologic interpretation of polychotomous regression outputs is emphasized, as well as providing illustration of the practical aspects of the statistical method.     

Maternal occupational exposure to chemical substances and the risk of infants small-for-gestational-age.

BACKGROUND: The association between maternal occupational exposure to specific chemical substances (organic solvents, carbon tetrachloride, herbicides, chlorophenols, polychlorinated biphenyls, aromatic amines, lead and lead compounds, mercury and mercury compounds) and birth of small-for-gestational-age (SGA) infants was evaluated using data from a prospective cohort study of 3,946 pregnant women in West Germany from 1987 to 1988. METHODS: Occupational, medical, and psychosocial information was gathered through a questionnaire from pregnant women who were recruited between 15 and 28 gestational weeks. Exposure to chemical substances at the current workplace was assessed by a job-exposure matrix constructed by Pannett in 1985 and weighted for the number of working hours per week. Women not working at the time of the interview, women with multiple births, and women with stillbirths were excluded from analysis. Data were analyzed using dichotomous and polytomous logistic regression to control for age, smoking status, alcohol consumption, body mass index, and number of former births. RESULTS: The results of the dichotomous logistic regression analysis suggest that leather work might be associated with the birth of infants small-for-gestational-age through exposure to chlorophenols (P = 0.02) and aromatic amines (P = 0.05). In the polytomous logistic regression analysis, only the association between exposure to mercury and growth retardation reached statistical significance (P = 0.02); however, the power of the study is limited. Further adjustment for income, shift work, and heavy physical work had no substantial effect on the results. CONCLUSIONS: These findings suggest that maternal exposure to specific chemicals at work may be a risk factor for the birth of SGA infants.     

Standardized estimates from categorical regression models

We consider the problem of interpreting categorical regression models, such as the polytomous logistic model, the continuation-ratio model, the stereotype model, and the cumulative-odds model. We present a method to convert categorical regression coefficients into estimates of standardized fitted probabilities, probability differences and probability ratios. We use a delta-method approach to estimate standard errors. We then present a small simulation study to compare different transforms for setting confidence limits, and provide an illustration of our approach in an observational study of drug therapy of polymyositis.     

A note on R2 measures for Poisson and logistic regression models when both models are applicable

The aim of many epidemiological studies is the regression of a dichotomous outcome (e.g., death or affection by a certain disease) on prognostic covariables. Thereby the Poisson regression model is often used alternatively to the logistic regression model. Modelling the number of events and individual outcomes, respectively, both models lead to nearly the same results concerning the parameter estimates and their significances. However, when calculating the proportion of explained variation, quantified by an R2 measure, a large difference between both models usually occurs. We illustrate this difference by an example and explain it with theoretical arguments. We conclude, the R2 measure of the Poisson regression quantifies the predictability of event rates, but it is not adequate to quantify the predictability of the outcome of individual observations.     

Predictive factors for pneumonia onset after cardiac surgery in Rio de Janeiro, Brazil.

OBJECTIVE: To develop a predictive system for the occurrence of nosocomial pneumonia in patients who had cardiac surgery performed. DESIGN: Retrospective cohort study.Setting. Two cardiologic tertiary care hospitals in Rio de Janeiro, Brazil. PATIENTS: Between June 2000 and August 2002, there were 1,158 consecutive patients who had complex heart surgery performed. Patients older than 18 years who survived the first 48 postoperative hours were included in the study. The occurrence of pneumonia was diagnosed through active surveillance by an infectious diseases specialist according to the following criteria: the presence of new infiltrate on a radiograph in association with purulent sputum and either fever or leukocytosis until day 10 after cardiac surgery. Predictive models were built on the basis of logistic regression analysis and classification and regression tree (CART) analysis. The original data set was divided randomly into 2 parts, one used to construct the models (ie, "test sample") and the other used for validation (ie, "validation sample"). RESULTS: The area under the receiver-operating characteristic (ROC) curve was 69% for the logistic regression model and 76% for the CART model. Considering a probability greater than 7% to be predictive of pneumonia for both models, sensitivity was higher for the logistic regression models, compared with the CART models (64% vs 56%). However, the CART models had a higher specificity (92% vs 70%) and global accuracy (90% vs 70%) than the logistic regression models. Both models showed good performance, based on the 2-graph ROC, considering that 84.6% and 84.3% of the predictions obtained by regression and CART analyses were regarded as valid. CONCLUSION: Although our findings are preliminary, the predictive models we created showed fairly good specificity and fair sensitivity.     

Ordinal regression model and the linear regression model were superior to the logistic regression models

OBJECTIVE: Ordinal scales often generate scores with skewed data distributions. The optimal method of analyzing such data is not entirely clear. The objective was to compare four statistical multivariable strategies for analyzing skewed health-related quality of life (HRQOL) outcome data. HRQOL data were collected at 1 year following catheterization using the Seattle Angina Questionnaire (SAQ), a disease-specific quality of life and symptom rating scale. STUDY DESIGN AND SETTING: In this methodological study, four regression models were constructed. The first model used linear regression. The second and third models used logistic regression with two different cutpoints and the fourth model used ordinal regression. To compare the results of these four models, odds ratios, 95% confidence intervals, and 95% confidence interval widths (i.e., ratios of upper to lower confidence interval endpoints) were assessed. RESULTS: Relative to the two logistic regression analysis, the linear regression model and the ordinal regression model produced more stable parameter estimates with smaller confidence interval widths. CONCLUSION: A combination of analysis results from both of these models (adjusted SAQ scores and odds ratios) provides the most comprehensive interpretation of the data.     

Efficiency of the logistic regression and Cox proportional hazards models in longitudinal studies

Both logistic regression and Cox proportional hazards models are used widely in longitudinal epidemiologic studies for analysing the relationship between several risk factors and a time-related dichotomous event. The two models yield similar estimates of regression coefficients in studies with short follow-up and low incidence of event occurrence. Further, with just one dichotomous covariate and identical censoring times for all subjects, the asymptotic relative efficiency of the two models is very close to 1 unless the duration of follow-up is extended. We generalize this result to several qualitative or quantitative covariates. This was motivated by the analysis of mortality data from a study where all subjects are followed up during the same fixed period without loss except by death. Logistic and Cox models were applied to these data. Similar results were obtained for the two models in shorter periods of follow-up of five years or less, but not in longer periods of ten years or more, where the survival rate was lower.     

A comparison of regression trees, logistic regression, generalized additive models, and multivariate adaptive regression splines for predicting AMI mortality

Clinicians and health service researchers are frequently interested in predicting patient-specific probabilities of adverse events (e.g. death, disease recurrence, post-operative complications, hospital readmission). There is an increasing interest in the use of classification and regression trees (CART) for predicting outcomes in clinical studies. We compared the predictive accuracy of logistic regression with that of regression trees for predicting mortality after hospitalization with an acute myocardial infarction (AMI). We also examined the predictive ability of two other types of data-driven models: generalized additive models (GAMs) and multivariate adaptive regression splines (MARS). We used data on 9484 patients admitted to hospital with an AMI in Ontario. We used repeated split-sample validation: the data were randomly divided into derivation and validation samples. Predictive models were estimated using the derivation sample and the predictive accuracy of the resultant model was assessed using the area under the receiver operating characteristic (ROC) curve in the validation sample. This process was repeated 1000 times-the initial data set was randomly divided into derivation and validation samples 1000 times, and the predictive accuracy of each method was assessed each time. The mean ROC curve area for the regression tree models in the 1000 derivation samples was 0.762, while the mean ROC curve area of a simple logistic regression model was 0.845. The mean ROC curve areas for the other methods ranged from a low of 0.831 to a high of 0.851. Our study shows that regression trees do not perform as well as logistic regression for predicting mortality following AMI. However, the logistic regression model had performance comparable to that of more flexible, data-driven models such as GAMs and MARS.