Polytomous regression did not outperform dichotomous logistic regression in diagnosing serious bacterial infections in febrile children

OBJECTIVE: To compare polytomous and dichotomous logistic regression analyses in diagnosing serious bacterial infections (SBIs) in children with fever without apparent source (FWS). STUDY DESIGN AND SETTING: We analyzed data of 595 children aged 1-36 months, who attended the emergency department with fever without source. Outcome categories were SBI, subdivided in pneumonia and other-SBI (OSBI), and non-SBI. Potential predictors were selected based on previous studies and literature. Four models were developed: a polytomous model, estimating probabilities for three diagnostic categories simultaneously; two sequential dichotomous models, which differed in variable selection, discriminating SBI and non-SBI in step 1, and pneumonia and OSBI in step 2; and model 4, where each outcome category was opposed to the other two. The models were compared with respect to the area under the receiver-operating characteristic curve (AUC) for each of the three outcome categories and to the variable selection. RESULTS: Small differences were found in the variables that were selected in the polytomous and dichotomous models. The AUCs of the three outcome categories were similar for each modeling strategy. CONCLUSION: A polytomous logistic regression analysis did not outperform sequential and single application of dichotomous logistic regression analyses in diagnosing SBIs in children with FWS.     

Simple dichotomous updating methods improved the validity of polytomous prediction models

ObjectivesPrediction models may perform poorly in a new setting. We aimed to determine which model updating methods should be applied for models predicting polytomous outcomes, which often suffer from one or more categories with low prevalence.Study Design and SettingWe used case studies on testicular and ovarian tumors. The original regression models were based on 502 and 2,037 patients and validated on 273 and 1,107 patients, respectively. The polytomous models combined dichotomous models for category A vs. B + C and B vs. C (sequential dichotomous modeling). Simple recalibration, revision, and redevelopment methods were considered. To assess discrimination (using dichotomous and polytomous c-statistics) and calibration (by comparing observed and expected prevalences) of these methods, the validation data were divided into updating and test parts. Five hundred such divisions were randomly generated, and the average test set results reported.ResultsNone of the updating methods could improve discrimination of the original models, but recalibration, revision, and redevelopment strongly improved calibration. Redevelopment was unstable with respect to overfitting and performance.ConclusionSimple dichotomous updating methods behaved well when applied to polytomous models. Our results suggest that recalibration is preferred, but larger validation sets may make revision or redevelopment a sensible alternative.     

Polytomous logistic regression analysis could be applied more often in diagnostic research

OBJECTIVE: Physicians commonly consider the presence of all differential diagnoses simultaneously. Polytomous logistic regression modeling allows for simultaneous estimation of the probability of multiple diagnoses. We discuss and (empirically) illustrate the value of this method for diagnostic research. STUDY DESIGN AND SETTING: We used data from a study on the diagnosis of residual retroperitoneal mass histology in patients presenting with nonseminomatous testicular germ cell tumor. The differential diagnoses include benign tissue, mature teratoma, and viable cancer. Probabilities of each diagnosis were estimated with a polytomous logistic regression model and compared with the probabilities estimated from two consecutive dichotomous logistic regression models. RESULTS: We provide interpretations of the odds ratios derived from the polytomous regression model and present a simple score chart to facilitate calculation of predicted probabilities from the polytomous model. For both modeling methods, we show the calibration plots and receiver operating characteristics curve (ROC) areas comparing each diagnostic outcome category with the other two. The ROC areas for benign tissue, mature teratoma, and viable cancer were similar for both modeling methods, 0.83 (95% confidence interval [CI]=0.80-0.85) vs. 0.83 (95% CI=0.80-0.85), 0.78 (95% CI=0.75-0.81) vs. 0.78 (95% CI=0.75-0.81), and 0.66 (95% CI=0.61-0.71) vs. 0.64 (95% CI=0.59-0.69), for polytomous and dichotomous regression models, respectively. CONCLUSION: Polytomous logistic regression is a useful technique to simultaneously model predicted probabilities of multiple diagnostic outcome categories. The performance of a polytomous prediction model can be assessed similarly to a dichotomous logistic regression model, and predictions by a polytomous model can be made with a user-friendly method. Because the simultaneous consideration of the presence of multiple (differential) conditions serves clinical practice better than consideration of the presence of only one target condition, polytomous logistic regression could be applied more often in diagnostic research.     

Assessing calibration of multinomial risk prediction models

Calibration, that is, whether observed outcomes agree with predicted risks, is important when evaluating risk prediction models. For dichotomous outcomes, several tools exist to assess different aspects of model calibration, such as calibration‐in‐the‐large, logistic recalibration, and (non‐)parametric calibration plots. We aim to extend these tools to prediction models for polytomous outcomes. We focus on models developed using multinomial logistic regression (MLR): outcome Y with k categories is predicted using k ? 1 equations comparing each category i (i = 2, … ,k) with reference category 1 using a set of predictors, resulting in k ? 1 linear predictors. We propose a multinomial logistic recalibration framework that involves an MLR fit where Y is predicted using the k ? 1 linear predictors from the prediction model. A non‐parametric alternative may use vector splines for the effects of the linear predictors. The parametric and non‐parametric frameworks can be used to generate multinomial calibration plots. Further, the parametric framework can be used for the estimation and statistical testing of calibration intercepts and slopes. Two illustrative case studies are presented, one on the diagnosis of malignancy of ovarian tumors and one on residual mass diagnosis in testicular cancer patients treated with cisplatin‐based chemotherapy. The risk prediction models were developed on data from 2037 and 544 patients and externally validated on 1107 and 550 patients, respectively. We conclude that calibration tools can be extended to polytomous outcomes. The polytomous calibration plots are particularly informative through the visual summary of the calibration performance. Copyright © 2014 John Wiley & Sons, Ltd.     

A new concordance measure for risk prediction models in external validation settings

Concordance measures are frequently used for assessing the discriminative ability of risk prediction models. The interpretation of estimated concordance at external validation is difficult if the case‐mix differs from the model development setting. We aimed to develop a concordance measure that provides insight into the influence of case‐mix heterogeneity and is robust to censoring of time‐to‐event data. We first derived a model‐based concordance (mbc) measure that allows for quantification of the influence of case‐mix heterogeneity on discriminative ability of proportional hazards and logistic regression models. This mbc can also be calculated including a regression slope that calibrates the predictions at external validation (c‐mbc), hence assessing the influence of overall regression coefficient validity on discriminative ability. We derived variance formulas for both mbc and c‐mbc. We compared the mbc and the c‐mbc with commonly used concordance measures in a simulation study and in two external validation settings. The mbc was asymptotically equivalent to a previously proposed resampling‐based case‐mix corrected c‐index. The c‐mbc remained stable at the true value with increasing proportions of censoring, while Harrell's c‐index and to a lesser extent Uno's concordance measure increased unfavorably. Variance estimates of mbc and c‐mbc were well in agreement with the simulated empirical variances. We conclude that the mbc is an attractive closed‐form measure that allows for a straightforward quantification of the expected change in a model's discriminative ability due to case‐mix heterogeneity. The c‐mbc also reflects regression coefficient validity and is a censoring‐robust alternative for the c‐index when the proportional hazards assumption holds. Copyright © 2016 John Wiley & Sons, Ltd.     

Comparación de métodos de escalamiento de actividades de la vida diaria en personas mayores

ObjectiveTo compare the psychometric properties of scales top measure activities of daily living, constructed with different scaling methods, and to check whether the most complex scales have higher discriminatory capacity.MethodSample of elderly people from the Spanish Survey on Disability, Personal Autonomy and Dependency We used 14 items that measured activities of daily living. Five scaling methods were applied: Sum and Rasch (both for dichotomous and polytomous items) and Guttman (dichotomous). We evaluated the discriminatory capacity (relative precision [RP]) and area under the curve (AUC).ResultsAll methods showed high Pearson correlations among them (0.765-0.993). They had similar discriminatory power when comparing extreme categories of individuals with no disability with severely limited (RP: 0.93-1.00). The polytomous Sum procedure showed the highest AUC (0.934; 95% confidence interval [95%CI]: 0.928-0.939) and Guttman the lowest (0.853; 95%CI: 0.845-0.861).ConclusionsPolytomous items have greater reliability than the dichotomous ones. Simplest methods (Sum) and most complex (Rasch) are equally valid. Guttman method presented worse discriminatory capacity.     

A MEASURE OF ASSOCIATION FOR ASSESSING PREDICTION ACCURACY THAT IS A GENERALIZATION OF NON-PARAMETRIC ROC AREA

There is a need for a measure of prediction accuracy that generalizes non-parametric receiver operating characteristic (ROC) area to polytomous ordinal patient state. We describe such a measure, prediction probability P_K, derived from Kim's measure of association. We show that the value of P_K equals the value of non-parametric ROC area for dichotomous patient state and is a meaningful generalization of non-parametric ROC area for polytomous state.     

Extending the c-statistic to nominal polytomous outcomes: the Polytomous Discrimination Index

Diagnostic problems in medicine are sometimes polytomous, meaning that the outcome has more than two distinct categories. For example, ovarian tumors can be benign, borderline, primary invasive, or metastatic. Extending the main measure of binary discrimination, the c‐statistic or area under the ROC curve, to nominal polytomous settings is not straightforward. This paper reviews existing measures and presents the polytomous discrimination index (PDI) as an alternative. The PDI assesses all sets of k cases consisting of one case from each outcome category. For each category i (i = 1, … ,k), it is assessed whether the risk of category i is highest for the case from category i. A score of 1∕k is given per category for which this holds, yielding a set score between 0 and 1 to indicate the level of discrimination. The PDI is the average set score and is interpreted as the probability to correctly identify a case from a randomly selected category within a set of k cases. This probability can be split up by outcome category, yielding k category‐specific values that result in the PDI when averaged. We demonstrate the measures on two diagnostic problems (residual mass histology after chemotherapy for testicular cancer; diagnosis of ovarian tumors). We compare the behavior of the measures on theoretical data, showing that PDI is more strongly influenced by simultaneous discrimination between all categories than by partial discrimination between pairs of categories. In conclusion, the PDI is attractive because it better matches the requirements of a measure to summarize polytomous discrimination. Copyright © 2012 John Wiley & Sons, Ltd.     

Dichotomous or categorical response? Analysing self-rated health and lifetime social class

BACKGROUND: Self-rated health is a commonly used measure of health status, usually having three to five categories. The measure is often collapsed into a dichotomous variable of good versus less than good health. This categorization has not yet been justified. METHODS: Using data from the 1958 British birth cohort, we examined the relationship between socioeconomic conditions, indicated by occupational class at four ages, and self-rated health. Results obtained for a dichotomous variable using logistic regression were compared with alternative methods for ordered categorical variables including polytomous regression, cumulative odds, continuation ratio and adjacent categories models. RESULTS AND CONCLUSIONS: Findings concerning the relationship between socioeconomic position and self-rated health yielded by a logistic regression model were confirmed by alternative statistical methods which incorporate the ordered nature of self-rated health. Similarity of results was found regarding size and significance of main effects, type of association and interactive effects.     

Sample size considerations and predictive performance of multinomial logistic prediction models

Multinomial Logistic Regression (MLR) has been advocated for developing clinical prediction models that distinguish between three or more unordered outcomes. We present a full-factorial simulation study to examine the predictive performance of MLR models in relation to the relative size of outcome categories, number of predictors and the number of events per variable. It is shown that MLR estimated by Maximum Likelihood yields overfitted prediction models in small to medium sized data. In most cases, the calibration and overall predictive performance of the multinomial prediction model is improved by using penalized MLR. Our simulation study also highlights the importance of events per variable in the multinomial context as well as the total sample size. As expected, our study demonstrates the need for optimism correction of the predictive performance measures when developing the multinomial logistic prediction model. We recommend the use of penalized MLR when prediction models are developed in small data sets or in medium sized data sets with a small total sample size (ie, when the sizes of the outcome categories are balanced). Finally, we present a case study in which we illustrate the development and validation of penalized and unpenalized multinomial prediction models for predicting malignancy of ovarian cancer.     

Preliminary state of development of prediction models for primary care physical therapy: a systematic review

ObjectiveTo summarize the methodological quality and developmental stage of prediction models for musculoskeletal complaints that are relevant for physical therapists in primary care.Study Design and SettingA systematic literature search was carried out in the databases of Medline, Embase, and Cinahl. Studies on prediction models for musculoskeletal complaints that can be used by primary care physical therapists were included. Methodological quality of the studies was assessed and relevant study characteristics were extracted.ResultsThe search retrieved 4,702 references of which 29 studies were included in this review. The study quality of the included studies showed substantial variation. The studied populations consisted mostly of back (n = 10) and neck pain (n = 6) patients, and patients with knee complaints (n = 4). Most studies (n = 22) used “perceived recovery” as primary outcome. Most prediction models (n = 18) were at the derivation level of development.ConclusionsMany prediction models are available for a wide range of patient populations. The developmental stage of most models is preliminary and the study quality is often moderate. We do not recommend physiotherapist to use these models yet. All models reviewed here are in the developmental stage and need validation and impact evaluation before using them in daily practice.     

Predictive accuracy and explained variation

Measures of the predictive accuracy of regression models quantify the extent to which covariates determine an individual outcome. Explained variation measures the relative gains in predictive accuracy when prediction based on covariates replaces unconditional prediction. A unified concept of predictive accuracy and explained variation based on the absolute prediction error is presented for models with continuous, binary, polytomous and survival outcomes. The measures are given both in a model-based formulation and in a formulation directly contrasting observed and expected outcomes. Various aspects of application are demonstrated by examples from three forms of regression models. It is emphasized that the likely degree of absolute or relative predictive accuracy often is low even if there are highly significant and relatively strong covariates.     

Prioritized concordance index for hierarchical survival outcomes

We propose an extension of Harrell's concordance (C) index to evaluate the prognostic utility of biomarkers for diseases with multiple measurable outcomes that can be prioritized. Our prioritized concordance index measures the probability that, given a random subject pair, the subject with the worst disease status as of a time τ has the higher predicted risk. Our prioritized concordance index uses the same approach as the win ratio, by basing generalized pairwise comparisons on the most severe or clinically important comparable outcome. We use an inverse probability weighting technique to correct for study-specific censoring. Asymptotic properties are derived using U-statistic properties. We apply the prioritized concordance index to two types of disease processes with a rare primary outcome and a more common secondary outcome. Our simulation studies show that when a predictor is predictive of both outcomes, the new concordance index can gain efficiency and power in identifying true prognostic variables compared to using the primary outcome alone. Using the prioritized concordance index, we examine whether novel clinical measures can be useful in predicting risk of type II diabetes in patients with impaired glucose resistance whose disease status can also regress to normal glucose resistance. We also examine the discrimination ability of four published risk models among ever smokers at risk of lung cancer incidence and subsequent death.     

Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets.

Clinical decision making often requires estimates of the likelihood of a dichotomous outcome in individual patients. When empirical data are available, these estimates may well be obtained from a logistic regression model. Several strategies may be followed in the development of such a model. In this study, the authors compare alternative strategies in 23 small subsamples from a large data set of patients with an acute myocardial infarction, where they developed predictive models for 30-day mortality. Evaluations were performed in an independent part of the data set. Specifically, the authors studied the effect of coding of covariables and stepwise selection on discriminative ability of the resulting model, and the effect of statistical "shrinkage" techniques on calibration. As expected, dichotomization of continuous covariables implied a loss of information. Remarkably, stepwise selection resulted in less discriminating models compared to full models including all available covariables, even when more than half of these were randomly associated with the outcome. Using qualitative information on the sign of the effect of predictors slightly improved the predictive ability. Calibration improved when shrinkage was applied on the standard maximum likelihood estimates of the regression coefficients. In conclusion, a sensible strategy in small data sets is to apply shrinkage methods in full models that include well-coded predictors that are selected based on external information.     

A nomogram was developed to enhance the use of multinomial logistic regression modeling in diagnostic research

ObjectivesWe developed a nomogram to facilitate the interpretation and presentation of results from multinomial logistic regression models.Study Design and SettingWe analyzed data from 376 frail elderly with complaints of dyspnea. Potential underlying disease categories were heart failure (HF), chronic obstructive pulmonary disease (COPD), the combination of both (HF and COPD), and any other outcome (other). A nomogram for multinomial model was developed to depict the relative importance of each predictor and to calculate the probability for each disease category for a given patient. Additionally, model performance of the multinomial regression model was assessed.ResultsPrevalence of HF and COPD was 14% (n = 54), HF 24% (n = 90), COPD 20% (n = 75), and Other 42% (n = 157). The relative importance of the individual predictors varied across these disease categories or was even reversed. The pairwise C statistics ranged from 0.75 (between HF and Other) to 0.96 (between HF and COPD and Other). The nomogram can be used to rank the disease categories from most to least likely within each patient or to calculate the predicted probabilities.ConclusionsOur new nomogram is a useful tool to present and understand the results of a multinomial regression model and could enhance the applicability of such models in daily practice.     

Power and sample size determination for group comparison of patient‐reported outcomes using polytomous Rasch models

The analysis of patient‐reported outcomes or other psychological traits can be realized using the Rasch measurement model. When the objective of a study is to compare groups of individuals, it is important, before the study, to define a sample size such that the group comparison test will attain a given power. The Raschpower procedure (RP) allows doing so with dichotomous items. The RP is extended to polytomous items. Several computational issues were identified, and adaptations have been proposed. The performance of this new version of RP is assessed using simulations. This adaptation of RP allows obtaining a good estimate of the expected power of a test to compare groups of patients in a large number of practical situations. A Stata module, as well as its implementation online, is proposed to perform the RP. Two versions of the RP for polytomous items are proposed (deterministic and stochastic versions). These two versions produce similar results in all of the tested cases. We recommend the use of the deterministic version, when the measure is obtained using small questionnaires or items with a few number of response categories, and the stochastic version elsewhere, so as to optimize computing time. Copyright © 2015 John Wiley & Sons, Ltd.     

Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework

Background

There is an increased interest in the use of multi-criteria decision analysis (MCDA) to support regulatory and reimbursement decision making. The EVIDEM framework was developed to provide pragmatic multi-criteria decision support in health care, to estimate the value of healthcare interventions, and to aid in priority-setting. The objectives of this study were to test 1) the influence of different weighting techniques on the overall outcome of an MCDA exercise, 2) the discriminative power in weighting different criteria of such techniques, and 3) whether different techniques result in similar weights in weighting the criteria set proposed by the EVIDEM framework.

Methods

A sample of 60 Dutch and Canadian students participated in the study. Each student used an online survey to provide weights for 14 criteria with two different techniques: a five-point rating scale and one of the following techniques selected randomly: ranking, point allocation, pairwise comparison and best worst scaling.

Results

The results of this study indicate that there is no effect of differences in weights on value estimates at the group level. On an individual level, considerable differences in criteria weights and rank order occur as a result of the weight elicitation method used, and the ability of different techniques to discriminate in criteria importance. Of the five techniques tested, the pair-wise comparison of criteria has the highest ability to discriminate in weights when fourteen criteria are compared.

Conclusions

When weights are intended to support group decisions, the choice of elicitation technique has negligible impact on criteria weights and the overall value of an innovation. However, when weights are used to support individual decisions, the choice of elicitation technique influences outcome and studies that use dissimilar techniques cannot be easily compared. Weight elicitation through pairwise comparison of criteria is preferred when taking into account its superior ability to discriminate between criteria and respondents’ preferences.

     

Visualizing and assessing discrimination in the logistic regression model

Logistic regression models are widely used in medicine for predicting patient outcome (prognosis) and constructing diagnostic tests (diagnosis). Multivariable logistic models yield an (approximately) continuous risk score, a transformation of which gives the estimated event probability for an individual. A key aspect of model performance is discrimination, that is, the model's ability to distinguish between patients who have (or will have) an event of interest and those who do not (or will not). Graphical aids are important in understanding a logistic model. The receiver‐operating characteristic (ROC) curve is familiar, but not necessarily easy to interpret. We advocate a simple graphic that provides further insight into discrimination, namely a histogram or dot plot of the risk score in the outcome groups. The most popular performance measure for the logistic model is the c‐index, numerically equivalent to the area under the ROC curve. We discuss the comparative merits of the c‐index and the (standardized) mean difference in risk score between the outcome groups. The latter statistic, sometimes known generically as the effect size, has been computed in slightly different ways by several different authors, including Glass, Cohen and Hedges. An alternative measure is the overlap between the distributions in the outcome groups, defined as the area under the minimum of the two density functions. The larger the overlap, the weaker the discrimination. Under certain assumptions about the distribution of the risk score, the c‐index, effect size and overlap are functionally related. We illustrate the ideas with simulated and real data sets. Copyright © 2010 John Wiley & Sons, Ltd.     

Development of an Easy-to-Use Visual Aid for the Prediction of Body Fat Based on Waist Circumference and Height in Asian Chinese Adults

BackgroundAdiposity is a major risk factor for metabolic and cardiovascular diseases. Initial prediction equations to estimate adiposity are complex, requiring skinfold measurements that cannot be obtained conveniently by the general population.ObjectiveTo develop simplified prediction equations to estimate body fat percentage (%BF) in Asian Chinese adults, evaluate the validity of the simplified %BF prediction equations, compare the simplified %BF prediction equations with an existing equation, and create visual charts to enable easy assessment of adiposity by the general public.DesignSimplified prediction equations were developed and evaluated for validity using anthropometric measurements obtained from a cross-sectional study.Participants and settingHealthy participants with no major diseases and not taking long-term medications were recruited in a cross-sectional study conducted at Clinical Nutrition Research Centre, Singapore, between June 2014 and October 2017. A total of 439 participants were used for model building (269 women and 170 men) and another 107 participants were used for evaluating validity (62 women and 45 men).Main outcome measuresSimplified but acceptable prediction models and generation of user-friendly charts.Statistical analyses performedSimplified sex-specific %BF prediction equations were developed using stepwise regression and the model-building dataset. The best models were selected using the Akaike information criterion. The models were further simplified and their performance was compared using the validation dataset before choosing the final prediction equations.ResultsThe final selected models for women and men included waist circumference and height with nonsignificant prediction bias in %BF of 0.84%±3.94% (P=0.098, Cohen’s d_z=0.21) and –0.98%±3.65% (P=0.079, Cohen’s d_z=0.27), respectively. The final equations were split into three height categories from which the sex-specific prediction charts were generated.ConclusionsThe sex-specific prediction charts provide a good visual guide for estimating %BF using height and waist circumference values that are easy to obtain by the general public.     

How long do preconception risk prediction models hold? Influence of selective fertility on model performance

Risk estimates derived from preconception prediction models can be used to counsel women with regard to any future pregnancies. Women with a high predicted risk of an adverse pregnancy outcome may decide more frequently not to try for another pregnancy than women with a low predicted risk. This prediction-guided selective fertility can cause a change in the composition of the pregnant population with respect to those parameters that are included in the prediction model. The question is whether such a change in composition could influence the performance parameters of the prediction model, such as sensitivity, specificity, positive and negative predictive values as well as the discriminative ability, when evaluating risks in the new population and whether it could compromise the longevity of the model. Using a hypothetical example, we show that the original sensitivity and specificity estimates of a preconception prediction model for an adverse pregnancy outcome no longer hold when the model is applied to a population affected by model-based selective fertility: sensitivity decreases, while specificity increases. However, individual patient risk estimates remain unbiased and discriminative ability, expressed as the area under the receiver operating characteristic curve, remains unaffected.