Goodness-of-fit issues in ROC curve estimation.

Zhou recently considered goodness-of-fit (GOF) testing for receiver operating characteristic (ROC) curves estimated by applying the binormal and other bidistributional models to rating method data. He interpreted significant GOF tests as evidence that different decision thresholds were applied to diseased and nondiseased subjects and concluded that, in such circumstances, an ROC curve does not exist. In this article the author demonstrates that the GOF test accommodates many alternative hypotheses and that a significant test result need not be equated with an interaction between disease status and decision criteria or with non-existence of an ROC curve. He develops a new family of ROC curves based on a fully parameterized bidistributional model. The family includes the binormal ROC curve, but generalizes its structure by signifying all identifiable deviations in parameters of the latent distributions that define the model. The family provides a unified framework of alternative models to the binormal assumption and of alternative hypotheses to the GOF test for that assumption.     

Dynamic thresholds and a summary ROC curve: Assessing prognostic accuracy of longitudinal markers

Cancer patients, chronic kidney disease patients, and subjects infected with HIV are routinely monitored over time using biomarkers that represent key health status indicators. Furthermore, biomarkers are frequently used to guide initiation of new treatments or to inform changes in intervention strategies. Since key medical decisions can be made on the basis of a longitudinal biomarker, it is important to evaluate the potential accuracy associated with longitudinal monitoring. To characterize the overall accuracy of a time‐dependent marker, we introduce a summary ROC curve that displays the overall sensitivity associated with a time‐dependent threshold that controls time‐varying specificity. The proposed statistical methods are similar to concepts considered in disease screening, yet our methods are novel in choosing a potentially time‐dependent threshold to define a positive test, and our methods allow time‐specific control of the false‐positive rate. The proposed summary ROC curve is a natural averaging of time‐dependent incident/dynamic ROC curves and therefore provides a single summary of net error rates that can be achieved in the longitudinal setting.     

Small area estimation of receiver operating characteristic curves for ordinal data under stochastic ordering

There has been a recent increase in the diagnosis of diseases through radiographic images such as x-rays, magnetic resonance imaging, and computed tomography. The outcome of a radiological diagnostic test is often in the form of discrete ordinal data, and we usually summarize the performance of the diagnostic test using the receiver operating characteristic (ROC) curve and the area under the curve (AUC). The ROC curve will be concave and called proper when the outcomes of the diagnostic test in the actually positive subjects are higher than in the actually negative subjects. The diagnostic test for disease detection is clinically useful when a ROC curve is proper. In this study, we develop a hierarchical Bayesian model to estimate the proper ROC curve and AUC using stochastic ordering in several domains when the outcome of the diagnostic test is discrete ordinal data and compare it with the model without stochastic ordering. The model without stochastic ordering can estimate the improper ROC curve with a nonconcave shape or a hook when the true ROC curve of the population is a proper ROC curve. Therefore, the model with stochastic ordering is preferable over the model without stochastic ordering to estimate the proper ROC curve with clinical usefulness for ordinal data.     

Notes on the overlap measure as an alternative to the Youden index: How are they related?

The receiver operating characteristic (ROC) curve is frequently used to evaluate and compare diagnostic tests. As one of the ROC summary indices, the Youden index measures the effectiveness of a diagnostic marker and enables the selection of an optimal threshold value (cut‐off point) for the marker. Recently, the overlap coefficient, which captures the similarity between 2 distributions directly, has been considered as an alternative index for determining the diagnostic performance of markers. In this case, a larger overlap indicates worse diagnostic accuracy, and vice versa. This paper provides a graphical demonstration and mathematical derivation of the relationship between the Youden index and the overlap coefficient and states their advantages over the most popular diagnostic measure, the area under the ROC curve. Furthermore, we outline the differences between the Youden index and overlap coefficient and identify situations in which the overlap coefficient outperforms the Youden index. Numerical examples and real data analysis are provided.     

A new parametric method based on S-distributions for computing receiver operating characteristic curves for continuous diagnostic tests

Receiver operating characteristic (ROC) curves provides a method for evaluating the performance of a diagnostic test. These curves represent the true positive ratio, that is, the true positives among those affected by the disease, as a function of the false positive ratio, that is, the false positives among the healthy, corresponding to each possible value of the diagnostic variable. When the diagnostic variable is continuous, the corresponding ROC curve is also continuous. However, estimation of such curve through the analysis of sample data yields a step-line, unless some assumption is made on the underlying distribution of the considered variable. Since the actual distribution of the diagnostic test is seldom known, it is difficult to select an appropriate distribution for practical use. Data transformation may offer a solution but also may introduce a distortion on the evaluation of the diagnostic test. In this paper we show that the distribution family known as the S-distribution can be used to solve this problem. The S-distribution is defined as a differential equation in which the dependent variable is the cumulative. This special form provides a highly flexible family of distributions that can be used as models for unknown distributions. It has been shown that classical statistical distributions can be represented accurately as S-distributions and that they occur in a definite subspace of the parameter space corresponding to the whole S-distribution family. Consequently, many other distributional forms that do not correspond to known distributions are provided by the S-distribution. This property can be used to model observed data for unknown distributions and is very useful in constructing parametric ROC curves in those cases. After fitting an S-distribution to the observed samples of diseased and healthy populations, ROC curve computation is straightforward. A ROC curve can be considered as the solution of a differential equation in which the dependent variable is the ratio of true positives and the independent variable is the ratio of false positives. This equation can be easily obtained from the S-distributions fitted to observed data. Using these results, we can compute pointwise confidence bands for the ROC curve and the corresponding area under the curve. We shall compare this approach with the empirical and the binormal methods for estimating a ROC curve to show that the S-distribution based method is a useful parametric procedure.     

A general regression methodology for ROC curve estimation

A method for applying generalized ordinal regression models to categorical rating data to estimate and analyze receiver operating characteristic (ROC) curves is presented. These models permit parsimonious adjustment of ROC curve parameters for relevant covariates through two regression equations that correspond to location and scale. Particular shapes of ROC curves are interpreted in relation to the kind of covariates included in the two regressions. The model is shown to be flexible because it is not restricted to the assumption of binormality that is commonly employed in smoothed ROC curve estimation, although the binormal model is one particular form of the more general model. The new method provides a mechanism for pinpointing the effect that interobserver variability has on the ROC curve. It also allows for the adjustment of ROC curves for temporal variation and case mix, and provides a way to assess the incremental diagnostic value of a test. The new methodology is recommended because it substantially improves the ability to assess diagnostic tests using ROC curves.     

Adjusting the generalized ROC curve for covariates

Receiver operating characteristic (ROC) curves and in particular the area under the curve (AUC), are widely used to examine the effectiveness of diagnostic markers. Diagnostic markers and their corresponding ROC curves can be strongly influenced by covariate variables. When several diagnostic markers are available, they can be combined by a best linear combination such that the area under the ROC curve of the combination is maximized among all possible linear combinations. In this paper we discuss covariate effects on this linear combination assuming that the multiple markers, possibly transformed, follow a multivariate normal distribution. The ROC curve of this linear combination when markers are adjusted for covariates is estimated and approximate confidence intervals for the corresponding AUC are derived. An example of two biomarkers of coronary heart disease for which covariate information on age and gender is available is used to illustrate this methodology.     

An evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve

The area under the receiver operating characteristic (ROC) curve serves as one means for evaluating the performance of diagnostic and predictive test systems. The most commonly used method for estimating the area under an ROC curve utilizes the maximum-likelihood-estimation technique, and a nonparametric method to calculate the area under an ROC curve was recently described. We compared the performance of these two methods. The results for the area under the ROC curve and the standard error of the estimate as calculated by each of the two methods exhibited high correlation. Generally, the nonparametric method yields lower area estimates than the maximum-likelihood-estimation technique. However, these differences generally were small, particularly with ROC curves derived from five or more cutoff points. Consistent results of hypothesis testing of the significance of differences between two ROC curves will be similar, regardless of which method is used, as long as one uses the same estimation technique on the two curves and as long as the two ROC curves being compared are of similar shape.     

A permutation test sensitive to differences in areas for comparing ROC curves from a paired design

The area under the receiver operating characteristic (ROC) curve (AUC) is a widely accepted summary index of the overall performance of diagnostic procedures and the difference between AUCs is often used when comparing two diagnostic systems. We developed an exact non-parametric statistical procedure for comparing two ROC curves in paired design settings. The test which is based on all permutations of the subject specific rank ratings is formally a test for equality of ROC curves that is sensitive to the alternatives of AUC difference. The operating characteristics of the proposed test were evaluated using extensive simulations over a wide range of parameters.The proposed procedure can be easily implemented in experimental ROC data sets. For small samples and for underlying parameters that are common in experimental studies in diagnostic imaging the test possesses good operating characteristics and is more powerful than the conventional non-parametric procedure for AUC comparisons.We also derived an asymptotic version of the test which uses an exact estimate of the variance in the permutation space and provides a good approximation even when the sample sizes are small. This asymptotic procedure is a simple and precise approximation to the exact test and is useful for large sample sizes where the exact test may be computationally burdensome.     

An exponential model used for optimal threshold selection on ROC curves

A two-parameter exponential equation for modeling a receiver operating characteristic (ROC) curve is presented, where the area under the curve is a simple function of one of the parameters. The model makes no distributional assumptions about the underlying normal and abnormal patient populations or about the shape of the resulting ROC curves. In a computer simulation of 75 ROC curves, the model provides a fit equivalent to the maximum likelihood estimate method commonly used for ROC curve fitting. Similar results are obtained using the model to fit ROC curve data from the literature. The model's equation calculates the true-positive ratio as a function of the false-positive ratio, and has a first derivative that is useful for finding the optimal decision threshold for a diagnostic testing procedure. In particular, the model is useful in a computer program for finding jointly optimal thresholds for multiple sequential tests.     

ROC curves for the initial assessment of new diagnostic tests.

New diagnostic tests are mainly evaluated by determining the sensitivity and specificity of the test. These test characteristics were originally meant to be used in making diagnoses. For evaluative purposes their usefulness is weakened by their susceptibility to selection and their dependence on the cut-off points that are used for test positivity. The plotting of a receiver operating characteristic (ROC) curve might be a solution to these problems. Furthermore, the ROC curve yields a measure for the diagnostic power of the test expressed in one number instead of two, namely the area under the curve (AUC). Finally, the ROC curve and its AUC permit easy comparison of different tests and the performance of different interpreters of one test. The construction and use of ROC curves are described and illustrated with data of a case-referent investigation into the relationship between iron status parameters and the presence of acute myocardial infarction. The AUCs of ferritin and serum iron, 0.61 and 0.68 respectively, are too low to suggest meaningful usefulness in clinical practice.     

Assessment of diagnostic markers by goodness-of-fit tests

Receiver operating characteristic (ROC) curves are useful statistical tools used to assess the precision of diagnostic markers or to compare new diagnostic markers with old ones. The most common index employed for these purposes is the area under the ROC curve (theta) and several statistical tests exist that test the null hypotheses H(0): theta= 0.5 or H(0): theta1=theta2, in the case of two-marker comparisons, against alternatives of interest. In this paper we show that goodness-of-fit of uniformity of the distribution of the false positive (true positive) rates can be used instead of tests based on the area index. A semi-parametric approach is based on a completely specified distribution of marker measurements for either the healthy (F) or diseased (G) subjects, and this is extended to the two-marker case. We then extend to the one- and two-marker case when neither distribution is specified (the non-parametric case). In general, ROC-based tests are more powerful than goodness-of-fit tests for location differences between the distributions of healthy and diseased subjects. However ROC-based tests are less powerful when location-scale differences exist (producing ROC curves that cross the diagonal) and are incapable of discriminating between healthy and diseased samples when theta=0.5 but F not equal G. In these cases, goodness-of-fit tests have a distinct advantage over ROC-based tests. In conclusion, ROC methodology should be used with recognition of its potential limitations and should be replaced by goodness-of-fit tests when appropriate. The latter are a viable alternative and can be used as a 'black box' or as an exploratory first step in the evaluation of novel diagnostic markers.     

Direct estimation of the area under the receiver operating characteristic curve in the presence of verification bias

The area under a receiver operating characteristic (ROC) curve (AUC) is a commonly used index for summarizing the ability of a continuous diagnostic test to discriminate between healthy and diseased subjects. If all subjects have their true disease status verified, one can directly estimate the AUC nonparametrically using the Wilcoxon statistic. In some studies, verification of the true disease status is performed only for a subset of subjects, possibly depending on the result of the diagnostic test and other characteristics of the subjects. Because estimators of the AUC based only on verified subjects are typically biased, it is common to estimate the AUC from a bias-corrected ROC curve. The variance of the estimator, however, does not have a closed-form expression and thus resampling techniques are used to obtain an estimate. In this paper, we develop a new method for directly estimating the AUC in the setting of verification bias based on U-statistics and inverse probability weighting (IPW). Closed-form expressions for the estimator and its variance are derived. We also show that the new estimator is equivalent to the empirical AUC derived from the bias-corrected ROC curve arising from the IPW approach.     

无金标准条件下含协变量的ROC曲线估计方法

目的阐明无金标准条件下,考虑协变量后估计ROC曲线的两部贝叶斯模型。方法介绍两部贝叶斯模型,结合实例,筛选无金标准条件下ROC曲线的影响因素,考虑协变量影响后,估计ROC曲线。结果两部贝叶斯模型不仅可探讨协变量对疾病状态的影响,而且可探讨协变量对诊断试验结果的影响,同时可计算不同协变量取值条件下ROC曲线下面积。结论两部贝叶斯模型可有效地解决无金标准条件下,考虑协变量影响的ROC曲线估计问题。     

Interval estimation for the difference in paired areas under the ROC curves in the absence of a gold standard test

Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard (GS) test on the true disease status is required to estimate the AUC. However, a GS test may sometimes be too expensive or infeasible. Therefore, in many medical research studies, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, using the expectation‐maximization (EM) algorithm in conjunction with a bootstrap method, we propose a maximum likelihood‐based procedure for the construction of confidence intervals for the difference in paired AUCs in the absence of a GS test. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and interval lengths. The proposed method is illustrated with two examples. Copyright © 2009 John Wiley & Sons, Ltd.     

A non-inferiority test for diagnostic accuracy based on the paired partial areas under ROC curves

Non-inferiority is a reasonable approach to assessing the diagnostic accuracy of a new diagnostic test if it provides an easier administration or reduces the cost. The area under the receiver operating characteristic (ROC) curve is one of the common measures for the overall diagnostic accuracy. However, it may not differentiate the various shapes of the ROC curves with different diagnostic significances. The partial area under the ROC curve (PAUROC) may present an alternative that can provide additional and complimentary information for some diagnostic tests which require false-positive rate that does not exceed a certain level. Non-parametric and maximum likelihood methods can be used for the non-inferiority tests based on the difference in paired PAUROCs. However, their performance has not been investigated in finite samples. We propose to use the concept of generalized p-value to construct a non-inferiority test for diagnostic accuracy based on the difference in paired PAUROCs. Simulation results show that the proposed non-inferiority test not only adequately controls the size at the nominal level but also is uniformly more powerful than the non-parametric methods. The proposed method is illustrated with a numerical example using published data.     

Meta-analysis of ROC curves.

The authors present a method to combine several independent studies of the same (continuous or semiquantitative) diagnostic test, where each study reports a complete ROC curve; a plot of the true-positive rate or sensitivity against the false-positive rate or one minus the specificity. The result of the analysis is a pooled ROC curve, with a confidence band, as opposed to earlier proposals that result in a pooled area under the ROC curve. The analysis is based on a two-parameter model for the ROC curve that can be estimated for each individual curve. The parameters are then pooled with a bivariate random-effects meta-analytic method, and a curve can be drawn from the pooled parameters. The authors propose to use a model that specifies a linear relation between the logistic transformations of sensitivity and one minus specificity. Specifically, they define V = In(sensitivity/(1 - sensitivity)) and U = In((1 - specificity)/specificity), and then D = V - U, S = V + U. The model is defined as D = alpha + betaS. The parameters alpha and beta are estimated using weighted linear regression with bootstrapping to get the standard errors, or using maximum likelihood. The authors show how the procedure works with continuous test data and with categorical test data.     

Maximizing an ROC‐type measure via linear combination of markers when the gold reference is continuous

Effectively combining many classification instruments or diagnostic measurements together to improve the classification accuracy of individuals is a common idea in disease diagnosis or classification. These ensemble‐type diagnostic methods can be constructed with respect to different kinds of performance criterions. Among them, the receiver operating characteristic (ROC) curve is the most popular criterion, which, together with some indexes derived from it, is commonly used to evaluate and summarize the performance of a classification instrument, such as a biomarker or a classifier. However, the usefulness of ROC curve and its related indexes relies on the existence of a binary label for each individual subject. In many disease diagnosis situations, such a binary variable may not exist, but only the continuous measurement of the true disease status is available. This true disease status is often referred to as the ‘gold standard’. The modified area under ROC curve (AUC)‐type measure defined by Obuchowski is a method proposed to accommodate such a situation. However, there is still no method for finding the optimal combination of diagnostic measurements, with respect to such an index, to have better diagnostic power than that of each individual measurement. In this paper, we propose an algorithm for finding the optimal combination with respect to such an extended AUC‐type measure such that the combined measurement can have more diagnostic power. We illustrate the performance of our algorithm by using some synthesized data and a diabetes data set. Copyright © 2012 John Wiley & Sons, Ltd.     

The area under an ROC curve with limited information.

The area under the receiver operating characteristic (ROC) curve of a diagnostic test can be used as a summary measure for its discriminative ability. If only a single point of an ROC curve is available, then the entire form of the ROC curve is unknown and the area under it cannot be calculated. Assuming that the unknown ROC curve is either monotone or concave, lower and upper bounds are derived for the area. From these bounds, the minmax approximations are obtained. Compared to only assuming monotonicity, assuming that the unknown ROC curve is concave renders a higher minmax approximation for the area under it, with tighter bounds.     

The partial area under the summary ROC curve

The area under the curve (AUC) is commonly used as a summary measure of the receiver operating characteristic (ROC) curve. It indicates the overall performance of a diagnostic test in terms of its accuracy at various diagnostic thresholds used to discriminate cases and non-cases of disease. The AUC measure is also used in meta-analyses, where each component study provides an estimate of the test sensitivity and specificity. These estimates are then combined to calculate a summary ROC (SROC) curve which describes the relationship between-test sensitivity and specificity across studies.The partial AUC has been proposed as an alternative measure to the full AUC. When using the partial AUC, one considers only those regions of the ROC space where data have been observed, or which correspond to clinically relevant values of test sensitivity or specificity. In this paper, we extend the idea of using the partial AUC to SROC curves in meta-analysis. Theoretical and numerical results describe the variation in the partial AUC and its standard error as a function of the degree of inter-study heterogeneity and of the extent of truncation applied to the ROC space. A scaled partial area measure is also proposed to restore the property that the summary measure should range from 0 to 1.The results suggest several disadvantages of the partial AUC measures. In contrast to earlier findings with the full AUC, the partial AUC is rather sensitive to heterogeneity. Comparisons between tests are more difficult, especially if an empirical truncation process is used. Finally, the partial area lacks a useful symmetry property enjoyed by the full AUC. Although the partial AUC may sometimes have clinical appeal, on balance the use of the full AUC is preferred.