Direct estimation of the area under the receiver operating characteristic curve with verification biased data

In medical diagnostic studies, verification of the true disease status might be partially missing based on results of diagnostic tests and other characteristics of subjects. Because estimates of area under the ROC curve (AUC) based on partially validated subjects are usually biased, it is usually necessary to estimate AUC from a bias-corrected ROC curve. In this article, various direct estimation methods of the AUC based on hybrid imputation [full imputations and mean score imputation (MSI)], inverse probability weighting, and the semiparametric efficient (SPE) approach are proposed and compared in the presence of verification bias when the test result is continuous under the assumption that the true disease status, if missing, is missing at random. Simulation results show that the proposed estimators are accurate for the biased sampling if the disease and verification models are correctly specified. The SPE and MSI based estimators perform well even under the misspecified disease/verification models. Numerical studies are performed to compare the finite sample performance of the proposed approaches with existing methods. A real dataset of neonatal hearing screening study is analyzed.     

Direct estimation of the area under the receiver operating characteristic curve in the presence of verification bias

The area under a receiver operating characteristic (ROC) curve (AUC) is a commonly used index for summarizing the ability of a continuous diagnostic test to discriminate between healthy and diseased subjects. If all subjects have their true disease status verified, one can directly estimate the AUC nonparametrically using the Wilcoxon statistic. In some studies, verification of the true disease status is performed only for a subset of subjects, possibly depending on the result of the diagnostic test and other characteristics of the subjects. Because estimators of the AUC based only on verified subjects are typically biased, it is common to estimate the AUC from a bias-corrected ROC curve. The variance of the estimator, however, does not have a closed-form expression and thus resampling techniques are used to obtain an estimate. In this paper, we develop a new method for directly estimating the AUC in the setting of verification bias based on U-statistics and inverse probability weighting (IPW). Closed-form expressions for the estimator and its variance are derived. We also show that the new estimator is equivalent to the empirical AUC derived from the bias-corrected ROC curve arising from the IPW approach.     

Estimating the receiver operating characteristic curve in matched case control studies

The matched case-control design is frequently used in the study of complex disorders and can result in significant gains in efficiency, especially in the context of measuring biomarkers; however, risk prediction in this setting is not straightforward. We propose an inverse-probability weighting approach to estimate the predictive ability associated with a set of covariates. In particular, we propose an algorithm for estimating the summary index, area under the curve corresponding to the Receiver Operating Characteristic curve associated with a set of pre-defined covariates for predicting a binary outcome. By combining data from the parent cohort with that generated in a matched case control study, we describe methods for estimation of the population parameters of interest and the corresponding area under the curve. We evaluate the bias associated with the proposed methods in simulations by considering a range of parameter settings. We illustrate the methods in two data applications: (1) a prospective cohort study of cardiovascular disease in women, the Women's Health Study, and (2) a matched case-control study nested within the Nurses' Health Study aimed at risk prediction of invasive breast cancer.     

Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data

The summary receiver operating characteristic (SROC) curve has been recommended to represent the performance of a diagnostic test, based on data from a meta-analysis. However, little is known about the basic properties of the SROC curve or its estimate. In this paper, the position of the SROC curve is characterized in terms of the overall diagnostic odds ratio and the magnitude of inter-study heterogeneity in the odds ratio. The area under the curve (AUC) and an index Q(*) are discussed as potentially useful summaries of the curve. It is shown that AUC is maximized when the study odds ratios are homogeneous, and that it is quite robust to heterogeneity. An upper bound is derived for AUC based on an exact analytic expression for the homogeneous situation, and a lower bound based on the limit case Q(*), defined by the point where sensitivity equals specificity: Q(*) is invariant to heterogeneity. The standard error of AUC is derived for homogeneous studies, and shown to be a reasonable approximation with heterogeneous studies. The expressions for AUC and its standard error are easily computed in the homogeneous case, and avoid the need for numerical integration in the more general case. SE(AUC) and SE(Q(*)) are found to be numerically close, with SE(Q(*)) being larger if the odds ratio is very large. The methods are illustrated using data for the Pap smear screening test for cervical cancer, and for three tests for the diagnosis of metastases in cervical cancer patients.     

Small area estimation of receiver operating characteristic curves for ordinal data under stochastic ordering

There has been a recent increase in the diagnosis of diseases through radiographic images such as x-rays, magnetic resonance imaging, and computed tomography. The outcome of a radiological diagnostic test is often in the form of discrete ordinal data, and we usually summarize the performance of the diagnostic test using the receiver operating characteristic (ROC) curve and the area under the curve (AUC). The ROC curve will be concave and called proper when the outcomes of the diagnostic test in the actually positive subjects are higher than in the actually negative subjects. The diagnostic test for disease detection is clinically useful when a ROC curve is proper. In this study, we develop a hierarchical Bayesian model to estimate the proper ROC curve and AUC using stochastic ordering in several domains when the outcome of the diagnostic test is discrete ordinal data and compare it with the model without stochastic ordering. The model without stochastic ordering can estimate the improper ROC curve with a nonconcave shape or a hook when the true ROC curve of the population is a proper ROC curve. Therefore, the model with stochastic ordering is preferable over the model without stochastic ordering to estimate the proper ROC curve with clinical usefulness for ordinal data.     

Small improvement in the area under the receiver operating characteristic curve indicated small changes in predicted risks

ObjectiveAdding risk factors to a prediction model often increases the area under the receiver operating characteristic curve (AUC) only slightly, particularly when the AUC of the model was already high. We investigated whether a risk factor that minimally improves the AUC may nevertheless improve the predictive ability of the model, assessed by integrated discrimination improvement (IDI).Study Design and SettingWe simulated data sets with risk factors and event status for 100,000 hypothetical individuals and created prediction models with AUCs between 0.50 and 0.95. We added a single risk factor for which the effect was modeled as a certain odds ratio (OR 2, 4, 8) or AUC increment (ΔAUC 0.01, 0.02, 0.03).ResultsAcross all AUC values of the baseline model, for a risk factor with the same OR, both ΔAUC and IDI were lower when the AUC of the baseline model was higher. When the increment in AUC was small (ΔAUC 0.01), the IDI was also small, except when the AUC of the baseline model was >0.90.ConclusionWhen the addition of a risk factor shows minimal improvement in AUC, predicted risks generally show minimal changes too. Updating risk models with strong risk factors may be informative for a subgroup of individuals, but not at the population level. The AUC may not be as insensitive as is frequently argued.     

Bayesian ROC curve estimation under verification bias

Receiver operating characteristic (ROC) curve has been widely used in medical science for its ability to measure the accuracy of diagnostic tests under the gold standard. However, in a complicated medical practice, a gold standard test can be invasive, expensive, and its result may not always be available for all the subjects under study. Thus, a gold standard test is implemented only when it is necessary and possible. This leads to the so‐called ‘verification bias’, meaning that subjects with verified disease status (also called label) are not selected in a completely random fashion. In this paper, we propose a new Bayesian approach for estimating an ROC curve based on continuous data following the popular semiparametric binormal model in the presence of verification bias. By using a rank‐based likelihood, and following Gibbs sampling techniques, we compute the posterior distribution of the binormal parameters intercept and slope, as well as the area under the curve by imputing the missing labels within Markov Chain Monte‐Carlo iterations. Consistency of the resulting posterior under mild conditions is also established. We compare the new method with other comparable methods and conclude that our estimator performs well in terms of accuracy. Copyright © 2014 John Wiley & Sons, Ltd.     

ROC曲线下面积的ML估计与假设检验

目的 探讨诊断试验中配对设计资料的ROC分析方法。方法 在双正态模型基础下应用ML估计方法计算ROC曲线下面积，正态近似法估计面积的可信区间及假设检验。结果 由迭代法进行参数估计，得到ROC曲线下的面积、面积的标准误及置信区间，可计算出面积比较的U检验统计量。结论 可用于配对设计的诊断试验的比较和评价，包括对连续性和等级分类资料的处理。     

The partial area under the summary ROC curve

The area under the curve (AUC) is commonly used as a summary measure of the receiver operating characteristic (ROC) curve. It indicates the overall performance of a diagnostic test in terms of its accuracy at various diagnostic thresholds used to discriminate cases and non-cases of disease. The AUC measure is also used in meta-analyses, where each component study provides an estimate of the test sensitivity and specificity. These estimates are then combined to calculate a summary ROC (SROC) curve which describes the relationship between-test sensitivity and specificity across studies.The partial AUC has been proposed as an alternative measure to the full AUC. When using the partial AUC, one considers only those regions of the ROC space where data have been observed, or which correspond to clinically relevant values of test sensitivity or specificity. In this paper, we extend the idea of using the partial AUC to SROC curves in meta-analysis. Theoretical and numerical results describe the variation in the partial AUC and its standard error as a function of the degree of inter-study heterogeneity and of the extent of truncation applied to the ROC space. A scaled partial area measure is also proposed to restore the property that the summary measure should range from 0 to 1.The results suggest several disadvantages of the partial AUC measures. In contrast to earlier findings with the full AUC, the partial AUC is rather sensitive to heterogeneity. Comparisons between tests are more difficult, especially if an empirical truncation process is used. Finally, the partial area lacks a useful symmetry property enjoyed by the full AUC. Although the partial AUC may sometimes have clinical appeal, on balance the use of the full AUC is preferred.     

Evaluating diagnostic tests: The area under the ROC curve and the balance of errors

Because accurate diagnosis lies at the heart of medicine, it is important to be able to evaluate the effectiveness of diagnostic tests. A variety of accuracy measures are used. One particularly widely used measure is the AUC, the area under the receiver operating characteristic (ROC) curve. This measure has a well‐understood weakness when comparing ROC curves which cross. However, it also has the more fundamental weakness of failing to balance different kinds of misdiagnoses effectively. This is not merely an aspect of the inevitable arbitrariness in choosing a performance measure, but is a core property of the way the AUC is defined. This property is explored, and an alternative, the H measure, is described. Copyright © 2010 John Wiley & Sons, Ltd.     

A Visicalc program for estimating the area under a receiver operating characteristic (ROC) curve

The area under the ROC curve interests us as a method for analyzing discrimination or detectability. One can assess a diagnostic test or probability assessor with respect to its degree of discrimination. The area under the ROC curve gives us the probability of correctly identifying abnormal from normal in a forced-choice, two-alternative problem. Previous methods used for calculating the area involved maximum likelihood estimation on a mainframe or minicomputer. This paper demonstrates an adaptation of a recently published nonparametric method for estimating the area. This adaptation takes advantage of electronic spreadsheet software and may be used on most (if not all) microcomputers. The paper develops the construction of the program needed for the necessary calculations.     

Assessing agreement with relative area under the coverage probability curve

There has been substantial statistical literature in the last several decades on assessing agreement, and coverage probability approach was selected as a preferred index for assessing and improving measurement agreement in a core laboratory setting. With this approach, a satisfactory agreement is based on pre‐specified high satisfactory coverage probability (e.g., 95%), given one pre‐specified acceptable difference. In practice, we may want to have quality control on more than one pre‐specified differences, or we may simply want to summarize the agreement based on differences up to a maximum acceptable difference. We propose to assess agreement via the coverage probability curve that provides a full spectrum of measurement error at various differences/disagreement. Relative area under the coverage probability curve is proposed for the summary of overall agreement, and this new summary index can be used for comparison of different intra‐methods or inter‐methods/labs/observers' agreement. Simulation studies and a blood pressure example are used for illustration of the methodology. Copyright © 2016 John Wiley & Sons, Ltd.     

An evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve

The area under the receiver operating characteristic (ROC) curve serves as one means for evaluating the performance of diagnostic and predictive test systems. The most commonly used method for estimating the area under an ROC curve utilizes the maximum-likelihood-estimation technique, and a nonparametric method to calculate the area under an ROC curve was recently described. We compared the performance of these two methods. The results for the area under the ROC curve and the standard error of the estimate as calculated by each of the two methods exhibited high correlation. Generally, the nonparametric method yields lower area estimates than the maximum-likelihood-estimation technique. However, these differences generally were small, particularly with ROC curves derived from five or more cutoff points. Consistent results of hypothesis testing of the significance of differences between two ROC curves will be similar, regardless of which method is used, as long as one uses the same estimation technique on the two curves and as long as the two ROC curves being compared are of similar shape.     

Advantages to transforming the receiver operating characteristic (ROC) curve into likelihood ratio co-ordinates

Traditionally, the receiver operating characteristic (ROC) curve for a diagnostic test plots true positives (sensitivity) against false positives (one minus specificity). However, this representation brings with it several drawbacks. A transformation to positive and negative likelihood ratio co-ordinates, scaled by base-ten logarithms, offers several advantages. First we motivate the use of positive and negative likelihood ratios, emphasizing their relationship to modification of the odds ratio. Then we highlight properties of likelihood ratios using the traditional ROC axes. Finally, we demonstrate ROC curves and their properties after conversion to likelihood ratio co-ordinates. These graphs do not waste space for tests lacking diagnostic power, and offer a simple visual assessment of a test's impact on the odds ratio.     

Receiver operating characteristic analysis under tree orderings of disease classes

Receiver operating characteristic (ROC) curve and its summary statistics (e.g., the area under curve (AUC)) are commonly used to evaluate the diagnostic accuracy for disease processes with binary classification. The ROC curve has been extended to ROC surface for scenarios with three ordinal classes or to hyper‐surface for scenarios with more than three classes. For classifier under tree or umbrella ordering in which the marker measurement for one class is lower or higher than those for the other classes, the commonly adopted diagnostic measures are the naive AUC (NAUC) based on a pooled class of all the unordered classes and the umbrella volume (UV) based on the concept of volume under surface. However, both NAUC and UV have some limitations. For example, NAUC depends on the sampling weights for all the classes in population, and UV has only been introduced for three‐class settings. In this article, we initiate the idea of a new ROC framework for tree or umbrella ordering (denoted as TROC) and propose the area under TROC curve (denoted as TAUC) as an appropriate diagnostic measure. The proposed TROC and TAUC share many nice features with the traditional ROC and AUC. Both parametric and nonparametric approaches are explored to construct the confidence interval estimation of TAUC. The performances of these methods are compared in simulation studies under a variety settings. At the end, the proposed methods are applied to a published microarray data set. Copyright © 2015 John Wiley & Sons, Ltd.     

Estimation of time-dependent area under the ROC curve for long-term risk prediction

Sensitivity, specificity, and area under the ROC curve (AUC) are often used to measure the ability of survival models to predict future risk. Estimation of these parameters is complicated by the fact that these parameters are time-dependent and by the fact that censoring affects their estimation just as it affects estimation of survival curves or coefficients of survival regression models. The authors present several estimators that overcome these complications. One approach is a recursive calculation over the ordered times of events, analogous to the Kaplan-Meier approach to survival function estimation. Another is to first apply Bayes' theorem to write the parameters of interest in terms of conditional survival functions that are then estimated by survival analysis methods. Simulation studies demonstrate that the proposed estimators perform well in practical situations, when compared with an estimator (c-statistic, from logistic regression) that ignores time. An illustration with data from a cardiovascular follow-up study is provided.     

Evaluating classification performance of biomarkers in two-phase case-control studies

Biomarkers are playing an increasingly important role in disease screening, early detection, and risk prediction. The two-phase case-control sampling study design is widely used for the evaluation of candidate biomarkers. The sampling probabilities for cases and controls in the second phase can often depend on other covariates (sampling strata). This biased sampling can lead to invalid inference on a biomarker's classification accuracy if not properly accounted for. In this paper, we adopt the idea of inverse probability weighting and develop inverse probability weighting–based estimators for various measures of a biomarker's classification performance, including the points on the receiver operating characteristics (ROCs) curve, the area under the ROC curve (area under the curve), and the partial area under the curve. In particular, we consider classification accuracy estimators using sampling weights estimated conditionally on sampling strata and further improve their efficiency through the use of estimated weights that additionally take into account the auxiliary variables available from the phase-one cohort. We develop asymptotic properties of the proposed estimators and provide analytical variance for making inference. Extensive simulation studies demonstrate excellent performance of the proposed weighted estimators, while the traditional empirical estimator can be severely biased. We also investigate the advantages in efficiency gain for estimating various classification accuracy estimators through the use of auxiliary variables in addition to sampling strata and apply the proposed method to examples from a renal artery stenosis study and a prostate cancer study.     

Interval estimation for the area under the receiver operating characteristic curve when data are subject to error

The area (A) under the receiver operating characteristic curve is commonly used to quantify the ability of a biomarker to correctly classify individuals into two populations. However, many markers are subject to measurement error, which must be accounted for to prevent understating their effectiveness. In this paper, we develop a new confidence interval procedure for A which is adjusted for measurement error using either external or internal replicated measurements. Based on the observation that A is a function of normal means and variances, we develop the procedure by recovering variance estimates needed from confidence limits for normal means and variances. Simulation results show that the procedure performs better than the previous ones based on the delta‐method in terms of coverage percentage, balance of tail errors and interval width. Two examples are presented. Copyright © 2010 John Wiley & Sons, Ltd.     

Bayesian estimation of the receiver operating characteristic curve for a diagnostic test with a limit of detection in the absence of a gold standard

The receiver operating characteristic (ROC) curve is commonly used for evaluating the discriminatory ability of a biomarker. Measurements for a diagnostic test may be subject to an analytic limit of detection leading to immeasurable or unreportable test results. Ignoring the scores that are beyond the limit of detection of a test leads to a biased assessment of its discriminatory ability, as reflected by indices such as the associated area under the curve (AUC). We propose a Bayesian approach for the estimation of the ROC curve and its AUC for a test with a limit of detection in the absence of gold standard based on assumptions of normally and gamma‐distributed data. The methods are evaluated in simulation studies, and a truncated gamma model with a point mass is used to evaluate quantitative real‐time polymerase chain reaction data for bovine Johne's disease (paratuberculosis). Simulations indicated that estimates of diagnostic accuracy and AUC were good even for relatively small sample sizes (n=200). Exceptions were when there was a high per cent of unquantifiable results (60 per cent) or when AUC was ?0.6, which indicated a marked overlap between the outcomes in infected and non‐infected populations. Copyright © 2010 John Wiley & Sons, Ltd.