Interval estimation for the difference in paired areas under the ROC curves in the absence of a gold standard test

Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard (GS) test on the true disease status is required to estimate the AUC. However, a GS test may sometimes be too expensive or infeasible. Therefore, in many medical research studies, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, using the expectation‐maximization (EM) algorithm in conjunction with a bootstrap method, we propose a maximum likelihood‐based procedure for the construction of confidence intervals for the difference in paired AUCs in the absence of a GS test. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and interval lengths. The proposed method is illustrated with two examples. Copyright © 2009 John Wiley & Sons, Ltd.     

医学诊断试验评价的ROC分析—重复测量诊断数据的ROC曲线

目的 探讨诊断试验中重复有序分类测量数据的ROC分析方法。方法 在有序模型基础上构造相应的ROC模型,然后选用累积logistic模型作为联系函数,采用GEE方法进行参数估计。结果 通过累积logistic模型的参数估计,能够得到ROC曲线方程、ROC曲线下面积及标准误。结论 上术方法影像诊断试验中常见的重复测量数据进行有效地分析与处理。     

Receiver operating characteristic analysis under tree orderings of disease classes

Receiver operating characteristic (ROC) curve and its summary statistics (e.g., the area under curve (AUC)) are commonly used to evaluate the diagnostic accuracy for disease processes with binary classification. The ROC curve has been extended to ROC surface for scenarios with three ordinal classes or to hyper‐surface for scenarios with more than three classes. For classifier under tree or umbrella ordering in which the marker measurement for one class is lower or higher than those for the other classes, the commonly adopted diagnostic measures are the naive AUC (NAUC) based on a pooled class of all the unordered classes and the umbrella volume (UV) based on the concept of volume under surface. However, both NAUC and UV have some limitations. For example, NAUC depends on the sampling weights for all the classes in population, and UV has only been introduced for three‐class settings. In this article, we initiate the idea of a new ROC framework for tree or umbrella ordering (denoted as TROC) and propose the area under TROC curve (denoted as TAUC) as an appropriate diagnostic measure. The proposed TROC and TAUC share many nice features with the traditional ROC and AUC. Both parametric and nonparametric approaches are explored to construct the confidence interval estimation of TAUC. The performances of these methods are compared in simulation studies under a variety settings. At the end, the proposed methods are applied to a published microarray data set. Copyright © 2015 John Wiley & Sons, Ltd.     

Direct estimation of the area under the receiver operating characteristic curve in the presence of verification bias

The area under a receiver operating characteristic (ROC) curve (AUC) is a commonly used index for summarizing the ability of a continuous diagnostic test to discriminate between healthy and diseased subjects. If all subjects have their true disease status verified, one can directly estimate the AUC nonparametrically using the Wilcoxon statistic. In some studies, verification of the true disease status is performed only for a subset of subjects, possibly depending on the result of the diagnostic test and other characteristics of the subjects. Because estimators of the AUC based only on verified subjects are typically biased, it is common to estimate the AUC from a bias-corrected ROC curve. The variance of the estimator, however, does not have a closed-form expression and thus resampling techniques are used to obtain an estimate. In this paper, we develop a new method for directly estimating the AUC in the setting of verification bias based on U-statistics and inverse probability weighting (IPW). Closed-form expressions for the estimator and its variance are derived. We also show that the new estimator is equivalent to the empirical AUC derived from the bias-corrected ROC curve arising from the IPW approach.     

A permutation test sensitive to differences in areas for comparing ROC curves from a paired design

The area under the receiver operating characteristic (ROC) curve (AUC) is a widely accepted summary index of the overall performance of diagnostic procedures and the difference between AUCs is often used when comparing two diagnostic systems. We developed an exact non-parametric statistical procedure for comparing two ROC curves in paired design settings. The test which is based on all permutations of the subject specific rank ratings is formally a test for equality of ROC curves that is sensitive to the alternatives of AUC difference. The operating characteristics of the proposed test were evaluated using extensive simulations over a wide range of parameters.The proposed procedure can be easily implemented in experimental ROC data sets. For small samples and for underlying parameters that are common in experimental studies in diagnostic imaging the test possesses good operating characteristics and is more powerful than the conventional non-parametric procedure for AUC comparisons.We also derived an asymptotic version of the test which uses an exact estimate of the variance in the permutation space and provides a good approximation even when the sample sizes are small. This asymptotic procedure is a simple and precise approximation to the exact test and is useful for large sample sizes where the exact test may be computationally burdensome.     

Direct estimation of the area under the receiver operating characteristic curve with verification biased data

In medical diagnostic studies, verification of the true disease status might be partially missing based on results of diagnostic tests and other characteristics of subjects. Because estimates of area under the ROC curve (AUC) based on partially validated subjects are usually biased, it is usually necessary to estimate AUC from a bias-corrected ROC curve. In this article, various direct estimation methods of the AUC based on hybrid imputation [full imputations and mean score imputation (MSI)], inverse probability weighting, and the semiparametric efficient (SPE) approach are proposed and compared in the presence of verification bias when the test result is continuous under the assumption that the true disease status, if missing, is missing at random. Simulation results show that the proposed estimators are accurate for the biased sampling if the disease and verification models are correctly specified. The SPE and MSI based estimators perform well even under the misspecified disease/verification models. Numerical studies are performed to compare the finite sample performance of the proposed approaches with existing methods. A real dataset of neonatal hearing screening study is analyzed.     

The partial area under the summary ROC curve

The area under the curve (AUC) is commonly used as a summary measure of the receiver operating characteristic (ROC) curve. It indicates the overall performance of a diagnostic test in terms of its accuracy at various diagnostic thresholds used to discriminate cases and non-cases of disease. The AUC measure is also used in meta-analyses, where each component study provides an estimate of the test sensitivity and specificity. These estimates are then combined to calculate a summary ROC (SROC) curve which describes the relationship between-test sensitivity and specificity across studies.The partial AUC has been proposed as an alternative measure to the full AUC. When using the partial AUC, one considers only those regions of the ROC space where data have been observed, or which correspond to clinically relevant values of test sensitivity or specificity. In this paper, we extend the idea of using the partial AUC to SROC curves in meta-analysis. Theoretical and numerical results describe the variation in the partial AUC and its standard error as a function of the degree of inter-study heterogeneity and of the extent of truncation applied to the ROC space. A scaled partial area measure is also proposed to restore the property that the summary measure should range from 0 to 1.The results suggest several disadvantages of the partial AUC measures. In contrast to earlier findings with the full AUC, the partial AUC is rather sensitive to heterogeneity. Comparisons between tests are more difficult, especially if an empirical truncation process is used. Finally, the partial area lacks a useful symmetry property enjoyed by the full AUC. Although the partial AUC may sometimes have clinical appeal, on balance the use of the full AUC is preferred.     

无金标准条件下含协变量的ROC曲线估计方法

目的阐明无金标准条件下,考虑协变量后估计ROC曲线的两部贝叶斯模型。方法介绍两部贝叶斯模型,结合实例,筛选无金标准条件下ROC曲线的影响因素,考虑协变量影响后,估计ROC曲线。结果两部贝叶斯模型不仅可探讨协变量对疾病状态的影响,而且可探讨协变量对诊断试验结果的影响,同时可计算不同协变量取值条件下ROC曲线下面积。结论两部贝叶斯模型可有效地解决无金标准条件下,考虑协变量影响的ROC曲线估计问题。     

Bayesian estimation of the receiver operating characteristic curve for a diagnostic test with a limit of detection in the absence of a gold standard

The receiver operating characteristic (ROC) curve is commonly used for evaluating the discriminatory ability of a biomarker. Measurements for a diagnostic test may be subject to an analytic limit of detection leading to immeasurable or unreportable test results. Ignoring the scores that are beyond the limit of detection of a test leads to a biased assessment of its discriminatory ability, as reflected by indices such as the associated area under the curve (AUC). We propose a Bayesian approach for the estimation of the ROC curve and its AUC for a test with a limit of detection in the absence of gold standard based on assumptions of normally and gamma‐distributed data. The methods are evaluated in simulation studies, and a truncated gamma model with a point mass is used to evaluate quantitative real‐time polymerase chain reaction data for bovine Johne's disease (paratuberculosis). Simulations indicated that estimates of diagnostic accuracy and AUC were good even for relatively small sample sizes (n=200). Exceptions were when there was a high per cent of unquantifiable results (60 per cent) or when AUC was ?0.6, which indicated a marked overlap between the outcomes in infected and non‐infected populations. Copyright © 2010 John Wiley & Sons, Ltd.     

A general regression methodology for ROC curve estimation

A method for applying generalized ordinal regression models to categorical rating data to estimate and analyze receiver operating characteristic (ROC) curves is presented. These models permit parsimonious adjustment of ROC curve parameters for relevant covariates through two regression equations that correspond to location and scale. Particular shapes of ROC curves are interpreted in relation to the kind of covariates included in the two regressions. The model is shown to be flexible because it is not restricted to the assumption of binormality that is commonly employed in smoothed ROC curve estimation, although the binormal model is one particular form of the more general model. The new method provides a mechanism for pinpointing the effect that interobserver variability has on the ROC curve. It also allows for the adjustment of ROC curves for temporal variation and case mix, and provides a way to assess the incremental diagnostic value of a test. The new methodology is recommended because it substantially improves the ability to assess diagnostic tests using ROC curves.     

Bayesian bootstrap estimation of ROC curve

Receiver operating characteristic (ROC) curve is widely applied in measuring discriminatory ability of diagnostic or prognostic tests. This makes the ROC analysis one of the most active research areas in medical statistics. Many parametric and semiparametric estimation methods have been proposed for estimating the ROC curve and its functionals. In this paper, we propose the Bayesian bootstrap (BB), a fully nonparametric estimation method, for the ROC curve and its functionals, such as the area under the curve (AUC). The BB method offers a bandwidth-free smoothing approach to the empirical estimate, and gives credible bounds. The accuracy of the estimate of the ROC curve in the simulation studies is examined by the integrated absolute error. In comparison with other existing curve estimation methods, the BB method performs well in terms of accuracy, robustness and simplicity. We also propose a procedure based on the BB approach to test the binormality assumption.     

具有协变量或干扰因素的诊断试验数据的ROC分析

目的 探讨具有协变量或干扰因素的诊断试验的评价问题。建立相应的统计模型及分析方法。方法 基于ROC分析给出变量ROC模型，采用有序logistic连续函数，利用SAS软件进行参数估计，得到有协为量情况下的ROC曲线方程及曲线下面积。文中结合检测动脉硬化的数据，说明了其用法。结果 采用文中给出的方法，可以有效地扣除协变量的影响，准确地评价诊断的作用。结论 本文提供的方法能够有效地解决诊断试验中具有干扰因素影响的问题。     

Estimation of ROC curve with complex survey data

The receiver operating characteristic (ROC) curve can be utilized to evaluate the performance of diagnostic tests. The area under the ROC curve (AUC) is a widely used summary index for comparing multiple ROC curves. Both parametric and nonparametric methods have been developed to estimate and compare the AUCs. However, these methods are usually only applicable to data collected from simple random samples and not surveys and epidemiologic studies that use complex sample designs such as stratified and/or multistage cluster sampling with sample weighting. Such complex samples can inflate variances from intra‐cluster correlation and alter the expectations of test statistics because of the use of sample weights that account for differential sampling rates. In this paper, we modify the nonparametric method to incorporate sampling weights to estimate the AUC and employ leaving‐one‐out jackknife methods along with the balanced repeated replication method to account for the effects of the complex sampling in the variance estimation of our proposed estimators of the AUC. The finite sample properties of our methods are evaluated using simulations, and our methods are illustrated by comparing the estimated AUC for predicting overweight/obesity using different measures of body weight and adiposity among sampled children and adults in the US Hispanic Health and Nutrition Examination Survey. Copyright © 2014 John Wiley & Sons, Ltd.     

Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data

The summary receiver operating characteristic (SROC) curve has been recommended to represent the performance of a diagnostic test, based on data from a meta-analysis. However, little is known about the basic properties of the SROC curve or its estimate. In this paper, the position of the SROC curve is characterized in terms of the overall diagnostic odds ratio and the magnitude of inter-study heterogeneity in the odds ratio. The area under the curve (AUC) and an index Q(*) are discussed as potentially useful summaries of the curve. It is shown that AUC is maximized when the study odds ratios are homogeneous, and that it is quite robust to heterogeneity. An upper bound is derived for AUC based on an exact analytic expression for the homogeneous situation, and a lower bound based on the limit case Q(*), defined by the point where sensitivity equals specificity: Q(*) is invariant to heterogeneity. The standard error of AUC is derived for homogeneous studies, and shown to be a reasonable approximation with heterogeneous studies. The expressions for AUC and its standard error are easily computed in the homogeneous case, and avoid the need for numerical integration in the more general case. SE(AUC) and SE(Q(*)) are found to be numerically close, with SE(Q(*)) being larger if the odds ratio is very large. The methods are illustrated using data for the Pap smear screening test for cervical cancer, and for three tests for the diagnosis of metastases in cervical cancer patients.     

Bayesian modeling and inference for diagnostic accuracy and probability of disease based on multiple diagnostic biomarkers with and without a perfect reference standard

The area under the receiver operating characteristic (ROC) curve (AUC) is used as a performance metric for quantitative tests. Although multiple biomarkers may be available for diagnostic or screening purposes, diagnostic accuracy is often assessed individually rather than in combination. In this paper, we consider the interesting problem of combining multiple biomarkers for use in a single diagnostic criterion with the goal of improving the diagnostic accuracy above that of an individual biomarker. The diagnostic criterion created from multiple biomarkers is based on the predictive probability of disease, conditional on given multiple biomarker outcomes. If the computed predictive probability exceeds a specified cutoff, the corresponding subject is allocated as ‘diseased’. This defines a standard diagnostic criterion that has its own ROC curve, namely, the combined ROC (cROC). The AUC metric for cROC, namely, the combined AUC (cAUC), is used to compare the predictive criterion based on multiple biomarkers to one based on fewer biomarkers. A multivariate random‐effects model is proposed for modeling multiple normally distributed dependent scores. Bayesian methods for estimating ROC curves and corresponding (marginal) AUCs are developed when a perfect reference standard is not available. In addition, cAUCs are computed to compare the accuracy of different combinations of biomarkers for diagnosis. The methods are evaluated using simulations and are applied to data for Johne's disease (paratuberculosis) in cattle. Copyright © 2015 John Wiley & Sons, Ltd.     

基于Matlab的受试者操作特征曲线分析

目的介绍新开发的受试者操作特征（Receiver Operating Characteristic,ROC）曲线分析软件。方法根据非参数法ROC曲线分析的基本原理,利用Matlab编写可视化的非参数法ROC曲线分析软件,对文献中的数据分别用本软件、SPSS及Analyse-it软件进行验证。最后利用本软件对两种技术检测的血清钠水平诊断洛矶山斑疹热（RMSF）的诊断作用进行评价和比较。结果本软件提供了非参数法ROC曲线分析的基本功能,能够直接输入或导入诊断试验数据,计算ROC曲线下面积并进行比较,并能保存ROC曲线操作点。与其他两个软件及文献对验证数据的ROC曲线分析结果完全相同,对应用实例进行了恰当的分析。结论本软件具有较完整的非参数法ROC曲线分析功能,在实际工作中具有一定的实用价值。     

Group sequential design for comparative diagnostic accuracy studies

In the field of diagnostic medicine, comparative clinical trials are necessary for assessing the utility of one diagnostic test over another. The area under the receiver operating characteristic (ROC) curve, commonly referred to as AUC, is a general measure of a test's inherent ability to distinguish between patients with and without a condition. Standardized AUC difference is the most frequently used statistic for comparing two diagnostic tests. In therapeutic comparative clinical trials with sequential patient entry, fixed sample design (FSD) is unjustified on ethical and economical grounds and group sequential design (GSD) is frequently used. In this paper, we argue that the same reasoning exists for the comparative clinical trials in diagnostic medicine and hence GSD should be utilized in this field for designing trials. Since computation of the stopping boundaries of GSD and data analysis after a group sequential test rely heavily on Brownian motion approximation, we derive the asymptotic distribution of the standardized AUC difference statistic and point out its resemblance to the Brownian motion. Boundary determination and sample size calculation are then illustrated through an example from a cancer clinical trial.     

临床试验评价的ROC分析方法

目的探讨新药临床试验效果的评价问题,给出一种新的统计分析方法。方法基于ROC分析给出多变量ROC模型,采用有序logit联系函数,利用SAS软件进行参数估计,得到有协变量及交互作用情况下的ROC曲线方程及曲线下面积。结果采用文中给出的方法,可以有效地扣除协变量的影响,用ROC曲线直观地评价药物之间的差别和作用。结论本文提供的方法能够更有效地对临床试验做出客观和准确的评价。     

A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets

The receiver operating characteristic (ROC) curve is a statistical tool for evaluating the accuracy of diagnostic tests. Investigators often compare the validity of two tests based on the estimated areas under the respective ROC curves. However, the traditional way of comparing entire areas under two ROC curves is not sensitive when two ROC curves cross each other. Also, there are some cutpoints on the ROC curves that are not considered in practice because their corresponding sensitivities or specificities are unacceptable. For the purpose of comparing the partial area under the curve (AUC) within a specific range of specificity for two correlated ROC curves, a non-parametric method based on Mann-Whitney U-statistics has been developed. The estimation of AUC along with its estimated variance and covariance is simplified by a method of grouping the observations according to their cutpoint values. The method is used to evaluate alternative logistic regression models that predict whether a subject has incident breast cancer based on information in Medicare claims data.     

连续变量诊断试验数据的ROC分析

目的 介绍一种连续诊断变量的ROC回归模型，以及在独立和相关结构下的参数估计方法，给出参数误差估计的Bootstrap方法。方法 应用SAS软件中的GENMOD过程和SAS语言编写的程序实现上述过程，并通过实例说明其应用效果。结果 利用ROC曲线方程，可以扣除协变量对诊断试验结果评价的影响，并能够计算出在不同协变量取值下的ROC曲线下面积，提供更为丰富和可靠的信息。结论 文中给出的ROC回归模型可以有效地用于连续变量诊断试验数据的ROC分析。