无金标准条件下含协变量的ROC曲线估计方法

目的阐明无金标准条件下,考虑协变量后估计ROC曲线的两部贝叶斯模型。方法介绍两部贝叶斯模型,结合实例,筛选无金标准条件下ROC曲线的影响因素,考虑协变量影响后,估计ROC曲线。结果两部贝叶斯模型不仅可探讨协变量对疾病状态的影响,而且可探讨协变量对诊断试验结果的影响,同时可计算不同协变量取值条件下ROC曲线下面积。结论两部贝叶斯模型可有效地解决无金标准条件下,考虑协变量影响的ROC曲线估计问题。     

A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets

The receiver operating characteristic (ROC) curve is a statistical tool for evaluating the accuracy of diagnostic tests. Investigators often compare the validity of two tests based on the estimated areas under the respective ROC curves. However, the traditional way of comparing entire areas under two ROC curves is not sensitive when two ROC curves cross each other. Also, there are some cutpoints on the ROC curves that are not considered in practice because their corresponding sensitivities or specificities are unacceptable. For the purpose of comparing the partial area under the curve (AUC) within a specific range of specificity for two correlated ROC curves, a non-parametric method based on Mann-Whitney U-statistics has been developed. The estimation of AUC along with its estimated variance and covariance is simplified by a method of grouping the observations according to their cutpoint values. The method is used to evaluate alternative logistic regression models that predict whether a subject has incident breast cancer based on information in Medicare claims data.     

具有协变量或干扰因素的诊断试验数据的ROC分析

目的 探讨具有协变量或干扰因素的诊断试验的评价问题。建立相应的统计模型及分析方法。方法 基于ROC分析给出变量ROC模型，采用有序logistic连续函数，利用SAS软件进行参数估计，得到有协为量情况下的ROC曲线方程及曲线下面积。文中结合检测动脉硬化的数据，说明了其用法。结果 采用文中给出的方法，可以有效地扣除协变量的影响，准确地评价诊断的作用。结论 本文提供的方法能够有效地解决诊断试验中具有干扰因素影响的问题。     

连续变量诊断试验数据的ROC分析

目的 介绍一种连续诊断变量的ROC回归模型，以及在独立和相关结构下的参数估计方法，给出参数误差估计的Bootstrap方法。方法 应用SAS软件中的GENMOD过程和SAS语言编写的程序实现上述过程，并通过实例说明其应用效果。结果 利用ROC曲线方程，可以扣除协变量对诊断试验结果评价的影响，并能够计算出在不同协变量取值下的ROC曲线下面积，提供更为丰富和可靠的信息。结论 文中给出的ROC回归模型可以有效地用于连续变量诊断试验数据的ROC分析。     

Combining biomarkers for classification with covariate adjustment

Combining multiple markers can improve classification accuracy compared with using a single marker. In practice, covariates associated with markers or disease outcome can affect the performance of a biomarker or biomarker combination in the population. The covariate‐adjusted receiver operating characteristic (ROC) curve has been proposed as a tool to tease out the covariate effect in the evaluation of a single marker; this curve characterizes the classification accuracy solely because of the marker of interest. However, research on the effect of covariates on the performance of marker combinations and on how to adjust for the covariate effect when combining markers is still lacking. In this article, we examine the effect of covariates on classification performance of linear marker combinations and propose to adjust for covariates in combining markers by maximizing the nonparametric estimate of the area under the covariate‐adjusted ROC curve. The proposed method provides a way to estimate the best linear biomarker combination that is robust to risk model assumptions underlying alternative regression‐model‐based methods. The proposed estimator is shown to be consistent and asymptotically normally distributed. We conduct simulations to evaluate the performance of our estimator in cohort and case/control designs and compare several different weighting strategies during estimation with respect to efficiency. Our estimator is also compared with alternative regression‐model‐based estimators or estimators that maximize the empirical area under the ROC curve, with respect to bias and efficiency. We apply the proposed method to a biomarker study from an human immunodeficiency virus vaccine trial. Copyright © 2017 John Wiley & Sons, Ltd.     

A new parametric method based on S-distributions for computing receiver operating characteristic curves for continuous diagnostic tests

Receiver operating characteristic (ROC) curves provides a method for evaluating the performance of a diagnostic test. These curves represent the true positive ratio, that is, the true positives among those affected by the disease, as a function of the false positive ratio, that is, the false positives among the healthy, corresponding to each possible value of the diagnostic variable. When the diagnostic variable is continuous, the corresponding ROC curve is also continuous. However, estimation of such curve through the analysis of sample data yields a step-line, unless some assumption is made on the underlying distribution of the considered variable. Since the actual distribution of the diagnostic test is seldom known, it is difficult to select an appropriate distribution for practical use. Data transformation may offer a solution but also may introduce a distortion on the evaluation of the diagnostic test. In this paper we show that the distribution family known as the S-distribution can be used to solve this problem. The S-distribution is defined as a differential equation in which the dependent variable is the cumulative. This special form provides a highly flexible family of distributions that can be used as models for unknown distributions. It has been shown that classical statistical distributions can be represented accurately as S-distributions and that they occur in a definite subspace of the parameter space corresponding to the whole S-distribution family. Consequently, many other distributional forms that do not correspond to known distributions are provided by the S-distribution. This property can be used to model observed data for unknown distributions and is very useful in constructing parametric ROC curves in those cases. After fitting an S-distribution to the observed samples of diseased and healthy populations, ROC curve computation is straightforward. A ROC curve can be considered as the solution of a differential equation in which the dependent variable is the ratio of true positives and the independent variable is the ratio of false positives. This equation can be easily obtained from the S-distributions fitted to observed data. Using these results, we can compute pointwise confidence bands for the ROC curve and the corresponding area under the curve. We shall compare this approach with the empirical and the binormal methods for estimating a ROC curve to show that the S-distribution based method is a useful parametric procedure.     

Predicting Elderly at Risk of Increased Future Healthcare Use: How Much Does Diagnostic Information Add to Prior Utilization?

We determined whether case-mix information from administrative data can identify those likely to be high users of healthcare in the following year. An individual's healthcare utilization equaled the number of days (between 1 and 365) during the year on which an individual received inpatient or outpatient services. A binary outcome was defined as using 92 days or more (i.e., being in the top 2%) in year two. We included case-mix data in the models from two risk adjustment systems, Adjusted Diagnostic Groups (ADGs) from Adjusted Clinical Groups and Hierarchical Condition Categories (HCCs) from Diagnostic Cost Groups. We examined three types of logistic regression models: (1) prior use models (year one utilization plus age and sex), (2) diagnostic models (HCCs and ADGs as dummy variables plus age and sex), and (3) combined models (prior use plus diagnostic models). For the models with the best c-statistics (i.e., area under the receiver operating characteristic (ROC) curve), we compared ROC curve plots. We also fit linear regression models and compared their sensitivity and specificity to the logistic models.Although diagnostic and prior use models performed comparably, the models with the best ROC curves in predicting high users of healthcare combined prior use and diagnostic information. Logistic and linear regression models discriminated between cases similarly. While prior utilization has traditionally been used to predict future healthcare use, we found that case-mix information may be as important as prior use in identifying those who may be the highest users of healthcare in the future.     

Comparing ROC curves derived from regression models

In constructing predictive models, investigators frequently assess the incremental value of a predictive marker by comparing the ROC curve generated from the predictive model including the new marker with the ROC curve from the model excluding the new marker. Many commentators have noticed empirically that a test of the two ROC areas often produces a non‐significant result when a corresponding Wald test from the underlying regression model is significant. A recent article showed using simulations that the widely used ROC area test produces exceptionally conservative test size and extremely low power. In this article, we demonstrate that both the test statistic and its estimated variance are seriously biased when predictions from nested regression models are used as data inputs for the test, and we examine in detail the reasons for these problems. Although it is possible to create a test reference distribution by resampling that removes these biases, Wald or likelihood ratio tests remain the preferred approach for testing the incremental contribution of a new marker. Copyright © 2012 John Wiley & Sons, Ltd.     

Adjusting the generalized ROC curve for covariates

Receiver operating characteristic (ROC) curves and in particular the area under the curve (AUC), are widely used to examine the effectiveness of diagnostic markers. Diagnostic markers and their corresponding ROC curves can be strongly influenced by covariate variables. When several diagnostic markers are available, they can be combined by a best linear combination such that the area under the ROC curve of the combination is maximized among all possible linear combinations. In this paper we discuss covariate effects on this linear combination assuming that the multiple markers, possibly transformed, follow a multivariate normal distribution. The ROC curve of this linear combination when markers are adjusted for covariates is estimated and approximate confidence intervals for the corresponding AUC are derived. An example of two biomarkers of coronary heart disease for which covariate information on age and gender is available is used to illustrate this methodology.     

Goodness-of-fit issues in ROC curve estimation.

Zhou recently considered goodness-of-fit (GOF) testing for receiver operating characteristic (ROC) curves estimated by applying the binormal and other bidistributional models to rating method data. He interpreted significant GOF tests as evidence that different decision thresholds were applied to diseased and nondiseased subjects and concluded that, in such circumstances, an ROC curve does not exist. In this article the author demonstrates that the GOF test accommodates many alternative hypotheses and that a significant test result need not be equated with an interaction between disease status and decision criteria or with non-existence of an ROC curve. He develops a new family of ROC curves based on a fully parameterized bidistributional model. The family includes the binormal ROC curve, but generalizes its structure by signifying all identifiable deviations in parameters of the latent distributions that define the model. The family provides a unified framework of alternative models to the binormal assumption and of alternative hypotheses to the GOF test for that assumption.     

ROC methodology within a monitoring framework

Receiver operating characteristic (ROC) methodology is widely used to evaluate and compare diagnostic tests. Generally, each diagnostic test is applied once to each subject in a population and the results, reported on a continuous scale, are used to construct the ROC curve. We extend the standard method to accommodate a framework in which the diagnostic test is repeated over time to monitor for occurrence of an event. Unlike the usual situation in which event status is static, the problem we address involves event status that is not constant over the monitoring period. Subjects generally are classified as non-events, or controls, until they experience events that convert them to cases. Viewing the data as incomplete discrete failure time data with time-varying covariates, potentially useful diagnostic markers can be related appropriately in time with the true condition and varying amounts of information per individual can be taken into account. The ROC curve provides an assessment of the performance of the test in combination with the schedule of testing. Within this framework, a computational simplification is introduced to calculate variances and covariances for the areas under the ROC curves. Periodic monitoring for reperfusion following thrombolytic treatment for acute myocardial infarction provides a detailed example, whereby the lengths of the testing interval combined with different diagnostic markers are compared.     

An exponential model used for optimal threshold selection on ROC curves

A two-parameter exponential equation for modeling a receiver operating characteristic (ROC) curve is presented, where the area under the curve is a simple function of one of the parameters. The model makes no distributional assumptions about the underlying normal and abnormal patient populations or about the shape of the resulting ROC curves. In a computer simulation of 75 ROC curves, the model provides a fit equivalent to the maximum likelihood estimate method commonly used for ROC curve fitting. Similar results are obtained using the model to fit ROC curve data from the literature. The model's equation calculates the true-positive ratio as a function of the false-positive ratio, and has a first derivative that is useful for finding the optimal decision threshold for a diagnostic testing procedure. In particular, the model is useful in a computer program for finding jointly optimal thresholds for multiple sequential tests.     

Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski‐Rockette model parameters

For the typical diagnostic radiology study design, each case (ie, patient) undergoes several diagnostic tests (or modalities) and the resulting images are interpreted by several readers. Often, each reader is asked to assign a confidence‐of‐disease rating to each case for each test, and the diagnostic tests are compared with respect to reader‐performance outcomes that are functions of the reader receiver operating characteristic (ROC) curves, such as the area under the ROC curve. These reader‐performance outcomes are frequently analyzed using the Obuchowski and Rockette method, which allows conclusions to generalize to both the reader and case populations. The simulation model proposed by Roe and Metz (RM) in 1997 emulates confidence‐of‐disease data collected from such studies and has been an important tool for empirically evaluating various reader‐performance analysis methods. However, because the RM model parameters are expressed in terms of a continuous decision variable rather than in terms of reader‐performance outcomes, it has not been possible to evaluate the realism of the RM model. I derive the relationships between the RM and Obuchowski‐Rockette model parameters for the empirical area under the ROC curve reader‐performance outcome. These relationships make it possible to evaluate the realism of the RM parameter models and to assess the performance of Obuchowski‐Rockette parameter estimates. An example illustrates the application of the relationships for assessing the performance of a proposed upper one‐sided confidence bound for the Obuchowski‐Rockette test‐by‐reader variance component, which is useful for sample size estimation.     

Testing for fetal pulmonary maturity: ROC analysis involving covariates, verification bias, and combination testing

The lecithin/sphingomyelin ratio (L/S) and the measured value of saturated phosphatidylcholine (SPC), amniotic fluid determinations obtained to assess fetal pulmonary maturity, were evaluated with receiver operating characteristic (ROC) curve analysis. The effects of covariates on the ROC curves were analyzed with a regression methodology that took into account all the available data when constructing an ROC curve for each subgroup. To correct for verification bias the authors used a logistic regression analysis to model the probability of verification, thereby permitting correction for verification bias of a fully stratified data set in spite of small cell frequencies. They examined combination testing with prediction rules using prospective logistic modeling, including as variables test results and clinical features. The L/S was found to be significantly better than SPC for assessing fetal pulmonary maturity. For older gestational age the L/S and SPC performed better than for younger gestational age. Contamination of the specimen degraded the ROC curves. Correcting for verification bias did not influence the ROC curves significantly but changed the cutoff value of the test variable for any particular operating point. Prediction rules to evaluate combination testing showed that obtaining the SPC level in addition to the L/S ratio added no significant information compared with the L/S only. Including gestational age in the prediction rule of either test improved the prediction.     

A non-inferiority test for diagnostic accuracy based on the paired partial areas under ROC curves

Non-inferiority is a reasonable approach to assessing the diagnostic accuracy of a new diagnostic test if it provides an easier administration or reduces the cost. The area under the receiver operating characteristic (ROC) curve is one of the common measures for the overall diagnostic accuracy. However, it may not differentiate the various shapes of the ROC curves with different diagnostic significances. The partial area under the ROC curve (PAUROC) may present an alternative that can provide additional and complimentary information for some diagnostic tests which require false-positive rate that does not exceed a certain level. Non-parametric and maximum likelihood methods can be used for the non-inferiority tests based on the difference in paired PAUROCs. However, their performance has not been investigated in finite samples. We propose to use the concept of generalized p-value to construct a non-inferiority test for diagnostic accuracy based on the difference in paired PAUROCs. Simulation results show that the proposed non-inferiority test not only adequately controls the size at the nominal level but also is uniformly more powerful than the non-parametric methods. The proposed method is illustrated with a numerical example using published data.     

Incorporating the time dimension in receiver operating characteristic curves: a case study of prostate cancer.

Early diagnosis of disease has potential to reduce morbidity and mortality. Biomarkers may be useful for detecting disease at early stages before it becomes clinically apparent. Prostate-specific antigen (PSA) is one such marker for prostate cancer. This article is concerned with modeling receiver operating characteristic (ROC) curves associated with biomarkers at various times prior to the time at which the disease is detected clinically, by two methods. The first models the biomarkers statistically using mixed-effects regression models, and uses parameter estimates from these models to estimate the time-specific ROC curves. The second directly models the ROC curves as a function of time prior to diagnosis and may be implemented using software packages with binary regression or generalized linear model routines. The approaches are applied to data from 71 prostate cancer cases and 71 controls who participated in a lung cancer prevention trial. Two biomarkers for prostate cancer were considered: total serum PSA and the ratio of free to total PSA. Not surprisingly, both markers performed better as the interval between PSA measurement and clinical diagnosis decreased. Although the two markers performed similarly eight years prior to diagnosis, it appears that total PSA performed better than the ratio measure at times closer to diagnosis. The area under the ROC curve was consistently greater for total PSA than for the ratio four and two years prior to diagnosis and at the time of diagnosis.     

Transformation-invariant and nonparametric monotone smooth estimation of ROC curves

When a new diagnostic test is developed, it is of interest to evaluate its accuracy in distinguishing diseased subjects from non-diseased subjects. The accuracy of the test is often evaluated by receiver operating characteristic (ROC) curves. Smooth ROC estimates are often preferable for continuous test results when the underlying ROC curves are in fact continuous. Nonparametric and parametric methods have been proposed by various authors to obtain smooth ROC curve estimates. However, there are certain drawbacks with the existing methods. Parametric methods need specific model assumptions. Nonparametric methods do not always satisfy the inherent properties of the ROC curves, such as monotonicity and transformation invariance. In this paper we propose a monotone spline approach to obtain smooth monotone ROC curves. Our method ensures important inherent properties of the underlying ROC curves, which include monotonicity, transformation invariance, and boundary constraints. We compare the finite sample performance of the newly proposed ROC method with other ROC smoothing methods in large-scale simulation studies. We illustrate our method through a real life example.     

Equivalence of binormal likelihood‐ratio and bi‐chi‐squared ROC curve models

A basic assumption for a meaningful diagnostic decision variable is that there is a monotone relationship between it and its likelihood ratio. This relationship, however, generally does not hold for a decision variable that results in a binormal receiver operating characteristic (ROC) curve. As a result, ROC curve estimation based on the assumption of a binormal ROC‐curve model produces improper ROC curves, which have ‘hooks’, are not concave over the entire domain and cross the chance line. Although in practice this ‘improperness’ is usually not noticeable, sometimes it is evident and problematic. To avoid this problem, Metz and Pan proposed basing ROC‐curve estimation on the assumption of a binormal likelihood‐ratio (binormal‐LR) model, which states that the decision variable is an increasing transformation of the likelihood‐ratio function of a random variable having normal conditional diseased and nondiseased distributions. However, their development is not easy to follow. I show that the binormal‐LR model is equivalent to a bi‐chi‐squared model in the sense that the families of corresponding ROC curves are the same. The bi‐chi‐squared formulation provides an easier‐to‐follow development of the binormal‐LR ROC curve and its properties in terms of well‐known distributions. Copyright © 2015 John Wiley & Sons, Ltd.     

临床试验评价的ROC分析方法

目的探讨新药临床试验效果的评价问题,给出一种新的统计分析方法。方法基于ROC分析给出多变量ROC模型,采用有序logit联系函数,利用SAS软件进行参数估计,得到有协变量及交互作用情况下的ROC曲线方程及曲线下面积。结果采用文中给出的方法,可以有效地扣除协变量的影响,用ROC曲线直观地评价药物之间的差别和作用。结论本文提供的方法能够更有效地对临床试验做出客观和准确的评价。     

An evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve

The area under the receiver operating characteristic (ROC) curve serves as one means for evaluating the performance of diagnostic and predictive test systems. The most commonly used method for estimating the area under an ROC curve utilizes the maximum-likelihood-estimation technique, and a nonparametric method to calculate the area under an ROC curve was recently described. We compared the performance of these two methods. The results for the area under the ROC curve and the standard error of the estimate as calculated by each of the two methods exhibited high correlation. Generally, the nonparametric method yields lower area estimates than the maximum-likelihood-estimation technique. However, these differences generally were small, particularly with ROC curves derived from five or more cutoff points. Consistent results of hypothesis testing of the significance of differences between two ROC curves will be similar, regardless of which method is used, as long as one uses the same estimation technique on the two curves and as long as the two ROC curves being compared are of similar shape.