Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods

RATIONALE AND OBJECTIVES: Several statistical methods have been developed for analyzing multireader, multicase (MRMC) receiver operating characteristic (ROC) studies. The objective of this article is to increase awareness of these methods and determine if their results are concordant for published datasets. MATERIALS AND METHODS: Data from three previously published studies were reanalyzed using five MRMC methods. For each method the 95% confidence intervals (CIs) for the mean of the readers' ROC areas for each diagnostic test, the P value for the comparison of the diagnostic tests' mean accuracies, and the 95% CIs for the mean difference in ROC areas of the diagnostic tests were reported. RESULTS: Important differences in P values and CIs were seen when using parametric versus nonparametric estimates of accuracy, and there were the expected differences for random-reader versus fixed-reader models. Controlling for these differences, the Dorfman-Berbaum-Metz (DBM), Obuchowski-Rockette, Beiden-Wagner-Campbell, and Song's multivariate Wilcoxon-Mann-Whitney (WMW) methods gave almost identical results for the fixed-reader model. For the random-reader model, the DBM, Obuchowski-Rockette, and Beiden-Wagner-Campbell methods yielded approximately the same inferences, but the CIs for the Beiden-Wagner-Campbell method tend to be broader. Ishwaran's hierarchical ROC sometimes yielded significance not found with other methods. Song's modification of DBM's jack-knifing algorithm sometimes led to different conclusions than the original DBM algorithm. CONCLUSION: In choosing and applying MRMC methods, it is important to recognize: (1) the distinction between random-reader and fixed-reader models, the uncertainties accounted for by each, and thus the level of generalizeability expected from each; (2) assumptions made by the various MRMC methods; and (3) limitations of a five- or six-reader study when the reader variability is great.     

Bayesian regression methodology for estimating a receiver operating characteristic curve with two radiologic applications: prostate biopsy and spiral CT of ureteral stones

RATIONALE AND OBJECTIVES: The authors evaluated two Bayesian regression models for receiver operating characteristic (ROC) curve analysis of continuous diagnostic outcome data with covariates. MATERIALS AND METHODS: Full and partial Bayesian regression models were applied to data from two studies (n = 180 and 100, respectively): (a) The diagnostic value of prostate-specific antigen (PSA) levels (outcome variable) for predicting disease after radical prostatectomy (gold standard) was evaluated for three risk groups (covariates) based on Gleason scores. (b) Spiral computed tomography was performed on patients with proved obstructing ureteral stones. The predictive value of stone size (outcome) was evaluated along with two treatment options (gold standard), as well as stone location (in or not in the ureterovesical junction [UVJ]) and patient age (covariates). Summary ROC measures were reported, and various prior distributions of the regression coefficients were investigated. RESULTS: (a) In the PSA example, the ROC areas under the full model were 0.667, 0.769, and 0.703, respectively, for the low-, intermediate-, and high-risk groups. Under the partial model, the area beneath the ROC curve was 0.706. (b) The ROC areas for patients with ureteral stones in the UVJ decreased dramatically with age but otherwise were close to that under the partial model (ie, 0.774). The prior distribution had greater influence in the second example. CONCLUSION: The diagnostic tests were accurate in both examples. PSA levels were most accurate for staging prostate cancer among intermediate-risk patients. Stone size was predictive of treatment option for all patients other than those 40 years or older and with a stone in the UVJ.     

Comparison of correlated receiver operating characteristic curves derived from repeated diagnostic test data

RATIONAL AND OBJECTIVES: It is common to administer the same diagnostic test more than once to the same set of patients. The purpose of this study was to develop two statistical methods for estimating and comparing correlated receiver operating characteristic (ROC) curves for data derived from repeated diagnostic tests. MATERIAL AND METHODS: Parametric and semiparametric transformation models were developed. These estimation methods were illustrated with data from 72 pigmented lesions suspected of being malignant melanoma. A diagnostic scoring system based on asymmetry, border irregularity, color variation, and diameter was used repeatedly, with or without a dermoscope. Statistical hypothesis tests were conducted to evaluate whether a dermoscope improved the clarity of the lesion features in the scoring system. The resulting ROC curves were constructed, along with characteristics and summary measures. RESULTS: The areas under the ROC curves were 0.885 (parametric method) and 0.893 (semiparametric method) without the dermoscope, and 0.916 (parametric) and 0.912 (semiparametric) with the dermoscope. The statistical hypothesis tests did not yield statistically significant differences between the underlying ROC curves for either estimation method. CONCLUSION: The two transformation models yielded similar results for estimation and comparison of the ROC curves. Although a dermoscope did not add extra information, the scoring system was accurate for diagnosing malignant melanoma.     

Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis

RATIONALE AND OBJECTIVES: The purpose of this study was to develop an alternative approach to random-effects, receiver operating characteristic analysis inspired by a general formulation of components-of-variance models. The alternative approach is a higher-order generalization of the Dorfman, Berbaum, and Metz (DBM) approach that yields additional information on the variance structure of the problem. MATERIALS AND METHODS: Six population experiments were designed to determine the six variance components in the DBM model. For practical problems, in which only a finite set of readers and patients are available, six analogous bootstrap experiments may be substituted for the population experiments to estimate the variance components. Monte Carlo simulations were performed on the population experiments, and those results were compared with the corresponding multiple-bootstrap estimates and those obtained with the DBM approach. Confidence intervals on the difference of ROC parameters for competing diagnostic modalities were estimated, and corresponding comparisons were made. RESULTS: For mean values, the agreement of present estimates of variance structures with population results was excellent and, when suitably weighted and mixed, similar to or closer than that with the DBM method. For many variance structures, the confidence intervals in this study for the difference in ROC area between modalities were comparable to those with the DBM method. When reader variability was large, however, mean confidence intervals from this study were tighter than those with the DBM method and closer to population results. CONCLUSION: The jackknife approach of DBM provides a linear approximation to receiver-operating-characteristic statistics that are intrinsically nonlinear. The multiple-bootstrap technique of this study, however, provides a more general, nonparametric, maximum-likelihood approach. It also yields estimates of the variance structure previously unavailable.     

Comparison of three methods for estimating the standard error of the area under the curve in ROC analysis of quantitative data

RATIONALE AND OBJECTIVES: Several methods have been proposed for estimating the standard error (SE) of the area under the curve (AUC) in receiver operating characteristic analysis. The authors examined the validity of three methods--the LABROC procedure, exponential approximation, and the method of DeLong et al (purely nonparametric)--for estimating the SE of the AUC in receiver operating characteristic analysis of quantitative diagnostic data. MATERIALS AND METHODS: The authors conducted a broad numerical investigation to assess how to estimate the SE of AUC in various configurations of binormal and nonbinormal pairs of distributions, in which one or both pair members were mixtures of Gaussian distributions (the samples included 100 in the diseased group and 100 in the nondiseased group). RESULTS: The authors found that exponential approximation of the SE of AUC slightly underestimates the observed SE of a nonparametric estimate of the AUC when the ratio of the standard deviation of distributions for diseased to nondiseased populations was greater than 2. With binormal data the observed SE tended to be smaller with the LABROC procedure (semiparametric) than with the method of DeLong et al, but the LABROC procedure yields more conservative estimates of SE with nonbinormal data. In particular, with bimodal data it often produces a more conservative (ie, larger) estimate of the actual (observed) fluctuation. CONCLUSION: Overall, the LABROC procedure and the method of DeLong et al yielded very close estimates of the SE of AUC, even with data generated from a nonbinormal model. The choice between these two methods can be based on users' preferences and practicality.     

剂量体积直方图参数对放射性肺炎的工作特征曲线预测分析

目的 应用受试者工作特征曲线(ROC)对剂量体积直方图参数(DVH)预测放射性肺炎(RP)进行分析,探究DVH参数预测RP的准确性(ACC)、敏感性(SEN)和特异性(SPE)。方法 收集118例接受三维适形调强放疗和化疗的非小细胞肺癌患者资料,回顾分析三维放疗计划系统中双肺V₅、V₁₀、V₁₃、V₂₀、V₃₀ (V_x为接受≥x Gy的相对肺体积) 和平均肺剂量 (MLD) 与治疗后出现≥2级RP (CTCAE3.0) 的相关性。对上述DVH参数应用ROC曲线进行回顾性分析,确定预测RP的ACC、SEN和SPE。结果 单因素分析显示,双肺V₅、V₁₀、V₁₃、V₂₀和MLD均与RP发生显著相关(χ²=4.786、5.771、6.366、7.367、6.945,P<0.05);双肺V₃₀、患者因素(年龄、性别、KPS评分、肿瘤位置、病理类型)和治疗因素(放疗总剂量、照射技术、化疗方案、化疗时机)与RP的发生风险无显著相关性。多因素分析显示双肺V₂₀与RP发生风险相关(χ²=10.96,OR=4.16, 95%CI 1.40~12.36,P<0.05),与其他DVH参数具有显著共线性(r=0.767~0.902,P<0.05)。ROC曲线证实双肺V₂₀能够预测RP的发生(Z=2.038,P<0.05),其预测的ACC、SEN和SPE分别为0.645(95%CI 0.498~0.793),0.650(95%CI 0.408~0.864) 和0.674(95%CI 0.571~0.765),其阳性预测值仅为28.9%。结论 双肺V₂₀与RP的发生风险相关,能够预测RP的发生,但是预测能力有限。     

Statistical validation based on parametric receiver operating characteristic analysis of continuous classification data

Rationale and Objectives. The accuracy of diagnostic test and imaging segmentation is important in clinical practice because it has a direct impact on therapeutic planning. Statistical validations of classification accuracy was conducted based on parametric receiver operating characteristic analysis, illustrated on three radiologic examples.

Materials and Methods. Two parametric models were developed for diagnostic or imaging data. Example 1: A semi-automated fractional segmentation algorithm was applied to magnetic resonance imaging of nine cases of brain tumors. The tumor and background pixel data were assumed to have bi-beta distributions. Fractional segmentation was validated against an estimated composite pixel-wise gold standard based on multi-reader manual segmentations. Example 2: The predictive value of 100 cases of spiral computed tomography of ureteral stone sizes, distributed as bi-normal after a nonlinear transformation, under two treatment options received. Example 3: One hundred eighty cases had prostate-specific antigen levels measured in a prospective clinical trial. Radical prostatectomy was performed in all to provide a binary gold standard of local and advanced cancer stages. Prostate-specific antigen level was transformed and modeled by bi-normal distributions. In all examples, areas under the receiver operating characteristic curves were computed.

Results. The areas under the receiver operating characteristic curves were: Example 1: Fractional segmentation of magnetic resonance imaging of brain tumors: meningiomas (0.924–0.984); astrocytomas (0.786–0.986); and other low-grade gliomas (0.896–0.983). Example 3: Ureteral stone size for treatment planning (0.813). Example 2: Prostate-specific antigen for staging prostate cancer (0.768).

Conclusion. All clinical examples yielded fair to excellent accuracy. The validation metric area under the receiver operating characteristic curves may be generalized to evaluating the performances of several continuous classifiers related to imaging.     

受试者工作特征曲线在评价血流动力学 参数监测机体容量状况时的诊断价值

目的探讨利用受试者工作特征(receiver operating characteristic,ROC)曲线评价不同血流动力学参数监测机体容量状况时的诊断价值。方法 30例美国麻醉协会Ⅰ～Ⅱ级拟在全麻下行胃肠手术的患者,麻醉诱导后连续监测平均动脉压(mean arterial pressure,MAP)、心率(heart rate,HR)、中心静脉压(centralvenous pressure,CVP)、心脏指数(cardiac index,CI)、每搏量变异度(stroke volume variation,SVV)等血流动力学参数,以0.4 ml/(kg.min)的速率静脉输注6%羟乙基淀粉130/0.4氯化钠注射液进行容量治疗,输注总量为7 ml/kg。记录输注羟乙基淀粉即刻和输注结束后3 min时MAP、HR、CVP、CI、SVV值,计算CI变化的百分比即ΔCI,ΔCI≥15%视为对容量治疗有反应,绘制各参数容量治疗前的ROC曲线并分析其在监测机体容量状况中的诊断意义。结果 Pearson相关性分析显示:SVV的基础值与ΔCI有显著的线性相关(r=0.629,P<0.01)。各血流动力学参数ROC曲线下面积分别为MAP 0.479、HR 0.699、CVP 0.361、CI 0.455、SVV 0.920,SVV的ROC曲线下面积与其他血流动力学参数比较差异有统计学意义(P<0.01);当潮气量为8 ml/kg、以ΔCI≥15%定义对容量治疗有无反应的标准时,SVV最佳诊断阈值为10.5%,监测容量反应的灵敏度为93.8%、特异性为77.8%。结论 SVV是评估机体容量状况的良好指标,其诊断性优于MAP、HR、CVP、CI。     

Statistical power in observer-performance studies: comparison of the receiver operating characteristic and free-response methods in tasks involving localization

RATIONALE AND OBJECTIVES: Statistical power, defined as the probability of detecting real differences between imaging modalities, determines the cost in terms of readers and cases of conducting receiver operating characteristic (ROC) studies. Neglect of location information in lesion-detection studies analyzed with the ROC method can compromise power. Use of the alternative free-response ROC (AFROC) method, which considers location information, has been discouraged, because it neglects intraimage correlations. The relative statistical powers of the two methods, however, have not been tested. The purpose of this study was to compare the statistical power of ROC and AFROC methods using simulations. MATERIALS AND METHODS: A new model including intraimage correlations was developed to describe the decision variable sampling and to simulate data for ROC and AFROC analyses. Five readers and 200 cases (half of which contained one signal) were simulated for each trial. Two hundred trials, equally split between the null hypothesis and alternative hypothesis, were run. Ratings were analyzed with the Dorfman-Berbaum-Metz method, and separation of the null hypothesis and alternative hypothesis distributions was calculated. RESULTS: The AFROC method yielded higher power than the ROC method. Separation of the null hypothesis and alternative hypothesis distributions was larger by a factor of 1.6 regardless of the presence or absence of intraimage correlations. The effect of the incorrect localizations during ROC analysis of localization data is believed to be the major reason for the enhanced power of the AFROC method. CONCLUSION: The AFROC method can yield higher power than the ROC method for studies involving lesion localization. Greater consideration of this methodology is warranted.     

Digital radiography of the chest

Digital radiography is an appropriate method for both bedside and in-department chest radiographs. Its major advantage in bedside chest radiography is its control of the displayed optical density of these radiographs. With dynamic range control processing, it improves the visibility of tubes and lines superimposed on the mediastinal tissues. When used for in-department chest radiography, it may offer slight advantages in the evaluation of disease in the mediastinum, but in general is equivalent to film-screen chest radiography. The main reasons for using digital chest radiography for in-department chest radiographs relate mainly to its use as a data entry point method of projection radiography for high-quality teleradiology or for its use in a picture archiving and communication system. Apart from these advantages, there is no reason to change from conventional to digital chest radiographs. Digital radiographs are, with certain systems, printed at smaller than life size. Because of this, there is a necessary period of learning as radiologists adjust to the new image size. The most important change in radiologists' work pattern appears to be the need to sit closer to the film. Findings of disease are smaller, but, with experience, just as easy to see.