ROC methodology in radiologic imaging

If the performance of a diagnostic imaging system is to be evaluated objectively and meaningfully, one must compare radiologists' image-based diagnoses with actual states of disease and health in a way that distinguishes between the inherent diagnostic capacity of the radiologists' interpretations of the images, and any tendencies to "under-read" or "over-read". ROC methodology provides the only known basis for distinguishing between these two aspects of diagnostic performance. After identifying the fundamental issues that motivate ROC analysis, this article develops ROC concepts in an intuitive way. The requirements of a valid ROC study and practical techniques for ROC data collection and data analysis are sketched briefly. A survey of the radiologic literature indicates the broad variety of evaluation studies in which ROC analysis has been employed.     

Standard versus inverted digital luminescence radiography in detecting pulmonary nodules: A ROC analysis

Objective

Observer performance tests were conducted to compare the effect of presenting digital luminescence radiography (DLR) monitor images in positive mode (“bones black”) or negative mode (“bones white”) in the detectability of subtle lung nodules.

Methods

Five radiologists independently reviewed digital radiographs of 55 patients with either (a) a single, small (6–12 mm), non- calcified peripheral nodule confirmed by chest CT (n = 47) or (b) normal finding (n = 8) confirmed by chest CT, respectively. Mean size of pulmonary nodules was 8.3 mm (range 6–12 mm, median 7 mm). Radiographs were displayed either in standard mode (bones white) or inverse intensity (bones black). A total of 550 observations resulted. For the evaluation ROC analysis was performed using a specialized computer algorithm.

Results

The standard presentation mode showed a sensitivity of 61.7% and a specificity of 72.5%, the inverse presentation mode a sensitivity of 68.1% and a specificity of 75.0%. ROC- analysis showed that the difference between the standard mode (Az- value 0.694) and the inverse mode (Az- value 0.810) was significant in favour of the inverse presentation mode (p = 0.001). This finding was especially observed in more experienced radiologist (Az- values 0.716 vs. 0.909, P < 0.001).

Conclusion

Our study demonstrates a significant advantage of the inverse mode in the detection of small pulmonary nodules compared with the commonly used negative mode when interpreted by more experienced radiologists.

Advance in knowledge

Inverse intensity images (“bones black”) may have some advantages in the detection of small pulmonary nodules in experienced readers when presented on a stand-alone display.     

Proper ROC analysis and joint ROC analysis of the satisfaction of search effect in chest radiology

RATIONALE AND OBJECTIVES: The purpose of this study was to test whether the satisfaction of search (SOS) effect in chest radiology could be demonstrated with proper receiver operating characteristic (ROC) curves and with joint detection and localization ROC curves. MATERIAL AND METHODS: Data from an earlier ROC study of SOS in chest radiology were analyzed with three proper ROC models and one ROC model for joint detection and localization. Fits of the models were compared on the basis of likelihood-ratio chi-squared statistics (G2). To examine further the validity of the SOS effect in chest radiology, the authors also replicated the earlier study with a new sample of readers, analyzing the new data with the same methods. RESULTS: The proper contaminated binormal model fit the data better than the other two proper ROC models. Contaminated binormal analysis of the earlier and the replication experiment demonstrated an SOS effect: a reduction in area under the ROC curve for detection of the native abnormalities with the addition of nodules. Similarly, joint ROC analysis producing curves that appropriately cross the chance line gave similar results. CONCLUSION: Preventing inappropriate chance line crossing reduces measurement error and provides more powerful statistical tests. Results of both experiments showed that the SOS effect in chest radiology can be demonstrated with ROC methods that avoid inappropriate crossing of the chance line.     

Continuous versus categorical data for ROC analysis: some quantitative considerations

RATIONALE AND OBJECTIVES: Several authors have encouraged the use of a quasi-continuous rating scale for data collection in receiver operating characteristic (ROC) curve analysis of diagnostic modalities, rather than rating scales based on five to seven ordinal categories or levels of suspicion. Although many investigators have gone over to this method, a discussion of the issues continues. The present work provides a quantitative analysis from the viewpoint of measurement science. MATERIALS AND METHODS: A simple model of the effect of data discretization or quantization on the measurement of the variance of noisy data was developed. Then Monte Carlo simulations of multiple-reader, multiple-case ROC experiments were performed and analyzed in terms of components-of-variance models to investigate the effect of data quantization in that more complex setting. RESULTS: For single-reader studies, discretization into five categories can reduce the precision of ROC measurements by a large amount. The effect may be attenuated in multireader studies. CONCLUSION: More precise measurements of diagnostic detection performance and thus more efficient use of resources are served by good measurement methods. These are promoted by the use of a quasi-continuous rating scale in ROC studies.     

A contaminated binormal model for ROC data: Part III. Initial evaluation with detection ROC data

RATIONALE AND OBJECTIVES: The authors' purpose was to evaluate how well the contaminated binormal receiver operating characteristic (ROC) model fits (a) degenerate data for which standard ROC models commonly fail and (b) nondegenerate data from exemplary experiments, for which the standard binormal model should be appropriate. MATERIALS AND METHODS: The authors studied two examples of binormally degenerate data, with and without interior points, and ROC rating data from four experiments in visual psychophysics and radiology. The plots of contaminated binormal ROC curves of the binormal degenerate data were examined. For ROC data with at least one interior point, the new model was compared with conventional models on the basis of likelihood-ratio chi2 statistics (G2). RESULTS: With no interior points, the contaminated binormal model gave results consistent with the fundamental principle underlying ROC analysis, that is, for a fixed false-positive probability, the higher the true-positive probability, the better the diagnostic performance. Contaminated binormal ROC curves go through the empirical ROC points of the degenerate data without crossing the chance line or climbing far above the true-positive fractions of the points. For several model ROC studies, the contaminated binormal model gave smaller G2 results than conventional ROC models, although the differences tended to be small, usually with little difference in ROC area. CONCLUSION: The contaminated binormal model fits binormal degenerate data better than conventional ROC models, and it offers an explanation for the degeneracy. The lower G2 values on some classic, nondegenerate ROC data suggest that contamination may not be limited to degenerate ROC data.     

Assessing DTI data quality using bootstrap analysis.

Diffusion tensor imaging (DTI) is an established method for characterizing and quantifying ultrastructural brain tissue properties. However, DTI-derived variables are affected by various sources of signal uncertainty. The goal of this study was to establish an objective quality measure for DTI based on the nonparametric bootstrap methodology. The confidence intervals (CIs) of white matter (WM) fractional anisotropy (FA) and Clinear were determined by bootstrap analysis and submitted to histogram analysis. The effects of artificial noising and edge-preserving smoothing, as well as enhanced and reduced motion were studied in healthy volunteers. Gender and age effects on data quality as potential confounds in group comparison studies were analyzed. Additional noising showed a detrimental effect on the mean, peak position, and height of the respective CIs at 10% of the original background noise. Inverse changes reflected data improvement induced by edge-preserving smoothing. Motion-dependent impairment was also well depicted by bootstrap-derived parameters. Moreover, there was a significant gender effect, with females displaying less dispersion (attributable to elevated SNR). In conclusion, the bootstrap procedure is a useful tool for assessing DTI data quality. It is sensitive to both noise and motion effects, and may help to exclude confounding effects in group comparisons.     

ROC analysis in gammagraphic diagnosis of hepatic cirrhosis

An application of the ROC statistical method is described on a sample of 161 patients who underwent hepatic gammagraphy and biopsies obtained by laparsocopy, and of whom 90 were found to have hepatic cirrhosis according to pathological anatomy. Five decision thresholds have been established in the elaboration of the gammagraphic diagnosis of cirrhosis. The true positive fraction (TPF) and false positive fraction (FPF) of these five thresholds are shown on an ordinate axis and an ROC curve is obtained. By means of a simplified cost-benefit analysis the optimal point of operation for this type of diagnosis is determined on this curve, according to which the diagnosis of cirrhosis should be considered positive when alterations in the size of the liver and distribution of the isotope in the liver are found simultaneously with splenomegaly.     

Detection of thrombus by using phase-image MR scans: ROC curve analysis

Spin-echo phase images have been shown to be sensitive to blood flow and have been used to differentiate slow flow from thrombus, with an apparent advantage in comparison with spin-echo intensity images alone. In order to quantify the diagnostic efficacy of phase images, a study was performed comparing the sensitivity and specificity of MR imaging in identifying intravascular thrombus using spin-echo magnitude images alone and combined with phase images. In 45 subjects, 66 vessels with questionable intraluminal signal were reviewed in a blinded manner by four radiologists using seven levels of certitude for the diagnosis of thrombus. Vessels in the thorax, abdomen, and pelvis were included in the evaluation and were selected on the basis of the presence of intraluminal signal, which raised the possibility of intravascular disease. Corroborative studies were available in all cases. Receiver-operator-characteristic curves were constructed for the accuracy of the decision of intraluminal thrombus vs flow signal when using magnitude images alone and when using magnitude plus phase images. Magnitude images identified thrombus with a sensitivity of 35% at a specificity of 90%. On the other hand, combining magnitude image and phase images yielded sensitivities of 85% and 83% at specificities of 90% and 95%, respectively. We conclude that addition of phase images substantially increases the level of confidence in detecting intravascular thrombosis.     

DIMA enlargement mammography in microcalcifications: a prospective study with ROC analysis

The purpose of this study was to investigate whether the four-fold magnification mammography (direct magnification, DIMA) technique would perform better than conventional 1.5-fold magnification mammography in the differentiation of breast microcalcifications into benign and malignant. Fifty patients with non-palpable microcalcifications detected by mammography were examined immediately prior to surgical biopsy using both a conventional (1.5-fold) and the DIMA (fourfold) magnification mammography techniques. The microcalcifications were classified by five experienced radiologists using morphological criteria. A receiver operating characteristics curve (ROC) analysis of the sensitivity and specificity of both techniques in assessing malignancy was then carried out. The DIMA mammography technique was slightly but non-significantly superior to the conventional method in detecting malignancy (p > 0.05). Coarse granular and pleomorphic calcifications were detected more frequently with the DIMA technique. Coarse calcifications were significantly more frequently associated with histologically benign findings, whereas fine granular calcifications were significantly more likely to be malignant lesions. Assessment of malignancy associated with microcalcifications using morphological criteria is not significantly improved by mammography techniques with higher magnification.     

ROC analysis comparing screen film mammography and digital mammography

PURPOSE: To compare the diagnosis performances of radiologists on screen film versus digital mammography. MATERIAL: and methods: Two sets of 123 mammograms, screen film mammography and storage phosphor digital mammography, are studied comparatively with ROC analysis. RESULTS: Phantom study show that conventional method give better scores for usual tension but the detectability of smaller microcalcification is equivalent. To obtain with digital technic the same conventional score you have to increase the radiation dose. Roc Curves, simulated "detection" mode showed that radiologists performed with higher accuracy with conventional system but this difference is weekly statistically significant. ROC Curves, simulated "diagnostic" mode showed the same results wit no statistically significant difference but when the decision to go to the biopsy is the gold standard, ROC Curves were essentially equivalent for both film screen and digital mammography system. The readers consistently considered the digital mammograms to be less suspicious for cancer findings. The agreement study as proposed by the FDA indicate that probability of a positive digital mammograms given a positive screen film is 75% (threshold value 90%) and the probability of a negative digital mammograms given a negative analog film is 85% (threshold value 85%). CONCLUSION: Analysis of specific discrepancies indicate that spatial resolution is an essential limiting factor for digital method but high resolution phosphor plate are interesting in imaging treated breast, radioluscent lesion, fatty benign tumor, hamartoma, intramammary lymph node, breast with prosthesis.     

ROC analysis of mammography and palpation for breast screening

Receiver operating characteristic curves (ROCs) for mammography and clinical palpation individually and in combination are generated using screening data from Breast Cancer Detection Demonstration Project (BCDDP) No. 25. Detailed standard records of disease features observed on screening were kept by all examiners. And actual disease states at examination were determined for each case by pathology or follow-up. The ROCs are produced from these objective data without the usual concern for examiners' ultimate diagnostic conclusions. Comparative analysis of the ROCs illustrates the clear superiority of mammography over clinical palpation as an individual screening modality as well as the further superiority of the combined use of both modalities.     

A permutation test for comparing ROC curves in multireader studies a multi-reader ROC, permutation test

RATIONALE AND OBJECTIVES: The aim of the study is to develop a permutation test to compare receiver operating characteristic (ROC) curves of two diagnostic modalities in a multireader paired design. MATERIALS AND METHODS: A statistical test for comparing two diagnostic modalities is developed based on all possible exchanges of the set of reader-ratings between the two modalities. An exact permutation test is formed by determining the frequency of the most extreme values of the statistic estimating the average difference in the areas under the ROC curves (AUCs). An asymptotic version of the test is constructed by obtaining the exact permutation variance and appealing to the asymptotic normality of the nonparametric estimator of the average difference in areas. Computer simulations were conducted to validate the type I error for small sample sizes. RESULTS: The new test provides a permutation approach for comparing ROC curves in a multireader paired-design setting in which effects of the readers are considered to be fixed. The type I error of the asymptotic test is close to the true value, even for samples as small as 20 normal and 20 abnormal cases. The test is designed to be sensitive to alternatives in which the AUCs of the two diagnostic modalities differ. CONCLUSIONS: The proposed test provides a powerful method for comparing two diagnostic modalities in a multireader paired-study design when the primary interest is to detect difference in average AUCs.