Components-of-Variance Models for Random-Effects ROC Analysis: The Case of Unequal Variance Structures Across Modalities

RATIONALE AND OBJECTIVES: Several of the authors have previously published an analysis of multiple sources of uncertainty in the receiver operating characteristic (ROC) assessment and comparison of diagnostic modalities. The analysis assumed that the components of variance were the same for the modalities under comparison. The purpose of the present work is to obtain a generalization that does not require that assumption. MATERIALS AND METHODS: The generalization is achieved by splitting three of the six components of variance in the previous model into modality-dependent contributions. Two distinct formulations of this approach can be obtained from alternative choices of the three components to be split; however, a one-to-one relationship exists between the magnitudes of the components estimated from these two formulations. RESULTS: The method is applied to a study of multiple readers, with and without the aid of a computer-assist modality. performing the task of discriminating between benign and malignant clusters of microcalcifications. Analysis according to the first method of splitting shows large decreases in the reader and reader-by-case components of variance when the computer assist is used by the readers. Analysis in terms of the alternative splitting shows large decreases in the corresponding modality-interaction components. CONCLUSION: A solution to the problem of multivariate ROC analysis without the assumption of equal variance structure across modalities has been provided. Alternative formulations lead to consistent results related by a one-to-one mapping. A surprising result is that estimates of confidence intervals and numbers of cases and readers required for a specified confidence interval remain the same in the more general model as in the restricted model.     

Analysis of Uncertainties in Estimates of Components of Variance in Multivariate ROC Analysis

RATIONALE AND OBJECTIVES: Solutions have previously been presented to the problem of estimating the components of variance in the general linear model used for multivariate receiver operating characteristic (ROC) analysis. The case where the variance components do not change across the modalities under comparison was first treated, followed by the case where they are permitted to change. No analysis of uncertainties in these estimates has been presented previously. MATERIALS AND METHODS: For the case where the variance components do not change across modalities, the "jackknife-after-bootstrap" resampling procedure can be used together with conventional linear propagation of variance to solve for the uncertainties in estimates of the components. For the case where the components are permitted to change across modalities, a slight elaboration of this procedure is presented. RESULTS: The approach was validated by Monte Carlo simulations, where uncertainties in estimates of the variance components calculated by the jackknife-after-bootstrap procedure were found to converge in the mean to the Monte Carlo results over many independent trials. The method is exemplified with data from a study of readers-with and without the aid of a computer-assist modality-given the task of discriminating benign from malignant masses in mammography. CONCLUSION: The present approach is relevant to a broad class of problems where estimates of multiple contributions to the variance observed in ROC assessment of diagnostic modalities are desired, in particular, for the assessment of multiple-reader studies of computer-aided diagnosis in radiology where the variance components may change across reading modalities (eg, unaided vs computer-aided reading).