Estimating and comparing diagnostic tests' accuracy when the gold standard is not binary

RATIONALE AND OBJECTIVES: Investigators often need to assess the accuracies of diagnostic tests when the gold standard is not binary-scale. The objective of this article is to describe nonparametric estimators of diagnostic test accuracy when the gold standard is continuous, ordinal, and nominal scale. MATERIALS AND METHODS: A nonparametric method of estimating and comparing the area under receiver operating characteristic (ROC) curves, proposed by DeLong et al, is extended to situations in which the gold standard is not binary. Two examples illustrate the methods. RESULTS: Measures of diagnostic test accuracy, their variance, and tests for comparing two diagnostic tests' accuracies in paired designs are presented for situations in which the gold standard is continuous, ordinal, and nominal scale. These summary measures of diagnostic test accuracy are analogous in form and interpretation to the area under the ROC curve. CONCLUSION: Dichotomizing the outcomes of a gold standard so that traditional ROC methods can be applied can lead to bias. The methods described here are useful for assessing and comparing summary test accuracy when the gold standard is not binary scale. They have limitations similar to other summary indices.