Hui and Walter's latent-class reference-free approach may be more useful in assessing agreement than diagnostic performance

BACKGROUND AND OBJECTIVE: Hui and Walter developed a latent class approach to assess the accuracy of a diagnostic procedure when no reference test is available. Our objective was to compare sensitivity and specificity estimates obtained with this reference-free approach and standard approaches, and to examine how and why they differed on a computerized tomography (CT) scan case study. STUDY DESIGN AND SETTING: We compared two sets of sensitivity and specificity estimates from four radiologists independently assessing tumoral and lymph node extension of 85 lung cancer patients with preoperative thoracic CT scan, those obtained relative to pathology findings from surgical specimens (reference set), and those derived from Hui and Walter's approach. RESULTS: The two sets of estimates significantly and markedly differed from each other. From simulations, we found that small-sample bias in Hui and Walter's estimates could be a major factor in explaining this difference. Furthermore, errors in pathology findings could account for part of this difference. Finally, our analyses revealed that the latent classes may differ intrinsically from the reference classes as defined from pathology findings and may have a different interpretation. CONCLUSION: Diagnostic parameters estimated with respect to latent classes may be more useful in providing a complete assessment of interobserver agreement than in assessing diagnostic performance.