Weighing gains and losses in criterion validity studies with test-based enrollment

Criterion validity studies sometimes use designs with test-based enrollment schemes. In such incomplete studies, a Reference Instrument (RI) is applied to unequal sampling fractions of subjects previously identified as positives or negatives by a new Test Instrument (TI+ and TI-). Focusing on sensitivity (Se) and specificity (Sp), this article addresses some issues concerning the precision of estimates, study costs, as well as the acceptability/convenience to subjects. For that purpose, examples are provided whereby three indicators-statistical efficiency differential (deltaS), cost differential (deltaC), and (in)convenience differential (deltaI)-are contrasted and discussed. Although a clear, fast-and-ready answer as to what constitutes an optimal study cannot be given, the article offers a rationale for weighing gains and losses. Among several scenarios, it is shown that an appropriately chosen incomplete study design may be as statistically efficient as one with a complete sampling scheme, yet is able to offer a ca. 15% cost reduction and about 20% fewer individuals needing to endure an invasive or logistically cumbersome RI. A special emphasis on the planning stages of an investigation is called for, precisely when the level of statistical precision the researcher is willing to accept can be weighed against the available budget and the degree of stress put on the subject that ought to be avoided.