Statistical Evaluation of the Use of Concurrent Controls in Treatment Screening Studies

A well‐designed pilot study can advance science by providing essential preliminary data to support or motivate further research, refine study logistics, and demonstrate proof of concept. Often, the outcomes of such studies can be quantified by a success/failure dichotomy. For example, a novel compound may show activation of a neural pathway, or it may not. When an intervention''s efficacy is quantified using a dichotomous outcome, probability mass functions can be enumerated to determine the probability that the observed result from a pilot study supports further evaluation of the intervention since there is only a finite, and often small, number of sample configurations possible. The purpose of this research was to determine the probability of an “efficacy signal” for pilot studies using one‐ and two‐sample pilot study designs. Efficacy signal was defined as the probability of observing a more favorable response proportion relative to a historical control (one‐sample setting) or to a concurrent control (two‐sample setting). An enumeration study (exact simulation) was conducted to calculate the efficacy signal probability. One‐sample study designs yielded higher probability of determining an efficacy signal than the two‐sample setting; however, sampling variation must be accounted for. A 68% score confidence interval is recommended for this purpose.