Regaining confidence in confidence intervals for the mean treatment effect

In many experiments, it is necessary to evaluate the effectiveness of a treatment by comparing the responses of two groups of subjects. This evaluation is often performed by using a confidence interval for the difference between the population means. To compute the limits of this confidence interval, researchers usually use the pooled t formulas, which are derived by assuming normally distributed errors. When the normality assumption does not seem reasonable, the researcher may have little confidence in the confidence interval because the actual one‐sided coverage probability may not be close to the nominal coverage probability. This problem can be avoided by using the Robbins–Monro iterative search method to calculate the limits. One problem with this iterative procedure is that it is not clear when the procedure produces a sufficiently accurate estimate of a limit. In this paper, we describe a multiple search method that allows the user to specify the accuracy of the limits. We also give guidance concerning the number of iterations that would typically be needed to achieve a specified accuracy. This multiple iterative search method will produce limits for one‐sided and two‐sided confidence intervals that maintain their coverage probabilities with non‐normal distributions. Copyright © 2014 John Wiley & Sons, Ltd.