Evaluating patients' experiences with individual physicians: a randomized trial of mail, internet, and interactive voice response telephone administration of surveys

BACKGROUND: There is increasing interest in measuring patients' experiences with individual physicians, and empirical evidence supports this area of measurement. However, the high cost of data collection remains a significant barrier. Survey modes with the potential to lower costs, such as Internet and interactive voice response (IVR) telephone, are attractive alternatives to mail, but their comparative response rates and data quality have not been tested. METHODS: We randomly assigned adult patients from the panels of 62 primary care physicians in California to complete a brief, validated patient questionnaire by mail, Internet (web), or IVR. After 2 invitations, web and IVR nonrespondents were mailed a paper copy of the survey ("crossover" to mail). We analyzed and compared (n = 9126) the response rates, respondent characteristics, substantive responses, and costs by mode (mail, web and IVR) and evaluated the impact of "crossover" respondents. RESULTS: Response rates were higher by mail (50.8%) than web (18.4%) or IVR (34.7%), but after crossover mailings, response rates in each arm were approximately 50%. Mail and web produced identical scores for individual physicians, but IVR scores were significantly lower even after adjusting for respondent characteristics. There were no significant physician-mode interactions, indicating that statistical adjustment for mode resolves the IVR effect. Web and IVR costs were higher than mail. CONCLUSIONS: The equivalence of individual physician results in mail and web modes is noteworthy, as is evidence that IVR results are comparable after adjustment for mode. However, the higher overall cost of web and IVR, as the result of the need for mailings to support these modes, suggests that they do not presently solve cost concerns related to obtaining physician-specific information from patients.