Questionnaire color and response rates to mailed surveys. A randomized trial and a meta-analysis

The authors conducted a randomized trial in Geneva, Switzerland, to assess whether response rates to a mailed survey could be increased by printing the questionnaire on green paper. The authors also conducted a meta-analysis of 10 experimental studies that tested the effect of colored questionnaires on response rates. The randomized trial showed no effect (relative risk of responding [RR] = 1.00). The meta-analysis showed that mailing questionnaires on pink paper increased response rates by 12% (RR = 1.12, 95% confidence interval = 1.01 to 1.25, p = 0.04). Other colors had no statistically significant effect (blue: RR = 1.03, p = 0.49; green: RR = 1.02, p = 0.23; yellow: RR = 0.96, p = 0.30). Overall, using colored instead of white paper had no effect (RR = 1.02, p = 0.17). Thus, printing questionnaires on colored paper does not substantially increase response rates in surveys, except for pink paper.     

Increasing response rates in physicians'' mail surveys: an experimental study.

It is becoming increasingly difficult to obtain high response rates in physicians' mail surveys. In 1983-84, we tested the effectiveness of two techniques among 604 Quebec physicians who had not responded to an initial letter. A handwritten thank you note at the bottom of the letter accompanying the questionnaire and a more personalized mailout package increased response rates by 40.7 per cent and 53.1 per cent, respectively, compared to control groups.     

Increasing response rates in community health surveys administered by telephone

This paper reports the effects of two methods used to increase response rates in a community health survey administered by telephone. Converting refusals resulted in an increase of 3.7% in the final response rate, while the investigation and identification of indeterminate telephone numbers increased the response rate by an additional 6.1%. Together, these methods resulted in an increase of 9.8%, from an initial response rate of 70.1% to a final lower-bound response rate of 79.9%. The use of these methods helped to reduce non-response bias at a minimal cost.     

Increasing Medicaid child health screenings: the effectiveness of mailed pamphlets, phone calls, and home visits.

OBJECTIVES. A randomized controlled trial was conducted to test the effectiveness and cost effectiveness of three outreach interventions to promote well-child screening for children on Medicaid. METHODS. In rural North Carolina, a random sample of 2053 families with children due or overdue for screening was stratified according to the presence of a home phone. Families were randomly assigned to receive a mailed pamphlet and letter, a phone call, or a home visit outreach intervention, or the usual (control) method of informing at Medicaid intake. RESULTS. All interventions produced more screenings than the control method, but increases were significant only for families with phones. Among families with phones, a home visit was the most effective intervention but a phone call was the most cost-effective. However, absolute rates of effectiveness were low, and incremental costs per effect were high. CONCLUSIONS. Pamphlets, phone calls, and home visits by nurses were minimally effective for increasing well-child screenings. Alternate outreach methods are needed, especially for families without phones.     

Hospital concentration and low-income populations: Evidence from New York State Medicaid

While a large body of evidence has examined hospital concentration, its effects on health care for low-income populations are less explored. We use comprehensive discharge data from New York State to measure the effects of changes in market concentration on hospital-level inpatient Medicaid volumes. Holding fixed hospital factors constant, a one percent increase in HHI leads to a 0.6% (s.e. = 0.28%) decrease in the number of Medicaid admissions for the average hospital. The strongest effects are on admissions for birth (-1.3%, s.e. = 0.58%). These average hospital-level decreases largely reflect redistribution of Medicaid patients across hospitals, rather than overall reductions in hospitalizations for Medicaid patients. In particular, hospital concentration leads to a redistribution of admissions from non-profit hospitals to public hospitals. We find evidence that for births, physicians serving high shares of Medicaid beneficiaries in particular experience reduced admissions as concentration increased. These reductions may reflect preferences among these physicians or reduced admitting privileges by hospitals as a means to screen out Medicaid patients.     

Factors influencing the effectiveness of mailed health surveys.

The authors investigated sources of bias in health surveys by examining responses to their 1989 questionnaire mailed to 1,255 Massachusetts men who were eligible for dental care provided by the Department of Veterans Affairs. After a maximum of three mailings and one telephone call to nonrespondents, a total of 1,049 veterans had responded out of 1,228 finally determined to be eligible, a response rate of 85 percent. The investigators found that small differences in univariate estimates would have occurred had the field phase been terminated after the first mailing, which had a response rate of 61 percent. To evaluate multivariate distributions, they duplicated their previously published logistic regression model for sources of dental care, using only those who responded to the first and second mailings. Although model fits would have been substantively the same had the field phase been terminated after the first or the second mailings, analysis of parameter estimates and their statistical significances suggested bias that would have led to different substantive conclusions, in some instances. Another potential source of bias in surveys was found to be item omission. Fifty-eight percent of respondents answered all 46 survey questions, and 90 percent answered at least 91 percent of the questions. Fewer questions were answered by those whose responses were received last, but trends regarding missing data by age or education were not statistically significant. Although the survey using this methodology met all objectives, subject nonresponses, the ineligibility of potential respondents, item nonresponses, and skewed distributions of outcome variables combined to reduce the statistical power to detect differences among groups or to alter the analysis of the differences.(ABSTRACT TRUNCATED AT 250 WORDS)     

Web-based and mailed questionnaires: a comparison of response rates and compliance

BACKGROUND: We assessed response rates and compliance for a printed questionnaire and a Web questionnaire in a Swedish population-based study and explored the influence of adding personalized feedback to the Web questionnaire. METHODS: We assigned 875 subjects to 1 of 3 groups: printed questionnaire, plain Web questionnaire, or Web questionnaire with personalized feedback. The questionnaire had 2 parts, first a general section and then a dietary section. RESULTS: The response rate for the general section was 64% for the printed questionnaire, compared with 50% for the Web questionnaire with feedback. For the dietary questionnaire, the rates were reversed, resulting in a total response rate for the dietary questionnaire that did not differ between printed and web questionnaire with feedback. CONCLUSIONS: Interactivity in the Web questionnaire increased compliance in completion of the second section of the questionnaire. Web questionnaires can be useful for research purposes in settings in which Internet access is high.     

Response rates and response bias for 50 surveys of pediatricians

RESEARCH OBJECTIVE: To track response rates across time for surveys of pediatricians, to explore whether response bias is present for these surveys, and to examine whether response bias increases with lower response rates. DATA SOURCE/STUDY SETTING: A total of 63,473 cases were gathered from 50 different surveys of pediatricians conducted by the American Academy of Pediatrics (AAP) since 1994. Thirty-one surveys targeted active U.S. members of the AAP, six targeted pediatric residents, and the remaining 13 targeted AAP-member and nonmember pediatric subspecialists. Information for the full target samples, including nonrespondents, was collected using administrative databases of the AAP and the American Board of Pediatrics. STUDY DESIGN: To assess bias for each survey, age, gender, location, and AAP membership type were compared for respondents and the full target sample. Correlational analyses were conducted to examine whether surveys with lower response rates had increasing levels of response bias. PRINCIPAL FINDINGS: Response rates to the 50 surveys examined declined significantly across survey years (1994-2002). Response rates ranged from 52 to 81 percent with an average of 68 percent. Comparisons between respondents and the full target samples showed the respondent group to be younger, to have more females, and to have less specialty-fellow members. Response bias was not apparent for pediatricians' geographical location. The average response bias, however, was fairly small for all factors: age (0.45 years younger), gender (1.4 percentage points more females), and membership type (1.1 percentage points fewer specialty-fellow members). Gender response bias was found to be inversely associated with survey response rates (r=-0.38). Even for the surveys with the lowest response rates, amount of response bias never exceeded 5 percentage points for gender, 3 years for age, or 3 percent for membership type. CONCLUSIONS: While response biases favoring women, young physicians, and nonspecialty-fellow members were found across the 52-81 percent response rates examined in this study, the amount of bias was minimal for these factors that could be tested. At least for surveys of pediatricians, more attention should be devoted by investigators to assessments of response bias rather than relying on response rates as a proxy of response bias.     

Effect of mailed reminders on the response rate in surveys among patients in general practice.

Randomized trials were performed in Denmark and The Netherlands to determine the effect of mailed reminders on the response rate in surveys among patients in general practice. In both countries, general practitioners handed out questionnaires to 200 adult patients who came to visit them. An intervention group of 100 patients received reminders at 3 weeks after the visit, whereas a control group of the remaining 100 patients did not receive reminders. The response rate was significantly higher in the intervention groups than in the control group in The Netherlands (86% versus 55%, respectively) but not in Denmark (87% versus 81%, respectively). Mailed reminders can improve the response rate in surveys related to a general practice, but they are not effective in all situations.     

The effect on response rates of offering a small incentive with a mailed questionnaire.

To determine whether response rates to a mailed questionnaire sent to population control subjects could be increased through offer of a small incentive, half of the control subjects (n = 477) in a case-control study of renal cell carcinoma were randomly selected to receive a contact letter offering a lottery ticket if a completed questionnaire was returned; the remaining subjects (n = 477) received the same letter but with no mention of a lottery ticket. Overall response rates did not differ between the two groups (72.6% versus 74.4%), although a higher percentage of those offered a lottery ticket responded without follow-up (24.4% versus 18.5%). Binomial regression modeling of the effect of the lottery ticket offer, sex, age, and percent of urban dwellers on response indicated a significant effect only for percent of urban dwellers, the rate of response increasing with a decreasing percentage of urban dwellers. The effect of sex was of borderline significance (P = 0.05), with females having the higher rate of response.     

Use of cognitive interview techniques in the development of nutrition surveys and interactive nutrition messages for low-income populations

The effectiveness of dietary surveys and educational messages is dependent in part on how well the target audience's information processing needs and abilities are addressed. Use of pilot testing is helpful; however, problems with wording and language are often not revealed. Cognitive interview techniques offer 1 approach to assist dietitians in understanding how audiences process information. With this method, respondents are led through a survey or message and asked to paraphrase items; discuss thoughts, feelings, and ideas that come to mind; and suggest alternative wording. As part of a US Department of Agriculture-funded nutrition education project, 23 cognitive interviews were conducted among technical community college students in North Carolina. Interview findings informed the development of tailored computer messages and survey questions. Better understanding of respondents' cognitive processes significantly improved the language and approach used in this intervention. Interview data indicated 4 problem areas: vague or ineffective instructions, confusing questions and response options, variable interpretation of terms, and misinterpretation of dietary recommendations. Interviews also provided insight into the meaning of diet-related stages of change. These findings concur with previous research suggesting that cognitive interview techniques are a valuable tool in the formative evaluation and development of nutrition surveys and materials.     

Selected questionnaire size and color combinations were significantly related to mailed survey response rates

OBJECTIVE: To determine the degree to which mailed survey response rates, response times, and nonresponse bias are affected by questionnaire size and color. STUDY DESIGN AND SETTING: Questionnaires were mailed to a random sample of 2,000 Mayo Clinic patients in one of four size/color "test" groups. One thousand three hundred nine surveys were completed, approximately two-thirds in each group. RESULTS: A small (6 (1/8) x 8 (1/4) in) questionnaire booklet on white paper had a higher response rate (68.4%) than a similarly sized questionnaire on blue paper (62.3%). A large (8 (1/4) x 11 in) questionnaire on white paper had a 62.7% rate, whereas a large, blue questionnaire had a response rate of 68.6%. Median response times did not differ by questionnaire size/color. No evidence of differential nonresponse bias was observed across the four test groups. CONCLUSION: This study supports the use of a small/white questionnaire format advocated by the Total Design Method advanced by Don Dillman at Washington State University. We observed a favorable response rate for a large questionnaire printed on blue paper; however, if time and resources are limited, use of a small/white questionnaire appears preferable.     

State Medicaid coverage and access to care for low-income adults

OBJECTIVE: Budgetary pressures have led some states to limit Medicaid eligibility. We evaluated access to care for all low-income adults by the extent of state Medicaid coverage. METHODS: Current Population Survey data compiled by the Kaiser Commission on Medicaid and the Uninsured were used to rank the 48 continental states by the extent of Medicaid coverage for low-income non-elderly adults during 2000-2003. Data from the Behavioral Risk Factor Surveillance System for 2000-2003 were used to assess indicators of access to care, including being unable to see a physician due to cost, not obtaining routine checkups, and four preventive services for appropriate age groups by state. Access gaps were calculated between low-income (under $25,000/year) and high-income ($50,000 or more/year) adults within each state to control for unmeasured economic and health system differences between states. RESULTS: Access gaps between high and low-income people who could not see physicians due to cost were significantly smaller in states with the broadest Medicaid coverage compared with states with the narrowest coverage (19.2% vs. 23.7%, p=.003). Significantly smaller access gaps also occurred in states with broader Medicaid coverage for cholesterol testing (16.0% vs. 18.7%, p=.01), and Pap testing (6.0% vs. 10.8%, p=.002), but not colorectal cancer screening (13.3% vs. 12.5%, p=.28), mammography (14.3% vs. 19.7%, p=.07), and routine checkup within two years (8.0% vs. 9.3%, p=.10). CONCLUSIONS: A state's level of Medicaid coverage was associated with access to physicians' services, cholesterol testing, and cervical cancer screening for low-income adults. Broad Medicaid coverage may be an effective strategy for states to improve access to care and preventive services for low-income adults.     

Increasing response rates to a smoking survey for U.S. Navy enlisted women

This study examined the effectiveness of several persistent strategies to increase the response to a smoking survey among newly enlisted U.S. Navy women. The stepped approach, which included the use of incentives, repeated mailings, alternative survey administration modes, and reminders, was evaluated in terms of effects on response rates and response bias. Demographic and baseline smoking-related characteristics were compared for those responding on time to the initial mailed follow-up survey, reluctant respondents who did not respond initially but eventually completed a survey after further prompting, and non-respondents. Results showed that incentives and persistent efforts were effective in substantially increasing the response among 2,231 eligible participants, more than doubling the response rate (from 24.9% to 52.7%). The characteristics of on-time, reluctant, and non-respondents did not differ significantly in terms of sociodemographic characteristics. On the other hand, on-time respondents were different from both reluctant respondents and non-respondents in terms of smoking-related behaviors.