Prenotification had no additional effect on the response rate and survey quality: a randomized trial

ObjectivesTo determine the effect of prenotification package on survey quality, including response rates, response time, percentage of nonresponse items, and cost.Study Design and SettingParticipants were randomized into two groups. In the first round mailing, participants in prenotification group received a prenotification package, whereas direct questionnaire mailing group participants received a questionnaire with prepaid return envelope only. In the second round mailing, both groups received the questionnaires. The trial was integrated into a study among 35–65-year-old female nurses in Hong Kong.ResultsA total of 367 nurses were included in the trial. A total of 362 mails were successfully delivered. The initial response rate in the first round mailing were 8.79% and 8.89% for prenotification and direct questionnaire mailing groups, respectively. After the second round mailing, the final response rate in prenotification and direct questionnaire mailing groups were 17.58% and 17.22%, respectively; no significant difference was found between the groups. There were no differences with respect to percentage of nonresponse items or response time, but the cost of prenotification group was HK$ 15.11 per response higher than direct mailing group.ConclusionPrenotification had no additional effect on the response rate and other survey quality compared with direct questionnaire mailing in a Hong Kong population.     

Prenotification but not envelope teaser increased response rates in a bulimia nervosa mental health literacy survey: A randomized controlled trial

ObjectivesUnderstanding attitudes to mental health issues can inform public health interventions. However, low response rates may contribute to nonresponse bias. In a randomized controlled trial we examined the effect of sending a prenotification postcard before the questionnaire and the placement of a short message on the survey envelope (teaser) on response rates to a mailed questionnaire about bulimia nervosa “mental health literacy”.Study Design and SettingQuestionnaires were mailed to 3,010 adults (50.6% female and 49.4% male) aged 18–65 years. In a 2 (pre-notification–present; absent) by 2 (teaser–present; absent) design, questionnaire recipients were randomly allocated to the experimental strategies. Outcomes considered were response rate, response time, and cost.ResultsThe overall response rate was 22.0%. Significant main effects showed higher response rates for the use of prenotification (present = 23.6%; absent = 20.3%), among female participants, and older participants. A significant interaction of teaser by gender indicated lower response rates for men who received the teaser but not for women. Older participants returned the questionnaire more promptly than younger participants. Females—but not males—who received the teaser were slower to return the questionnaire. Higher response rates for participants receiving the postcard compensated for increased costs, particularly for males and older participants.ConclusionResponse rates to a mental health postal survey can be increased through the use of prenotification.     

Effects of incentives and prenotification on response rates and costs in a national web survey of physicians

Little is known about what strategies are cost-effective in increasing participation among physicians in surveys that are conducted exclusively via the web. To assess the effects of incentives and prenotification on response rates and costs, general internists (N = 3,550) were randomly selected from the American Medical Association (AMA) Masterfile and assigned to experimental groups that varied in the amount of a promised incentive (none, entry into a $200 lottery, $50, or $100) and prenotification (none, prenotification letter only, or prenotification letter containing a $2 preincentive). Results indicated that the response rates were highest in the groups promised $100 and $50, respectively. While the postal prenotification letter increased response rates, the inclusion of a small token $2 preincentive had no effect on participation. Further, unlike mail surveys of physicians, the $2 preincentive was not cost-effective. Among physicians, larger promised incentives of $50 or $100 are more effective than a nominal preincentive in increasing participation in a web-only survey. Consistent with prior research, there was little evidence of nonresponse bias among the experimental groups.     

An advance letter did not increase the response rates in a telephone survey: a randomized trial

ObjectiveTo test the impact of an advance letter on response and cooperation rates in a nationwide telephone survey, given previous inconsistent results.Study Design and SettingWithin the context of a larger telephone survey, 1,000 Australian households were randomly selected to take part in this trial. Half were randomly allocated to receive an advance letter, whereas the remainder did not receive any advance communication. Response and cooperation rates were compared between the two groups.ResultsA total of 244 interviews were completed, 134 of which were with households that had been sent an advance letter. Intention-to-treat analysis revealed no significant difference in response between those who had received a letter and those who had not (26.8% vs. 22.0%, respectively). In addition, there was no significant difference between the groups in terms of either cooperation (78.4% vs. 79.7%) or response rate (56.3% vs. 57.9%), and no clear differences emerged in terms of the demographic characteristics of the two groups.ConclusionAn advance letter was not seen to be effective in increasing response or cooperation rates in a nationwide telephone survey. Researchers should consider alternative methods of increasing participation in telephone surveys.     

Mode of delivery, but not questionnaire length, affected response in an epidemiological study of eating-disordered behavior

BACKGROUND AND OBJECTIVES: The effects of questionnaire length and mode of delivery on response rates were examined in an epidemiological study of eating-disordered behavior. METHODS: Short (8 pages) and long (14 pages) questionnaires were posted or hand-delivered to a community sample of 802 women. Nonrespondents who received the first questionnaire by hand delivery received a reminder letter and replacement questionnaire by post; those who received the initial questionnaire by post were further randomized to receive the first reminder by hand delivery or by post, in short or long form. A second reminder letter and questionnaire (in short or long form) were posted to all remaining nonrespondents. RESULTS: The overall response rate was 52.9%. This is a conservative estimate of true response, because in a substantial proportion of cases (12.2%) individuals were no longer resident at the listed address. There was a significant effect of mode of delivery on response, favoring hand delivery, at both the initial mailout and first reminder. There was no effect of questionnaire length on response to the initial mailout, although overall response was significantly higher for the longer form. It was estimated that an overall response of 58.0% would have been achieved had first reminders been hand-delivered to all nonrespondents who received the initial mailout by post. CONCLUSIONS: Delivery of questionnaires by hand may be an effective way to increase response rates in epidemiological research, but little is to be gained by reducing questionnaire length.     

Does a deadline improve men's participation in self-administered health surveys? A randomized controlled trial in general practice

BACKGROUND: Self-administered questionnaires are commonly used in experimental studies to elicit quality of life or other outcomes. Hence, achieving an acceptable level of follow-up from patients is critical to minimizing bias. Many methods for maximizing follow-up remain untested. It is also unclear what level of follow-up is required to prevent bias being introduced. METHODS: We recruited 246 men from general practice surgeries in Sydney, Australia. These 246 men were randomized to receive a covering letter with their follow-up questionnaire either advising of a deadline to reply (Deadline, n = 126) or a standard letter without a deadline (No Deadline, n = 120). Four standardized reminder prompts subsequently were administered. We calculated interim response rates and the final proportion of follow-up questionnaires received according to group. We also compared scores on two main outcomes, namely, knowledge and decisional conflict at each time when reminder prompts were administered. RESULTS: One hundred and twelve (88.9%) men in the Deadline group returned their follow-up questionnaires compared with 102 (85.0%) men in the No Deadline group. This difference was not statistically significant [odds ratio = 1.41, 95% confidence interval (CI) = 0.67-2.99; p = 0.36]. Time to response also was not significantly affected by cover letter received (hazard ratio = 0.96; 95% CI = 0.73-1.25; p = 0.76). Results of the original RCT were similar in terms of direction and effect size at all times irrespective of when reminder prompts were administered. CONCLUSION: The addition of a deadline adds no further impact in improving response rates from male patients compared with an unspecified letter. Despite the accepted wisdom that higher response protects against bias, differences in outcomes were consistent throughout the post-test data collection period.     

Increasing response to a postal survey of sedentary patients - a randomised controlled trial [ISRCTN45665423]

Background

A systematic review identified a range of methods, which can influence response rates. However, analysis specific to a healthcare setting, and in particular, involving people expected to be poor responders, was missing, We examined the effect of pre-warning letters on response rates to a postal survey of sedentary patients whom we expected a low rate of response.

Methods

Participants were randomised to receive a pre-warning letter or no pre-warning letter, seven days before sending the main questionnaire. The main questionnaire included a covering letter and pre-paid return envelope. After seven days, non-responders were sent a reminder letter and seven days later, another reminder letter with a further copy of the questionnaire and return envelope.

Results

627 adults, with a mean age of 48 years (SD 13, range 18 to 78) of whom 69.2% (434/627) were women, were randomised. 49.0% (307/627) of patients were allocated to receive a pre-warning letter and 51.0% (320/627) no pre-warning letter, seven days in advance of posting the main questionnaire. The final response rate to the main questionnaire was 30.0% (92/307) amongst those sent a pre-warning letter and 20.9% (67/320) not sent a pre-warning letter, with an adjusted odds ratio of 1.60 (95% CI 1.1, 2.30).

Conclusions

The relatively low cost method of sending a pre-warning letter had a modest impact on increasing response rates to a postal questionnaire sent to a group of patients for whom a low response rate was anticipated. Investigators should consider incorporating this simple intervention when conducting postal surveys, to reduce the potential for nonresponse bias and to increase the study power. Methods other than postal surveys may be needed however when a low response rate to postal surveys is likely.     

Labour-saving strategies to maintain survey response rates: a randomised trial

To evaluate response-aiding strategies feasible in large surveys, we randomly allocated general practitioners (GPs) to one of four intervention groups: Group 1 received 'exhaustive' telephone prompts by a medical peer in advance of a questionnaire; Group 2, inclusion of an embossed pen with the questionnaire; Group 3, an advance letter prompt; and Group 4, a 'single attempt' advance telephone prompt by a non-medical research assistant. Follow-up procedures were identical. Response rates by group were not significantly different overall (χ²4.59, df=3, p=0.20) although advance prompts by a medical peer were significantly more effective than other strategies for male GPs. The difference in overall response rates between males (63%) and females (74%) was significant (χ²=15.40, df=1, p<0.01). No other response bias was evident. Our demonstration of a significant interaction between respondent sex and response-aiding strategy invites further research.     

Reducing questionnaire length did not improve physician response rate: a randomized trial

ObjectivesTo examine the effect of reducing questionnaire length on the response rate in a physician survey.Study Design and SettingA postal four double-page questionnaire on end-of-life decision making was sent to a random sample of 1,100 general practitioners, 400 elderly care physicians, and 500 medical specialists. Another random sample of 500 medical specialists received a shorter questionnaire of two double pages. After 3 months and one reminder, all nonresponding physicians received an even shorter questionnaire of one double page.ResultsTotal response was 64% (1,456 of 2,269 eligible respondents). Response rate of medical specialists for the four double-page questionnaire was equal to that of the two double-page questionnaire (190 and 191 questionnaires were returned, respectively). The total response rate increased from 53% to 64% after sending a short one double-page questionnaire (1,203–1,456 respondents).ConclusionThe results of our study suggest that reducing the length of a long questionnaire in a physician survey does not necessarily improve response rate. To improve response rate and gather more information, researchers could decide to send a drastically shortened version of the questionnaire to nonresponders.     

Use of Monetary and Nonmonetary Incentives to Increase Response Rates Among African Americans in the Wisconsin Pregnancy Risk Assessment Monitoring System

From 2009 to 2010, an experiment was conducted to increase response rates among African American mothers in the Wisconsin Pregnancy Risk Assessment Monitoring System (PRAMS). Sample members were randomly assigned to groups that received a prepaid, cash incentive of $5 (n = 219); a coupon for diapers valued at $5 (n = 219); a coupon for diapers valued at 6 (n = 210); or no incentive (n = 209). Incentives were included with the questionnaire, which was mailed to respondents. We examined the effects of the incentives on several outcomes, including response rates, cost effectiveness, survey response distributions, and item nonresponse. Response rates were significantly higher for the cash group than for the coupon (42.5 vs. 32.4%, P < .05) or no incentive group (42.5 vs. 30.1%, P < .01); the coupon and no incentive groups performed similarly. While absolute costs were the highest for the cash group, the cost per completed survey was the lowest. The incentives had limited effects on response distributions for specific survey questions. Although respondents completing the survey by mail in the cash and coupon groups exhibited a trend toward being less likely to have missing data, the effect was not significant. Compared to a coupon or no incentive, a small cash incentive significantly improved response rates and was cost effective among African American respondents in Wisconsin PRAMS. Incentives had only limited effects, however, on survey response distributions, and no significant effects on item nonresponse.     

The effect of two mailing strategies on the response to a survey of physicians

In 1989, the authors tested the effectiveness of two response-enhancing techniques, a postage stamped or franked return envelope and a prenotification letter, in a survey of pregnancy among 10,047 resident physicians in the United States. The techniques were randomly assigned using a factorial design. No significant interactions were observed between the techniques. After two mailings, those who received a stamped return envelope had a response of 71.2%, compared with 68.2% for those who received a franked return envelope (95% confidence interval 1.3-4.9%). Men who received the stamped envelope had a 5.9% greater response than those who received the franked envelope (p less than 0.001), but the type of postage did not influence response among women (p = 0.84); this interaction was statistically significant (p = 0.006). Physicians who received a prenotification letter had a response of 69.0%, compared with 70.5% for those who did not receive the letter (95% confidence interval -3.3 to 0.2%). The authors conclude that seemingly minor changes in survey design could have saved from 12% to 19% of the total cost of the study.     

Vaccination of high-risk patients for influenza. A comparison of telephone and mail reminder methods

During the 1984-1985 influenza season two study groups were used to compare telephone and letter reminder methods with a control group that received no reminder to determine which was the most effective strategy to increase influenza vaccination rates among the high-risk patient population of a university-based family practice. Seven hundred eighty-seven high-risk patients were randomly assigned to one of the three study groups: a mailed-reminder group, a telephone-reminder group, and a control group. Vaccination rates for both reminder methods were significantly higher than for the control group (P less than .02), and if successfully contacted, the telephone-reminder group had a significantly better vaccination rate than the mailed-reminder group (P less than .05). If successful telephone contact can be made, this reminder method is more effective than a letter reminder to increase influenza vaccination rates among high-risk patients.     

Increasing response rates of gynaecologists to a survey: a randomised trial of telephone prompts

Abstract: This study used a randomised controlled trial to evaluate the effectiveness of telephone prompts to increase the response rate of gynaecologists to a survey. A nine-page questionnaire about management of women with abnormal Pap smears was mailed to Fellows of the Royal Australian College of Obstetricians and Gynaecologists. A random sample was contacted in advance by telephone, and nonrespondents received a telephone reminder 14 days after the initial mail-out. The control group was mailed questionnaires and a reminder letter conventionally. On Day 25, nonresponders in both groups were mailed a second questionnaire. Of the intervention group, 84 per cent responded to the survey compared to 69 per cent of the control group (P = 0.005). A telephone call in anticipation of a mailed questionnaire is highly recommended.     

Late response to patient-reported outcome questionnaires after surgery was associated with worse outcome

ObjectivesNonresponse to patient-reported outcome (PRO) questionnaires after surgery might bias the results. Our aim was to gauge the potential impact of nonresponse bias by comparing the outcomes of early and late responders.Study Design and SettingThis study compares 59,565 early and 20,735 late responders who underwent a hip or knee replacement, hernia repair, or varicose vein (VV) surgery. The association between timeliness of response and three outcomes (the mean postoperative disease-specific PRO and generic PRO scores and the proportion reporting a fair or poor result) was examined by regression analysis.ResultsLate responders were slightly more likely to be young, nonwhite, deprived, and have a more severe preoperative condition with poorer quality of life. Late response was associated with a slightly poorer outcome in all four procedures although not statistically significant (P < 0.05) for VV surgery. Adjusting for preoperative characteristics reduced the strength of the associations, but they remained statistically significant.ConclusionAs nonresponse to PRO questionnaires introduces slight bias, differences in response rates between hospitals should be taken into account when making comparisons so as to avoid overestimating the performance of those with lower response rates and failing to detect poor performing hospitals.     

“Bird in the hand” cash was more effective than prize draws in increasing physician questionnaire response

ObjectiveTo investigate the effects of two monetary incentives on response rates to postal questionnaires from primary care physicians (PCPs).Study Design and SettingThe PCPs were randomized into three arms (n = 550 per arm), namely (1) €5 sent with the questionnaire (cash); (2) entry into a draw on return of completed questionnaire (prize); or (3) no incentive. Effects of incentives on response rates and item nonresponse were examined, as was cost-effectiveness.ResultsResponse rates were significantly higher in the cash (66.1%; 95% confidence interval [CI]: 61.9, 70.4%) and prize arms (44.8%; 95% CI: 40.1, 49.3%) compared with the no-incentive arm (39.9%; 95% CI: 35.4, 44.3%). Adjusted relative risk of response was 1.17 (95% CI: 1.02, 1.35) and 1.68 (95% CI: 1.48, 1.91) in the prize and cash arms, respectively, compared with the no-incentive group. Costs per completed questionnaire were €9.85, €11.15, and €6.31 for the cash, prize, and no-incentive arms, respectively. Compared with the no-incentive arm, costs per additional questionnaire returned in the cash and prize arms were €14.72 and €37.20, respectively.ConclusionBoth a modest cash incentive and entry into a prize draw were effective in increasing response rates. The cash incentive was most effective and the most cost-effective. Where it is important to maximize response, a modest cash incentive may be cost-effective.     

Telephone reminders are a cost effective way to improve responses in postal health surveys

STUDY OBJECTIVE: To assess the effectiveness of a telephone reminder in increasing responses to postal surveys and to calculate the differential costs per completed questionnaire. DESIGN: Randomised controlled trial. SETTING: Australian university and rehabilitation medicine practice. PARTICIPANTS: The trial was conducted in 1999 among the 143 non-respondents to a questionnaire about work related neck and upper body disorders. The questionnaire was sent to two Australian female samples: 200 office workers (Sample A) and 92 former rehabilitation medicine patients (Sample B). A reminder letter, another copy of the questionnaire and a final letter were sent at two week intervals. Half of the non-respondents within each sample were randomly selected to receive a telephone reminder just after the second mailout of the questionnaire. All direct costs were calculated. MAIN RESULTS: Responses were significantly higher among those who received the telephone reminder intervention (relative risk 2.54, 95% confidence intervals 1.43 to 4.52). Analysed by intention to phone, 47% of non-respondents in Sample A and 38% in Sample B returned a complete questionnaire after the intervention, compared with 21% and 10%, respectively, in the control groups. For the 112 women (combined samples) who returned completed questionnaires before randomisation, the average cost per respondent was AUD14. There was a higher total cost for the intervention groups (AUD851 versus AUD386 for controls), but the significantly higher number of additional completed responses (31 versus 12) resulted in a 15% lower marginal cost per completed questionnaire in those groups. CONCLUSION: Telephone reminders are cost effective in improving responses to postal surveys.     

Lifestyle surveys--the complete answer?

STUDY OBJECTIVES: These were as follows: to study incompleteness of data, herein called item non-response, generated by a self completion questionnaire; to identify the characteristics of item non-responders and the types of questions liable to high item non-response rates; and to discuss possible reasons for item non-response. DESIGN: Item non-response patterns in 12,307 responders (62%) to a representative postal survey based on a stratified sample drawn from family health services authorities' (FHSA) registers were investigated. MAIN OUTCOME MEASURES: Data were analysed for item non-response in three groups depending on when the questionnaire was returned (wave analysis). The overall completion rate of the questionnaire was examined and the natural logarithm of the proportion of completed questions was used as an outcome variable in multiple regression analysis. Item nonresponse to key questions and questions of different types was examined. RESULTS: Wave analysis: the overall completion rate of the questionnaire was 86% in questionnaires returned before the first reminder and 83%-84% in those sent back after subsequent reminders. Overall pattern of item non-response; respondents failed to complete a mean of 15% and a median of 10% of the questionnaire. All questions in the questionnaire had some item non-response, ranging from 1% to 85%. Completion rates were associated with gender, age, indicators of lower socioeconomic status, and general health status. Individual questions: particular types of questions were liable to have higher item non-response, for example, linked binary questions. CONCLUSIONS: Item non-response in population postal surveys is likely to present problems in the interpretation of data by introducing bias additional to that of total non-response. Item non-response does not increase greatly with later returns, suggesting that the quality of data across responses generated by two reminders is similar. There are obstacles to reducing item non-response, such as respondent error or socioeconomic and health characteristics of the general population, that cannot be totally overcome. However, the evidence that individuals tend to complete only options within questions that apply to them and their positive behaviour is useful information for those designing questionnaires and interpreting survey data.     

Late response and item nonresponse in the Finbalt Health Monitor survey

BACKGROUND: The Finbalt Health Monitor is a collaborative system for monitoring the health-related behaviour, practices and lifestyles in Estonia, Finland, Latvia and Lithuania. This system is based on nationally representative samples and self-administered mailed questionnaires. In comparing the results of national surveys, the awareness of the direction and socioeconomic patterning of the response bias is essential. METHODS: The data were gathered from the cross-sectional surveys conducted in 1998 from Estonia (n = 1362), Finland (n = 3504), Latvia (n = 2322) and Lithuania (n = 1874). An analysis was made of the prevalence of late response, completeness of information obtained from respondents and the magnitude of response bias on the prevalence estimates of health behaviour indicators. RESULTS: The response rates were comparatively high: 68% in Estonia, 70% in Finland, 77% in Latvia and 62% in Lithuania. Late response was weakly related to age, education or place of residence. The total proportion of missing information was below 10% and the sociodemographic patterning for this missing information was similar in all countries. Thus, older and less-educated respondents had more missing information on their questionnaires. Response bias of the prevalence estimates was minimal when it was calculated by using information obtained from late respondents. CONCLUSIONS: The level of nonresponse and missing information was comparable in different countries, not information on health behaviour. Therefore special efforts are needed to design a questionnaire form which appears equally relevant to all respondent groups. The follow-up mailings were an effective way to increase the total response rate, but it was unlikely that they provided an effective way to reach the 'hard core' nonrespondents.     

Promoting mammography use through progressive interventions: is it effective?

This study evaluated interventions implemented with women in a health maintenance organization who had not utilized their annual free mammogram referral 45 days (step 2) and 95 days (step 3) after its mailing. The step 2 evaluation compared mammography utilization for women randomly assigned to receive a brief reminder letter or no reminder: 42% of the reminder group vs 28% of the control group obtained mammograms. The step 3 evaluation compared utilization for women randomly assigned to receive a second reminder, a letter suggesting a preventive office visit, or telephone counseling: 29% of those who received telephone counseling, 14% of those who received a letter, and 12% of those who received a second reminder obtained mammograms.     

Patient reminder letters to promote annual mammograms: a randomized controlled trial

BACKGROUND: This study assessed the effects of a reminder letter from a physician (relative to a mammography facility letter or no letter) on appointment compliance among women 50-74 years of age due for an annual screening mammogram. METHODS: A total of 1,562 women were randomly as signed to the groups. Each Group 1 subject received a reminder letter from her physician, each Group 2 subject received a reminder letter from her mammography facility, and Group 3 served as a control group. RESULTS: The return rates for Groups 1, 2, and 3 were 47.7, 46.6, and 28.3%, respectively; the overall difference was significant using a chi(2) analysis (P < 0.001). Bonferroni pairwise comparisons indicated no difference between Groups 1 and 2 but significant differences (P < 0.001) between Group 3 and the other two groups. Logistic regression indicated that relative to Group 3, the adjusted odds of returning for Groups 1 and 2 were 2.37 and 2.24, respectively. CONCLUSIONS: Mammography providers and their patients likely will benefit from in-reach reminder systems. Physicians who do not use reminder systems should refer their patients to facilities that use these systems.