Nonresponse bias in a mail survey of physicians

With the increased pressure on survey researchers to achieve high response rates, it is critical to explore issues related to nonresponse. In this study, the authors examined the effects of nonresponse bias in a mail survey of physicians (N = 3,400). Because slightly more than one half of the sample did not respond to the survey, there was potential for bias if nonresponders differed significantly from responders with respect to key demographic and practice variables. They analyzed response status and timing of response with respect to five variables: gender, region, specialty, urbanicity, and survey length. The potential consequences of nonresponse bias on the survey estimates were then analyzed. Men were more likely to respond, as were physicians receiving a shorter questionnaire. Repeated follow-up attempts reduced gender response bias because male physicians were more likely to be early responders. Overall, higher response rates were not associated with lower response bias.     

Analysis of nonresponse bias in a population-based case-control study on lung cancer

The objective of this study was to identify characteristics of nonrespondents and late respondents in a population-based case-control study on lung cancer conducted in the city of Turin (Italy). Information about demographic and socioeconomic variables of 634 cases and 859 controls who responded to an interview, as well as of 154 cases and 154 controls who did not respond were obtained from census and the public register of Turin. The socioeconomic level of nonrespondents was high in cases but low in controls. Late respondent controls (i.e., individuals who responded after contact through their general practitioner) had socioeconomic characteristics comparable with those of nonrespondents, while they were similar to respondents with respect to demographic variables. The interview of late respondents halved, from 14 to 7%, the magnitude of the bias introduced by nonresponse on the estimate of the association between educational level and lung cancer. Nonresponse, associated with socioeconomic status, is an important potential source of bias in population-based case-control studies, which should always be considered and discussed. The direction and magnitude of the bias can be quantified. General practitioners may contribute to decrease nonresponse bias. Caution should be used in inferring characteristics of nonrespondents on the basis of those of late respondents.     

Absence of nonresponse bias in a study of sport-caught Great Lakes fish consumption and conception failure

INTRODUCTION: We have reported that men's lifetime sport-caught Great Lakes fish consumption was associated with lifetime prevalence of conception delay or failure. Those cross-sectional data were based on responses to a postal questionnaire. The present study was conducted to evaluate whether nonresponse bias could explain the cross-sectional findings. METHODS: We conducted telephone interviews with 230 men and 38 women who did not respond to the original survey, and compared these individuals to the original responders with respect to key demographic, behavioral, and reproductive characteristics. RESULTS: Nonresponders were approximately 1.5 years older at interview, were more likely to be Caucasian, and reported higher incomes than responders. Among men, nonresponders had fished fewer days in the past year (12% reported no fishing, compared to 4. 3% of responders). Almost one half of nonresponders reported no fish consumption in the past year, compared to one quarter of responders. Nonresponders were more likely than responders to have ever conceived a live-born child, had more children, and were less likely to intend to have additional children in the next 5 years. Among both responders and nonresponders there was an increased prevalence of a period of conception failure among men who reported consuming greater quantities of sport-caught Great Lakes fish. DISCUSSION: Our study provides support for the cross-sectional analyses presented previously, insofar as nonresponse bias is unlikely to have a major role in the observed association.     

Late response to patient-reported outcome questionnaires after surgery was associated with worse outcome

ObjectivesNonresponse to patient-reported outcome (PRO) questionnaires after surgery might bias the results. Our aim was to gauge the potential impact of nonresponse bias by comparing the outcomes of early and late responders.Study Design and SettingThis study compares 59,565 early and 20,735 late responders who underwent a hip or knee replacement, hernia repair, or varicose vein (VV) surgery. The association between timeliness of response and three outcomes (the mean postoperative disease-specific PRO and generic PRO scores and the proportion reporting a fair or poor result) was examined by regression analysis.ResultsLate responders were slightly more likely to be young, nonwhite, deprived, and have a more severe preoperative condition with poorer quality of life. Late response was associated with a slightly poorer outcome in all four procedures although not statistically significant (P < 0.05) for VV surgery. Adjusting for preoperative characteristics reduced the strength of the associations, but they remained statistically significant.ConclusionAs nonresponse to PRO questionnaires introduces slight bias, differences in response rates between hospitals should be taken into account when making comparisons so as to avoid overestimating the performance of those with lower response rates and failing to detect poor performing hospitals.     

Use of medication by Brazilian retirees: 2 - response rate and item completeness in a postal survey in Belo Horizonte, Minas Gerais State, Brazil

This paper reports on the response rate and completeness of item response in a self-administered postal survey questionnaire on use of medication by retirees 60 years or older under the Brazilian Social Security System, in Belo Horizonte, Minas Gerais State, in 2003. Questionnaires were sent in two rounds to 800 postal addresses of subjects selected by simple random sampling. The response rate was 47.8%, and there were no significant differences in the selected characteristics between respondents and non-respondents, or between early and late respondents. For almost all socio-demographic and health variables, item omission was less than or equal to 5% for both the entire sample and early or late responders. For all the medicines used, the dose and laboratory were the main items omitted. The findings indicate that details on drug use should be reevaluated in self-administered questionnaires.     

Nonresponse bias and cost-effectiveness in a Norwegian survey of family physicians

Low response rates are a common problem in surveys of family physicians leading to uncertainty about the validity of results. In this study, the authors examined the association between multiple reminders and nonresponse bias, survey estimates and costs in a survey of family physicians in Norway (N = 3,463). After three postal reminders and one telephone follow-up, the response rate was 65.9%. They analyzed differences in nine demographic and practice variables between respondents and nonrespondents, the effect of nonresponse bias on survey estimates, and the cost-effectiveness of each reminder. Statistically significant differences between respondents and nonrespondents were found for six variables. However, demographic and practice variables had little association with the main outcome variables, and the overall survey estimates changed little with additional reminders. In addition, the cost-effectiveness of the final reminders was poor.     

Increasing response to a postal survey of sedentary patients - a randomised controlled trial [ISRCTN45665423]

Background

A systematic review identified a range of methods, which can influence response rates. However, analysis specific to a healthcare setting, and in particular, involving people expected to be poor responders, was missing, We examined the effect of pre-warning letters on response rates to a postal survey of sedentary patients whom we expected a low rate of response.

Methods

Participants were randomised to receive a pre-warning letter or no pre-warning letter, seven days before sending the main questionnaire. The main questionnaire included a covering letter and pre-paid return envelope. After seven days, non-responders were sent a reminder letter and seven days later, another reminder letter with a further copy of the questionnaire and return envelope.

Results

627 adults, with a mean age of 48 years (SD 13, range 18 to 78) of whom 69.2% (434/627) were women, were randomised. 49.0% (307/627) of patients were allocated to receive a pre-warning letter and 51.0% (320/627) no pre-warning letter, seven days in advance of posting the main questionnaire. The final response rate to the main questionnaire was 30.0% (92/307) amongst those sent a pre-warning letter and 20.9% (67/320) not sent a pre-warning letter, with an adjusted odds ratio of 1.60 (95% CI 1.1, 2.30).

Conclusions

The relatively low cost method of sending a pre-warning letter had a modest impact on increasing response rates to a postal questionnaire sent to a group of patients for whom a low response rate was anticipated. Investigators should consider incorporating this simple intervention when conducting postal surveys, to reduce the potential for nonresponse bias and to increase the study power. Methods other than postal surveys may be needed however when a low response rate to postal surveys is likely.     

Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of GPs

BACKGROUND: Non-response is an important potential source of bias in survey research. With evidence of falling response rates from GPs, it is of increasing importance when undertaking postal questionnaire surveys of GPs to seek to maximize response rates and evaluate the potential for non-response bias. OBJECTIVES: Our aim was to investigate the effectiveness of follow-up procedures when undertaking a postal questionnaire study of GPs, the use of publicly available data in assessing non-response bias and the development of regression models predicting responder behaviour. METHOD: A postal questionnaire study was carried out of a random sample of 600 GPs in Wales concerning their training and knowledge in palliative care. RESULTS: A cumulative response rate graph permitted optimal timing of follow-up mailings: a final response rate of 67.6% was achieved. Differences were found between responders and non-responders on several parameters and between sample and population on some parameters: some of these may bias the sample data. Logistic regression analysis indicated medical school of qualification and current membership of the Royal College of General Practitioners to be the only significant predictors of responders. Late responders were significantly more likely to have been qualified for longer. CONCLUSIONS: This study has several implications for future postal questionnaire studies of GPs. The optimal timing of reminders may be judged from plotting the cumulative response rate: it is worth sending at least three reminders. There are few parameters that significantly predict GPs who are unlikely to respond; more of these may be included in the sample, or they may be targeted for special attention. Publicly available data may be used readily in the analysis of non-response bias and generalizability.     

Determinants of non-response in an occupational exposure and health survey in New Zealand

Objective: Study the determinants of non‐response and the potential for non‐response bias in a New Zealand survey of occupational exposures and health. Methods: A random sample of 10,000 New Zealanders aged 20–64 years were invited by mail to take part in a telephone survey. Multiple logistic regression was used to study the determinants of non‐response. Whether occupational exposure, lifestyle and health indicators were associated with non‐response was studied by standardising their prevalence towards the demographic distribution of the source population, and comparing early with late responders. Results: The response rate was 37%. Younger age, Māori descent, highest and lowest deprivation groups and being a student, unemployed, or retired were determinants of non‐contact. Refusal was associated with older age and being a housewife. Prevalence of key survey variables were unchanged after standardising to the demographic distribution of the source population. Conclusions: Following up the non‐responders to the mailed invitations with telephone calls more than doubled the response rate and improved the representativeness of the sample. Although the response rate was low, we found no evidence of major non‐response bias. Implications: Judgement regarding the validity of a survey should not be based on its response rate.     

Comparison of characteristics between respondents and nonrespondents in a national survey of Japanese elderly using six year follow-up study]

PURPOSES: The purpose of this study were twofold: 1) to examine differences between respondents and nonrespondents in sociodemographic or health characteristics, 2) to study nonresponse effects on relationships between variables, using a 6 year follow-up study for both respondents and nonrespondents at the initial survey. METHODS: The data were collected in 1987 through a national survey of Japanese adults aged 60 and over. A total of 2,200 interviews were completed from the list of 3,288 names. In 1993, 1,010 nonrespondents excluding persons who had died, moved, or whose addresses were unknown in the prior interview, were recontacted through a mail questionnaire. A total of 559 persons completed the mail questionnaire. Of the original 2,200 baseline interviewees, some by proxy interviews, 2,260 persons were reinterviewed, at the same time as the mail survey. Sociodemographic and health variables (age and sex), social indicators (educational attainment, marital status, and job status), health status (mortality, existence of diseases, and activities of daily living), subjective well-being (life satisfaction, self-rated health, and economic satisfaction) were compared between respondents and nonrespondents. Relationships between self-rated health and sociodemographic or health variables were examined by multiple regression analysis. RESULTS: 1. Compared to people who participated in the survey, norespondents were likely to be male, in the lower age categories, and with higher educational attainment at the follow-up survey. Also, life satisfaction and self-rated health were lower in nonrespondents than in respondents. Reasons for nonresponse varied but appeared to be somewhat related to characteristics of nonrespondents. 2. No significant relationships between self-rated health and sociodemographic or health variables appeared for the respondent group and also when including the nonrespondent group. CONCLUSION: While differences between respondents and nonrespondents on certain variables were significant, relationships between self-rated health and sociodemographic variables were not observed.     

Census and geographic differences between respondents and nonrespondents in a case-control study of non-Hodgkin lymphoma

To quantify nonresponse bias and estimate its potential impact, the authors compared census-based socioeconomic and demographic factors and geographic locations among respondents and nonrespondents in a multicenter case-control study of non-Hodgkin lymphoma (1998-2000). Using a geographic information system, the authors mapped current addresses and linked them to the 2000 US Census database to determine group-level demographic and socioeconomic information. They used logistic regression analysis to compute the risk of being a nonrespondent, separately for cases and controls. They used spatial scan methods to evaluate spatial clustering at each study center. Among cases at one or more centers, nonresponse was significantly associated with non-White race, lower household income, a greater proportion of multiple-unit housing, fewer years of education, and living in a more urbanized area. For most factors, the authors observed similar patterns among controls, although findings were mostly nonsignificant. They found two nonrandom elliptical clusters in Los Angeles, California, and Detroit, Michigan, that disappeared after adjustment for the demographic factors. The authors determined the bias in non-Hodgkin lymphoma risk associated with census-tract educational level by comparing risks among respondents and all subjects. The bias was 8%, indicating that the socioeconomic and demographic differences between respondents and nonrespondents did not result in a large bias in the risk estimate for education.     

Lifestyle surveys--the complete answer?

STUDY OBJECTIVES: These were as follows: to study incompleteness of data, herein called item non-response, generated by a self completion questionnaire; to identify the characteristics of item non-responders and the types of questions liable to high item non-response rates; and to discuss possible reasons for item non-response. DESIGN: Item non-response patterns in 12,307 responders (62%) to a representative postal survey based on a stratified sample drawn from family health services authorities' (FHSA) registers were investigated. MAIN OUTCOME MEASURES: Data were analysed for item non-response in three groups depending on when the questionnaire was returned (wave analysis). The overall completion rate of the questionnaire was examined and the natural logarithm of the proportion of completed questions was used as an outcome variable in multiple regression analysis. Item nonresponse to key questions and questions of different types was examined. RESULTS: Wave analysis: the overall completion rate of the questionnaire was 86% in questionnaires returned before the first reminder and 83%-84% in those sent back after subsequent reminders. Overall pattern of item non-response; respondents failed to complete a mean of 15% and a median of 10% of the questionnaire. All questions in the questionnaire had some item non-response, ranging from 1% to 85%. Completion rates were associated with gender, age, indicators of lower socioeconomic status, and general health status. Individual questions: particular types of questions were liable to have higher item non-response, for example, linked binary questions. CONCLUSIONS: Item non-response in population postal surveys is likely to present problems in the interpretation of data by introducing bias additional to that of total non-response. Item non-response does not increase greatly with later returns, suggesting that the quality of data across responses generated by two reminders is similar. There are obstacles to reducing item non-response, such as respondent error or socioeconomic and health characteristics of the general population, that cannot be totally overcome. However, the evidence that individuals tend to complete only options within questions that apply to them and their positive behaviour is useful information for those designing questionnaires and interpreting survey data.     

Participation in a mail survey: role of repeated mailings and characteristics of nonrespondents among recent mothers

This study analysed the characteristics of respondent and nonrespondent mothers at each stage of a survey procedure, from a initial questionnaire to a reminder letter and two repeated mailings. Of 938 mothers of liveborn children who, while maternity inpatients, received a questionnaire and information about a mail survey to follow 2 months later, 828 completed and returned the initial questionnaire, 708 agreed to participate in the mail survey and were sent the mail questionnaire, and 612 finally completed and returned the questionnaire at 2 months. There were differences between respondents and non-respondents for socio-demographic factors at each stage of the process. The final response rate to the mail questionnaire was higher among mothers who were younger, were breast feeding, and had more education, an occupation and fewer children. The characteristics of late respondents were intermediate between those of early to middle respondents and nonrespondents for age, educational level, breast feeding and occupation. Maternal and infant health varied only slightly according to response status. Repeated mailings increased response and diminished selection. A mail questionnaire after contact in a maternity ward is a cost-effective means of gathering data about a large sample of recent mothers and their children.     

A comparison of general practitioner response rates to electronic and postal surveys in the setting of the National STI Prevention Program

Objective: To compare the response rates achieved for an online survey with a postal survey of general practitioners (GPs) as a method to evaluate the National STI Prevention Program. Methods: All GPs in Australia were asked to complete an online survey. A further sample of 509 GPs were asked to complete a postal survey. Response rates to both recruitment methods were compared. The demographic characteristics of responders were compared to the entire GP population of Australia. Results: Twenty GPs completed the online survey (response rate <0.1%). Sixty‐three GPs completed the postal survey (response rate 12.4%). The demographic characteristics of those responding to the postal survey showed no statistically significant difference compared to the general GP population. Conclusion: Our postal survey had a higher response rate than the online survey. Our response to the postal survey was lower than other similar studies and is likely to be due to a lack of incentives and follow‐up. Even with the low response rate it appears that postal surveys can provide a good representation of the overall population. Implications: Despite growing use of online surveys, postal surveys should still be the method of choice whenever possible. Postal surveys should include incentives and further follow‐up of the initial recruitment should be conducted.     

Late response and item nonresponse in the Finbalt Health Monitor survey

BACKGROUND: The Finbalt Health Monitor is a collaborative system for monitoring the health-related behaviour, practices and lifestyles in Estonia, Finland, Latvia and Lithuania. This system is based on nationally representative samples and self-administered mailed questionnaires. In comparing the results of national surveys, the awareness of the direction and socioeconomic patterning of the response bias is essential. METHODS: The data were gathered from the cross-sectional surveys conducted in 1998 from Estonia (n = 1362), Finland (n = 3504), Latvia (n = 2322) and Lithuania (n = 1874). An analysis was made of the prevalence of late response, completeness of information obtained from respondents and the magnitude of response bias on the prevalence estimates of health behaviour indicators. RESULTS: The response rates were comparatively high: 68% in Estonia, 70% in Finland, 77% in Latvia and 62% in Lithuania. Late response was weakly related to age, education or place of residence. The total proportion of missing information was below 10% and the sociodemographic patterning for this missing information was similar in all countries. Thus, older and less-educated respondents had more missing information on their questionnaires. Response bias of the prevalence estimates was minimal when it was calculated by using information obtained from late respondents. CONCLUSIONS: The level of nonresponse and missing information was comparable in different countries, not information on health behaviour. Therefore special efforts are needed to design a questionnaire form which appears equally relevant to all respondent groups. The follow-up mailings were an effective way to increase the total response rate, but it was unlikely that they provided an effective way to reach the 'hard core' nonrespondents.     

Selected questionnaire size and color combinations were significantly related to mailed survey response rates

OBJECTIVE: To determine the degree to which mailed survey response rates, response times, and nonresponse bias are affected by questionnaire size and color. STUDY DESIGN AND SETTING: Questionnaires were mailed to a random sample of 2,000 Mayo Clinic patients in one of four size/color "test" groups. One thousand three hundred nine surveys were completed, approximately two-thirds in each group. RESULTS: A small (6 (1/8) x 8 (1/4) in) questionnaire booklet on white paper had a higher response rate (68.4%) than a similarly sized questionnaire on blue paper (62.3%). A large (8 (1/4) x 11 in) questionnaire on white paper had a 62.7% rate, whereas a large, blue questionnaire had a response rate of 68.6%. Median response times did not differ by questionnaire size/color. No evidence of differential nonresponse bias was observed across the four test groups. CONCLUSION: This study supports the use of a small/white questionnaire format advocated by the Total Design Method advanced by Don Dillman at Washington State University. We observed a favorable response rate for a large questionnaire printed on blue paper; however, if time and resources are limited, use of a small/white questionnaire appears preferable.     

Comparison of early and delayed respondents to a postal health survey: a questionnaire study of personality traits and neuropsychological symptoms

Delayed response to a mail survey is related to age, lifestyle and socioeconomic status, and may provoke bias in epidemiological study. We investigated whether delayed respondents were associated with their personality traits and neuropsychological symptoms. Two hundred and ninety-eight painters from a Scottish dockyard cohort and their 571 male controls (general subjects) randomly recruited from the local residents completed a questionnaire, which included 24 statements of the Eysenck personality scales and 22 questions of neuropsychological symptoms. There was a similar distribution of delayed response between general subjects and painters, with a total of 55.6% early (returning questionnaires within 4 weeks), 33.4% intermediate (within 8 weeks) and 11.0% late respondents (after 8 weeks). The delayed response was related to only a few individual statements or symptoms, which varied between general subjects and painters. There were no significant differences in scores in statements of personality traits and neuropsychological symptoms among the three respondent groups, except for the late respondents in painters having an increased score of total neuropsychological symptoms at borderline significance. After adjustment for confounding the case–control analysis showed no significant association of the high scores of social conformity, neuroticism and symptoms with a delayed response. This study suggests that response to a postal health survey may not be influenced or biased by personality traits and neuropsychological symptoms.     

Contribution of follow-up of nonresponders to prevalence and risk estimates: a Norwegian respiratory health survey

Achieving a high response rate is often expensive and time consuming. Does an extensive survey effort change prevalence estimates and exposure-disease relations? In 1998-1999, the authors conducted a population-based respiratory health survey in two Norwegian counties (Oslo, n = 20,000; Hordaland, n = 5,000) of a random sample of the adult population aged 15-70 years. A postal questionnaire was mailed, with as many as two reminder letters. A 25% random sample of postal nonresponders was contacted by ordinary or cellular telephone. Cumulative response rates after the first mailing, first reminder, second reminder, and telephone follow-up were 42.7%, 60.7%, 68.3%, and 79.9%, respectively. Compared with initial postal responders, responders at later stages were younger, and more were male. Late responders had a lower educational level, were more often smokers, and were occupationally exposed to dusts or gases more frequently. After the authors adjusted for these factors, late responders were found to have less asthma, attacks of breathlessness, and hay fever. Hardly any changes in prevalences and odds ratios were noted when initial responders were compared with all responders. The additional contribution of sending reminder letters and conducting a telephone follow-up to prevalence estimates and the exposure-disease relation was small. A single letter would have produced nearly the same prevalence estimates and risk factor associations.     

Nine-year follow-up of a survey on smoking habits in Florence (Italy): higher mortality among non-responders

BACKGROUND: Smoking prevalence is often assessed in random samples of a population. Non-response bias has been rarely investigated. METHODS: In 1989 a survey on smoking habits in Florence, Italy, was carried out (response rate: 85%). For responders and non-responders (3,621 subjects) the life status as of 1998 was assessed. Poisson regression models were fitted to estimate age-adjusted risks of death (RR) of non-responders for overall mortality and for the most important causes of death, taking the whole series of responders, postal responders and telephone responders as the reference in different analyses. This analysis included 2,071 subjects aged >/=45 years. RESULTS: Compared to the whole series of responders, mortality from all causes was significantly higher among non-responders in males (RR = 1.74; 95% CI: 1.23-2.44) and females (RR = 2.45; 95% CI: 1.79-3.29). The higher risk was seen for smoking-related and 'other' causes of death. Among females the difference was more evident for smoking-related causes (RR = 3.14; 95% CI: 1.66-5.93), among males the higher risk was similar for both groups of causes. The excess of mortality was less evident when telephone responders alone were taken as reference. CONCLUSIONS: The follow-up of subjects enrolled in a survey on smoking habits shows high mortality risks among non-responders. The data indirectly suggest that smoking was (or had been) more widespread among non-responders, in particular among females. Therefore, the prevalence of smokers assessed through this survey, focussed on smoking habit, may be underestimated. Telephone contact with non-responders to the postal questionnaire attenuated the selection bias of responders, but even with telephone back-up the response bias persisted.