Assessing research outcomes by postal questionnaire with telephone follow-up. TOTAL Study Group. Trial of Occupational Therapy and Leisure

BACKGROUND: Face-to-face assessment of research outcomes is expensive and may introduce bias. Postal questionnaires offer a cheaper alternative which avoids observer bias, but non-response and incomplete response reduce the effective sample size and may be equally serious sources of bias. This study examines the extent and potential effects of missing data in the postal collection of outcomes for a large rehabilitation trial. METHODS: Questionnaires containing a number of established scales were posted to participants in a trial of occupational therapy after stroke. Response was maximized by telephone and postal reminders, and incomplete questionnaires were followed up by telephone. Scale scores obtained by imputing values to questionnaire items missing on return were compared with those achieved by telephone follow-up. FINDINGS: Response to the initial posting was 60%, rising to 85% after reminders. Participants receiving the experimental treatment were more likely to respond without a reminder. There were no significant differences on any known factors between eventual responders and non-responders. Of the questionnaires, 43% were incomplete on return: partial responders were significantly different to complete responders on baseline disability and home circumstances. Of the incomplete questionnaires, 71% were resolved by telephone follow-up. In these, the scale scores achieved by telephone were generally higher than those derived by conventional imputation. CONCLUSION: Postal outcome assessment achieved a good response rate, but considerable effort was needed to minimize non-response and incomplete response, both of which could have been serious sources of bias.     

Lifestyle surveys--the complete answer?

STUDY OBJECTIVES: These were as follows: to study incompleteness of data, herein called item non-response, generated by a self completion questionnaire; to identify the characteristics of item non-responders and the types of questions liable to high item non-response rates; and to discuss possible reasons for item non-response. DESIGN: Item non-response patterns in 12,307 responders (62%) to a representative postal survey based on a stratified sample drawn from family health services authorities' (FHSA) registers were investigated. MAIN OUTCOME MEASURES: Data were analysed for item non-response in three groups depending on when the questionnaire was returned (wave analysis). The overall completion rate of the questionnaire was examined and the natural logarithm of the proportion of completed questions was used as an outcome variable in multiple regression analysis. Item nonresponse to key questions and questions of different types was examined. RESULTS: Wave analysis: the overall completion rate of the questionnaire was 86% in questionnaires returned before the first reminder and 83%-84% in those sent back after subsequent reminders. Overall pattern of item non-response; respondents failed to complete a mean of 15% and a median of 10% of the questionnaire. All questions in the questionnaire had some item non-response, ranging from 1% to 85%. Completion rates were associated with gender, age, indicators of lower socioeconomic status, and general health status. Individual questions: particular types of questions were liable to have higher item non-response, for example, linked binary questions. CONCLUSIONS: Item non-response in population postal surveys is likely to present problems in the interpretation of data by introducing bias additional to that of total non-response. Item non-response does not increase greatly with later returns, suggesting that the quality of data across responses generated by two reminders is similar. There are obstacles to reducing item non-response, such as respondent error or socioeconomic and health characteristics of the general population, that cannot be totally overcome. However, the evidence that individuals tend to complete only options within questions that apply to them and their positive behaviour is useful information for those designing questionnaires and interpreting survey data.     

Investigating non-response bias in a survey of disablement in the community: implications for survey methodology.

STUDY OBJECTIVE--The aim was to investigate the pattern of age specific non-response bias in a two phase survey of disablement in the community. It seeks to examine patterns of response in different age groups to a household based postal questionnaire, and the implication of such trends for the estimation of prevalence of reported dependence. It also looks at the effect that the readiness to respond during the first phase postal questionnaire had on participation in the interview based second phase of the study. DESIGN AND SETTING--A two stage survey of disablement in the population was undertaken. A first phase postal questionnaire was sent to 25,168 households in Calderdale, West Yorkshire, England, to ascertain the prevalence of physical disability. The second phase comprised in depth interviews with a sample of individuals identified in the first phase as being disabled. RESPONDENTS--A total of 21,889 postal questionnaires were returned (87%) representing households containing 42,826 people aged 16 years and over. A disproportionately stratified random sample of 950 respondents reporting disability was taken for the second phase. Of these 891 were still available, and 838 (94%) were interviewed. MEASUREMENTS AND MAIN RESULTS--A study of the timing of response to a postal questionnaire showed that patterns differed for different age groups. The estimated prevalence of those aged 65 years and over who were dependent was steady over time whereas for those in the 16-64 age range the estimated prevalence fell as the survey progressed, indicating a tendency for those who were dependent to respond sooner. Examination of the relationship of responses at phase 1 and phase 2 showed that response to invitation to interview was much less in those who had responded later, and presumably more reluctantly, in the first phase. CONCLUSIONS--These findings raise questions about how different patterns of response might be indicative of bias which could differentially affect final age specific prevalence estimates. They also have methodological implications for the follow up of reluctant responders both to increase the response rate and to secure cooperation in the second phase of a two phase survey.     

A confidence interval approach to investigating non-response bias and monitoring response to postal questionnaires.

STUDY OBJECTIVE--The aim was to develop an alternative method of investigating non-response bias in postal surveys, including a method of calculating a final full (100%) coverage confidence interval which avoids the wide intervals of existing approaches. DESIGN AND SETTING--As part of a two stage survey of disablement in the community, a first phase postal questionnaire was sent to 25,168 households in Calderdale, West Yorkshire, England. Confidence intervals were calculated to investigate the precision of estimates using a "no bias" model, where the prevalence in non-responders is assumed to be the same as in responders. RESPONDENTS--A total of 21,889 postal questionnaires were returned (87%), representing households containing 42,826 people aged 16 years and over. This was achieved by the original post (1st wave, 57% response); two further postal follow ups (2nd and 3rd waves, taking the response to 73% and 81% respectively), the latter including a small personal call back; and a final postal follow up (the 4th wave). RESULTS--The cumulative estimated prevalence of those with dependence was plotted as the survey progressed. The final wave full coverage estimated prevalence for those aged 16-64 years was 12.8 per 1000 with 95% confidence intervals of 11.3-14.4 per 1000. The integrity of this estimate holds as long as the true prevalence in non-responders is within the calculated non-response confidence interval under the no bias assumption, 9.7-16.0 per 1000 people. This latter interval represents the tolerance of prevalence in non-responders implied by the no bias assumption. CONCLUSIONS--The findings have general implications for monitoring non-response bias in postal screening questionnaires. The confidence interval approach developed in this paper offers an alternative to existing regression based estimates, giving an indication of the range of prevalence amongst non-responders that could be tolerated before the no bias assumption used by the model is breached. It is suggested that this approach can be used to determine both the extent of bias, and to aid decision making about the appropriate juncture to terminate follow up. It highlights the potential, particularly in the context of a computerised survey operation, of methodological investigation occurring simultaneously with survey operation.     

Baseline characteristics are not sufficient indicators of non-response bias follow up studies.

STUDY OBJECTIVE--The aim was to examine whether baseline characteristics from a cross sectional survey provided sufficient information regarding non-response bias in a follow up study when compared with information on hospital admissions in the intervening years. DESIGN--This was an 11 year follow up study of a cohort selected in 1974 with register information on hospital admissions during follow up. SETTING--The study was based on a sample of cement workers from a particular Portland cement factory with suitable controls from other occupations. PARTICIPANTS--A total of 1404 men participated in the first survey in 1974, including a questionnaire and lung function tests. In 1985 1070 men were alive and of these, 928 men (87%) responded to a postal questionnaire. MAIN RESULTS--Non-responders in 1985 did not differ markedly from responders when smoking habits, respiratory symptoms, and lung function were examined in 1974. During follow up, non-responders had twice as high rates of hospital admission due to respiratory diseases as responders. These differences remained present after adjusting for minor differences in age and smoking habits. CONCLUSIONS--Equal distributions of baseline characteristics among responders and non-responders in a follow up study do not preclude non-response bias.     

Postal surveys of physicians gave superior response rates over telephone interviews in a randomized trial

BACKGROUND AND OBJECTIVES: To compare general practitioner (GP) response to a telephone interview with response to a postal survey with three reminders in a randomized controlled trial. METHODS: GPs were randomly assigned to either a telephone interview or a postal survey. GPs in the telephone group were mailed a letter of invitation and asked to undertake a telephone interview. GPs in the postal group were mailed a letter of invitation and questionnaire. Non-responders were sent up to three reminders, the final by registered post. Response rates were calculated for each group. RESULTS: 416 GPs were randomized to the telephone interview and 451 to the postal survey. Eighty-six in the telephone group and 30 in the postal were ineligible. One hundred thirty-four GPs completed the telephone interview with a response rate of 40.6% (95% confidence interval [CI]: 35.3%, 46.1%). Two hundred fifty-two GPs completed the postal survey with a response rate of 59.9% (95%CI: 55.0%, 64.6%). The difference in response was 19.3% (95%CI: 12.2%, 26.3%). CONCLUSIONS: These results show that postal surveys with three reminders can have superior response rates compared with a telephone interview.     

Does non-responder bias have a significant effect on the results in a postal questionnaire study?

Background and aim: In epidemiological questionnaire studies results can be influenced by non-responder bias. However, in respiratory epidemiology this has been analysed in very few recently published papers. The aim of our paper is to assess if the results found in our previous postal questionnaire study in an adult population in Northern Finland were biased by non-response. Methods: A random sample of 385 persons from the 1284 non-responders in a previous postal questionnaire study was examined. The same questionnaire as in the original study was again mailed to these persons, and those still not answering were contacted by phone. Results: Totally 183 complete answers (48%) were collected. Lack of interest (56%) and forgetting to mail the response letter (22%) were the most common reasons to non-response. Typical non-responders were young men and current smokers who less frequently reported respiratory symptoms in exercise and asthma than the responders in the original study. Answers collected by phone gave for some questions higher prevalence rates than postal answers. Conclusion: Firstly, in this population the response rate (83.6%) in the original study was high enough to provide reliable results for respiratory symptoms and diseases, only the prevalence of current smoking was biased by non-response. Secondly, the methods used for collecting responses in a non-response study may influence the results.     

Non-responders to a postal questionnaire on respiratory symptoms and diseases

Only few data have been published about non-responders in epidemiological studies on respiratory diseases. The aim of this study was to examine the characteristics of the non-responders and the reasons for non-response in a survey of respiratory health. All 9132 subjects, born 1925–1926, 1940–1941, 1955–1956 and 1970–1971 and living in eight representative areas in Norrbotten, Sweden, were invited to a prevalence study on asthma, chronic bronchitis and respiratory symptoms. The response rate was 85%. A sample of the non-responders from the cross-sectional postal questionnaire study was contacted by telephone and interviewed using the same questionnaire as had been used in the postal survey. Of the 1397 non-responders a stratified sample of 182 subjects were selected for this study and 144 agreed to participate. The response rate was increasing by increasing age. The main reason for non-response was that the subjects had forgotten to mail the questionnaire, lack of interest or lack of time. There were significantly higher proportions of current smokers and manual workers among the non-responders. The prevalence rates of wheezing, long-standing cough, sputum production, attacks of breathlessness, asthma and use of asthma medicines were significantly higher among the non-responders compared with the responders according both univariate and multivariate logistic regression analyses, in which the influences of age, sex, smoking habits, socio-economic group and area of domicile were taken into account. The prevalence of respiratory symptoms and diseases was slightly underestimated in the postal survey.     

Does anonymity increase response rate in postal questionnaire surveys about sensitive subjects? A randomised trial.

STUDY OBJECTIVE--The aim of the study was to determine whether complete anonymity improves the response rates to a postal questionnaire. DESIGN--The study derived from a series of postal surveys on AIDS knowledge conducted on six different dates in 1986 and 1987. The sample was randomly divided into two, each group being sent the same questionnaire. One group was informed that the replies were anonymous, the other that they were not. The latter were sent reminders. SETTING--Recipients of the questionnaires were drawn from the Southampton electoral rolls. PARTICIPANTS--300 people in each survey (total 1800) were sent questionnaires, representing on each occasion a different 1:500 systematic sample. RESULTS--Response rate was 49% for the anonymous questionnaires and 51% for the numbered questionnaires. Reminders boosted the response in the numbered group to 72%. CONCLUSIONS--There is no evidence that anonymity improves response to postal questionnaires, but the use of reminders may do so.     

Using the Internet to conduct surveys of health professionals: a valid alternative?

OBJECTIVE: The purpose of this study was to examine whether Internet-based surveys of health professionals can provide a valid alternative to traditional survey methods. METHODS: (i) Systematic review of published Internet-based surveys of health professionals focusing on criteria of external validity, specifically sample representativeness and response bias. (ii) Internet-based survey of GPs, exploring attitudes about using an Internet-based decision support system for the management of familial cancer. RESULTS: The systematic review identified 17 Internet-based surveys of health professionals. Whilst most studies sampled from professional e-directories, some studies drew on unknown denominator populations by placing survey questionnaires on open web sites or electronic discussion groups. Twelve studies reported response rates, which ranged from nine to 94%. Sending follow-up reminders resulted in a substantial increase in response rates. In our own survey of GPs, a total of 268 GPs participated (adjusted response rate = 52.4%) after five e-mail reminders. A further 72 GPs responded to a brief telephone survey of non-respondents. Respondents to the Internet survey were more likely to be male and had significantly greater intentions to use Internet-based decision support than non-respondents. CONCLUSIONS: Internet-based surveys provide an attractive alternative to postal and telephone surveys of health professionals, but they raise important technical and methodological issues which should be carefully considered before widespread implementation. The major obstacle is external validity, and specifically how to obtain a representative sample and adequate response rate. Controlled access to a national list of NHSnet e-mail addresses of health professionals could provide a solution.     

Impact of a postcard versus a questionnaire as a first reminder in a postal lifestyle survey.

STUDY OBJECTIVE--The study aimed to consider the impact of two different types of reminder on response rates and costs in a postal survey. DESIGN--The study was a cross sectional survey. A self-completion lifestyle questionnaire was used. Those who did not respond after the initial mailing were randomly allocated to receive either a postcard or questionnaire as a first reminder. All outstanding non-responders received a questionnaire as a second reminder. SUBJECTS--A representative sample of 698 adults aged 16-70 was used, drawn from a family health services authority register. MAIN RESULTS--Postcard reminders were as effective as questionnaire reminders in increasing response whether one or two reminders are sent. The costs per response were calculated. Two questionnaires as reminders were found to be 1.7 times more expensive than a postcard plus questionnaire. Including the initial mailing, the cost per response using all questionnaires was 1.3 times the cost when a postcard was used for the first reminder. CONCLUSIONS--To increase the response to a postal survey effectively and economically, two reminders should be sent--first a postcard and then a questionnaire.     

A low response rate does not necessarily indicate non-response bias in gastroenterology survey research: a population-based study

Aim

To estimate the potential for response bias in standard mailed questionnaires used in surveys of GI symptoms in a community.

Subjects and methods

Validated self-report tools have been developed to measure functional gastrointestinal (GI) disorders but response rates in community surveys have been rapidly declining in many parts of the world. Whether a lower community response rate introduces significant response bias in GI survey research is unknown. A questionnaire was mailed to a total of 5,069 randomly selected subjects. The overall response rate was 52 %. A random sample of 723 of these subjects (428 responders and 295 non-responders, stratified by age and gender) was selected for medical record abstraction (including both inpatient and outpatient history).

Results

The odds for response increased in those with a higher body mass index (odds ratio (OR):1.02 [95 % CI: 1.01, 1.03]), more health care seeking behavior for non-GI problems (OR: 1.97 [95 % CI: 1.43, 2.72]), and for those who had responded to a previous survey (OR: 4.84 [95 % CI: 2.84, 8.26]). Responder status was not significantly associated with any GI symptoms or a diagnosis of GI or non-GI disease (with two exceptions, diverticulosis and skin disease).

Conclusions

Despite a response rate of only 52 %, the results of a community-based GI survey do not appear to be impacted by non-response bias in a major way. A low survey response rate does not necessarily indicate non-response bias.     

Evaluation of non-response bias in mental health determinants and outcomes in a large sample of pre-adolescents

Since non-response may jeopardize the validity of studies, comprehensive assessment of non-response is a prerequisite for proper interpretation of study findings. Recently, the baseline assessment of the TRacking Adolescents Individual Lives Survey (TRAILS), a prospective cohort study among Dutch pre-adolescents, was completed. The aim of this report is to examine non-response bias by comparing responders and non-responders regarding mental health determinants (e.g., demographics and cognitive performance) and outcomes, as well as associations between the two. Furthermore, we examine whether extended efforts to recruit participants contribute to the prevention or reduction of non-response bias. Thanks to various recruitment procedures, the initial response rate of 66% increased to a final rate of 76%. The extended efforts to recruit participants prevented non-response bias in the prevalence rates of psychopathology. Although non-responders differed from responders with respect to several individual characteristics, no significant differences were found regarding associations between these characteristics and psychopathology. We conclude that TRAILS provides a solid basis to improve our understanding of the development of mental health during adolescence.     

Investigating non-response bias in mail surveys.

Losses in follow-up that are biased with respect to outcome invalidate the results. There are many ways of dealing with non-response in follow-up studies. Three separate methods were used to investigate a potential bias in a mail survey of 2471 disabled people. At a response rate of 84%, the non-respondents were significantly different from the respondents with respect to the outcome, return to work and vocational training. The success rate in terms of the outcome was negatively related to the number of reminders. Significant differences were found in response rates according to age, social class, impairments, previous employment record, and completion of rehabilitation courses. There is no safe level of response rates below 100%. However small the non-response, a possible bias as a result of it must be investigated.     

Characteristics of non-response in the Danish Health Interview Surveys, 1987-1994

BACKGROUND: The types and quantity of non-response in surveys influence the extent to which the results may be generalized. This study analysed trends in non-response in the Danish Health Interview Surveys from 1987 to 1994 and used the National Patient Registry to assess whether non-response biased the estimated population prevalence of morbidity when solely based on responders. METHODS: The data were for the 23,096 adults sampled for the Danish Health Interview Surveys in 1987, 1991 and 1994. All were followed using the National Patient Registry to obtain such information as hospital admissions. RESULTS: Non-response increased from 20.0% in 1987 to 22.6% in 1994. Four combinations of background variables characterized the non-response: gender and age; gender and civil status; county of residence and age; survey year and age. Non-respondents and respondents had identical gender- and age-standardized hospital admission rates for approximately 5 years before and 2 years after data collection, but non-respondents had a significantly higher rate immediately before and during data collection. Admissions rates were analysed according to reasons for non-response. Refusers had a lower admission rate than respondents before data collection but similar during and after data collection. The rate was higher during the whole period among ill or disabled non-respondents. Among people who could not be contacted during the data collection period a higher admission rate was only found immediately before and during data collection. CONCLUSIONS: Although admission rates differed between respondents and non-respondents these differences were too small to bias the estimated population prevalence of morbidity when solely based on respondents.     

Postal survey on airborne occupational exposure and respiratory disorders in Norway: causes and consequences of non-response.

STUDY OBJECTIVE--The aim was to examine causes for non-response in a community survey, and how non-response influences prevalence estimates of some exposure and disease variables, and associations between the variables. DESIGN--This was a cross sectional questionnaire study with two reminder letters. The questionnaire asked for information on smoking habits, occupational airborne exposure and respiratory disorders. SETTING--A random sample of 4992 subjects from the general population aged 15-70 years of Hordaland County, Norway. MAIN RESULTS--The overall response rate was 90%, with a 63% response to the initial letter. The response rates to the first and second reminder letters were 56% and 36% respectively. In 20% of the non-respondents an uncompleted questionnaire was returned with cause for non-response; in two thirds of these the cause for non-response was that the subject was not resident at the mailing address. A home visit to a random sample of 50 urban non-respondents provided further information on 29 subjects. A wrong address at the Central Population Registry and the subject's feeling of lack of personal benefit from a postal survey were the major reasons for non-response. Smokers were late respondents and subjects with respiratory disorders tended to be early respondents. CONCLUSION--The main reasons for non-response were a wrong mailing address and a feeling of lack of personal benefit from responding. Using only the initial letter would have changed the estimated prevalence of smokers from 39% to 35%. Otherwise, the estimated prevalence of the exposure and disease variables as well as the associations between them were only slightly changed after including the respondents to the first and second reminder letters.     

Female specialists were more likely to respond to a postal questionnaire about drug treatments for Alzheimer disease

OBJECTIVE: To assess demographic predictors of response for specialists who were mailed a postal questionnaire on drug treatments for Alzheimer disease. STUDY DESIGN AND SETTING: The questionnaire was sent to 317 specialists in Quebec, Canada. Demographic variables included specialty, urban/rural practice, language, sex, and 'number of years since receiving a medical license.' The specialists were stratified according to responder status (i.e., respondent/nonrespondent), and respondents were further stratified as early or late responders. Variables differing between these strata were entered into logistic regression models to see if they predicted response. RESULTS: Only 'female sex' was a predictor of response in the respondent/nonrespondent analysis (OR 2.03; 95% CI 1.17, 3.53). No demographic variables predicted early or late response. CONCLUSION: Researchers planning postal questionnaires should target male specialists with modified or additional mailings to increase response and reduce the potential for nonresponse bias. Caution should be exercised when comparing early vs. late responders as a means of assessing nonresponse bias.     

GP survey response rate: a miscellany of influencing factors

BACKGROUND: GP response to surveys is acknowledged to vary widely. The minimization of non-response bias and the generalizability of findings are fundamental research issues. OBJECTIVE: The aim of this study was to identify the factors that had influenced GPs' decisions to complete a questionnaire. METHODS: A short questionnaire eliciting GPs' views on minor ailment consultations was sent to 759 GPs from eight English health authorities. The response rate was 54.5%. Factors perceived by respondents to have influenced their decision to complete this questionnaire were also assessed. Subsequently, a feedback summary, together with a short evaluation form, was sent to those GPs requesting it. RESULTS: The response from GPs located in the London area was significantly lower than that from those elsewhere. Respondents identified questionnaire length and the originating institution as the two major factors influencing their decision to return the survey. A single mailing of the evaluation form yielded a response rate of >60% predominantly positive comments. CONCLUSION: Many factors influence a GP's decision to complete a survey. The effect of location has, to date, largely been ignored. Furthermore, this study suggested feedback to be an important issue. Within health services research, all possible factors need to be considered to maximize response, reduce non-response bias and ultimately facilitate the dissemination of findings.     

A comparison of general practitioner response rates to electronic and postal surveys in the setting of the National STI Prevention Program

Objective: To compare the response rates achieved for an online survey with a postal survey of general practitioners (GPs) as a method to evaluate the National STI Prevention Program. Methods: All GPs in Australia were asked to complete an online survey. A further sample of 509 GPs were asked to complete a postal survey. Response rates to both recruitment methods were compared. The demographic characteristics of responders were compared to the entire GP population of Australia. Results: Twenty GPs completed the online survey (response rate <0.1%). Sixty‐three GPs completed the postal survey (response rate 12.4%). The demographic characteristics of those responding to the postal survey showed no statistically significant difference compared to the general GP population. Conclusion: Our postal survey had a higher response rate than the online survey. Our response to the postal survey was lower than other similar studies and is likely to be due to a lack of incentives and follow‐up. Even with the low response rate it appears that postal surveys can provide a good representation of the overall population. Implications: Despite growing use of online surveys, postal surveys should still be the method of choice whenever possible. Postal surveys should include incentives and further follow‐up of the initial recruitment should be conducted.     

Partial questionnaire designs, questionnaire non-response, and attributable fraction: applications to adult onset asthma

The attributable fraction (AF) is often used to explore the policy implications of an association between a disease and an exposure. To date, there have been no proposed estimators of AF in the context of partial questionnaire designs (PQD). The PQD, first proposed in a public health context by Wacholder is often used to enhance response rates in questionnaires. It involves eliciting responses from each subject on preassigned subsets of questions, thereby reducing the burden of response. We propose a computationally efficient method of estimating logistic (or more generally, binary) regression parameters from a PQD model where there is non-response to the questionnaire and the rates of non-response differ between sub-populations. Assuming a log-linear model for the distribution of missing covariates, we employ the methods of Wacholder to motivate consistent estimating equations, and weight each subject's contribution to the estimating function by the inverse probability of responding to the questionnaire. We also propose techniques for goodness-of-fit to assist in model selection. We then use the PQD regression estimates to derive an estimate of AF similar to that proposed by Bruzzi. Finally, we demonstrate our methods using data obtained from a study on adult occupational asthma, conducted within a Massachusetts HMO. Although we concentrate on a particular type of missing data mechanism, other missing data techniques can be incorporated into AF estimation in a similar manner.