A comparison of response rate, data quality, and cost in the collection of data on sexual history and personal behaviors. Mail survey approaches and in-person interview

The authors examined differences in rate of response, data quality, and cost between mail approaches and in-person interview in the collection of data on sexual history and personal behaviors. A sample of women from a midwestern United States university (n = 342) was identified from health service medical records as having been seen for a sexually transmitted disease (cases) or a contraceptive visit (controls) during the latter half of 1985. The women were randomly assigned to one of three data collection strategies. A total of 268 subjects (78%) participated. Results indicated no differences in validity by method of data collection or by case-control status but there were significant differences in completeness, cost, and response rates. In-person interviews resulted in more complete data than mail approaches, although all instruments had low proportions of missing data (0.001-0.006). Response rate differences were not found when data collection methodologies were compared (75-82%) but were found in case-control analyses. Cases were consistently less likely to participate and significantly less likely to respond by mail (p less than 0.05). The cost of the in-person interview was approximately four times that of the mail survey for the data collection. Implications of the case-control response rate difference suggest that mail methodologies, although low in cost, may introduce sampling bias in studies of sexually transmitted diseases.     

Comparisons of the costs and quality of patient data collection by mail versus telephone versus in-person interviews

The costs and quality of three modes of patient data collectionwere compared in a survey on the quality of health care services.Nine hundred and forty-eight non-institutionalized participantsfrom four patient categories (asthma, rheumatic diseases, disabledand dependent elderly) were divided into three different groupsfor the allocation of either a self-administered mail survey,a computer-assisted telephone interview or an in-person interview.The three modes were compared in terms of the total survey costs,the survey response rates and the data quality. The resultsshow that the mail mode was the most cost-effective and yieldedconsistently high response rates. The telephone interviews yieldedfar more complete data than the other interview modes. Althoughin-person interviews have traditionally been considered to yieldhigher response rates, less non-response bias and better dataquality, the results of this study show different figures infavour of mail surveys.     

The effect of different methods of collecting data: Mail,telephone and filter data collection issues in utility measurement

When collecting data researchers can interview participants, conduct telephone interviews, or mailout questionnaires. Often mixed methods are used. Whether these methods produce equivalent data is under-researched in the health-related quality of life (HRQoL) field. In addition, the effect of using filter questions has not been researched among HRQoL multi-attribute utility instruments. This study randomly sampled from Melbourne, Australia, and employed a test–retest design to investigate whether mail or telephone interview made any difference to Assessment of Quality of Life (AQoL) instrument utility scores. A filter question AQoL version investigated the effect of filters on scores. There was no significant difference in standard AQoL scores between mail self-completion and telephone interview, regardless of which was administered first. Inclusion of filter questions encouraged respondents to select the best response category thereby screening out minor health conditions. The effect was to increase utility scores by 0.06 or 7%. This effect has not been previously reported in utility instruments and has profound implications for economic evaluations using cost-utility analysis; there are implications for researchers using filter questions in general. In conclusion, researchers should feel confident that utility scores elicited from the standard AQoL through self-completion mail and telephone interview administrations are directly comparable.     

Telephone follow-up was more expensive but more efficient than postal in a national stroke registry

ObjectiveTo compare the efficiency and differential costs of telephone- vs. mail-based assessments of outcome in patients registered in a national clinical quality of care registry, the Australian Stroke Clinical Registry (AuSCR).Study Design and SettingThe participants admitted to hospital with stroke or transient ischemic attack were randomly assigned to complete a health questionnaire by mail or telephone interview at 3–6 months postevent. Response rate, researcher burden, and costs of each method were compared.ResultsCompared with the participants in the mail questionnaire arm (n = 277; 50% female; mean age: 70 years), those in the telephone arm (n = 282; 45% female; mean age: 68 years) required a shorter time to complete the follow-up (mean difference: 24.2 days; 95% confidence interval [CI]: 15.0, 33.5 days). However, the average cost of completing a telephone follow-up was greater (US$20.87 vs. US$13.86) and had a similar overall response to the mail method (absolute difference: 0.57%; 95% CI: ?4.8%, 6%).ConclusionPosthospital stroke outcome data were slower to collect by mail, but the method achieved a similar completion rate and was significantly cheaper to conduct than follow-up telephone interview. Findings are informative for planning outcome data collection in large numbers of patients with acute stroke.     

Use of medications by elderly Brazilian retirees: 1--methodology and response rates of a multicenter survey

This paper reports methodological aspects and response rates of a survey with the main objective of describing the drug utilization profile of retirees of the Institute of National Social Security of Brazil, 60 years of age or older, in three simple random samples: Brazil (countrywide), Belo Horizonte municipality, and Rio de Janeiro municipality. This cross-sectional study employed two approaches: mail (self-report questionnaires) and home interviews (questionnaires completed by interviewers). This strategy allowed evaluation of response agreement between the mail and home interview approaches. In the national sample of 3,000 individuals, 1,025 only responded to the self-report questionnaire. Despite this the low response rate, there was reasonable resemblance between respondents and non-respondents, suggesting that this sample is representative of the target population. In Belo Horizonte and Rio de Janeiro the initial samples of 800 individuals were replicated for both approaches, mail and home interview. The response rates to the mail surveys were 46.8% and 34.4% in Belo Horizonte and Rio de Janeiro, respectively. The response rates to the home interviews were 80.3% and 70.7% in Belo Horizonte and Rio de Janeiro, respectively, after unavoidable attrition.     

Apparent validity of alternative survey methods

A comparison of results using three methods of obtaining the same survey data is reported. The assumption is made that in relation to a question for which there is a socially desirable response, e.g., use of seat restraints for children, those reporting nonuse or infrequent use are giving truthful answers. In an experiment in health education, follow-up assignments were made randomly, one third being interviewed face-to-face, one third by telephone, and the rest by a postal questionnaire. There were no significant differences by method of follow-up in the scores reflecting reported changes in behavior following instruction. Both the telephone and mail methods, however, appear to yield a higher proportion of "truthful", i.e., socially undersirable, responses; 30.4% of those contacted by telephone admitted to not using a restraint on the last trip, compared with 26.3% of those questioned by mail and 17.5% in a household interview. It is recommended that careful consideration be given to these alternative methods of obtaining data, particularly in view of their lower cost and possibly greater validity.     

National health surveys by mail or home interview: effects on response

STUDY OBJECTIVE: To study the effect of using a mail questionnaire or home interviews on the size and the selectivity of response to national health surveys. DESIGN: The interview survey and the mail survey were both carried out in the same country (the Netherlands) using the same sample frame, the same study period (1998) and collected partly the same data on demographic, socioeconomic and health characteristics. SETTING: The Netherlands. PARTICIPANTS: Dutch non-institutionalised inhabitants aged 25 years and over. MAIN RESULTS: Response to the mail survey was lower (46.9%, n=3664) than to the interview survey (58.5%, n=6061). The mail survey gave higher response rates for women and lower response rates for persons with lower levels of education. Respondents to the mail survey reported lower rates of smoking but a slightly worse health status and higher figures on the use of health care services. No differences by method of data collection were found for age, marital status, region, household composition, work status and categories of body mass index. CONCLUSION: Although the response of the mail survey was lower than the home interview survey, respondents showed generally small differences, with exception of level of education.     

Mail follow-up of chronic diseases in northeast Thailand

The use of mailed questionnaires to follow patients with chronic diseases was investigated in 650 patients from 16 provinces of northeastern Thailand who sought care at the University Hospital. The overall response rate to the first mail follow-up was 73.2%. If all three mail follow-ups were considered, up to 86.6% of all patients responded. The demographic variables appeared to be similar between the respondents and the non-respondents. Cancer and deterioration of disease status had an adverse effect on the response rate. Utilization of peripheral health centres to supplement mail follow-up in these sub-populations was suggested. Almost 78% of 650 patients were followed and interviewed to assess the validity of mailed response. The percentages of respondents and non-respondents who were followed were almost identical. Of the 504 patients followed, 92% were located. The successful location was over 98% for the respondents and only 54% for non-respondents. The agreement of some parameter such as death and survival time after discharge was very high between mail and intensive follow-up data. Less distinctive events had lower agreement rates. Some precautionary notes were voiced and the possibility of extension of mail follow-up in collection of information and supervision of health care was suggested.     

Mixed-mode survey of female veterinarians yields high response rate

In a recent study of female veterinarians, a subgroup of healthprofessionals growing rapidly in number, the authors employeda mixed-mode survey design in targeting the cohort of womengraduating from all US veterinary colleges during the 11-yearperiod 1970–80 (n = 2,997). The questionnaire elicitedinformation on a variety of health and occupational factorsand required 35 minutes on average to complete. In the firststage, a modified version of Dillman's Total Design Method formailed, self-administered questionnaires was employed, yieldinga response rate of 82.9%. In the second stage, a telephone interviewof all mail non-respondents was attempted, yielding a responserate here of only 30.1%, but increasing the overall responserate among those contacted to 90.2%. Non-respondents differedlittle from mail (early) or telephone (late) respondents withrespect to year of graduation and geographic region of veterinarycollege attendance. Gentle probing of telephone non-respondentssuggested the personal nature of some questions and the amountof time required to answer all questions were the main reasonsthey chose not to participate. It therefore appears that conventionalsurvey techniques may be successfully employed in health studiesof health professionals, particularly if issues of great concernto the target population are addressed.     

A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey

This paper compares respondents to mailed questionnaires with those nonrespondents subsequently interviewed by telephone in a survey of Massachusetts women aged 45-55 years conducted in 1981-1982. This mixed mode approach produced 8,050 responses, giving a response rate of 77%. This rate is similar to rates obtained in many surveys that employed in-person interviews, which are still widely used in health surveys but are increasingly expensive. Telephone respondents differed socioeconomically from mail respondents, suggesting that telephone follow-up of nonrespondents may have reduced nonresponse bias in this survey. Thus, a mixed mode approach may be superior to a mail-only approach with respect to this aspect of data quality. Women responding by mail were more likely to hold professional jobs, to have relatively high household incomes, and to have more years of education. Controlling for these socioeconomic differences did not, however, remove differences in reported health outcomes between mail and telephone respondents. These differences may be explained by less complete recall in the telephone interviews or they may arise from actual differences in health profiles between early (i.e., mail) and late (i.e., telephone) respondents. Although a mixed mode approach may reduce nonresponse bias, more research is required concerning the reasons for response differences between modes and to eliminate any differences caused by problems in data quality.     

The effect of non-response on estimates of health care utilisation: linking health surveys and registers

BACKGROUND: Non-response in health surveys may lead to bias in estimates of health care utilisation. The magnitude, direction and composition of the bias are usually not well known. When data from health surveys are merged with data from registers at the individual level, analyses can reveal non-response bias. Our aim was to estimate the composition, direction and magnitude of non-response bias in the estimation of health care costs in two types of health interview surveys. METHODS: The surveys were (1) a national personal interview survey of 22 484 Danes (2) a telephone interview survey of 5000 Danes living in Funen County. Data were linked with register information on health care utilisation in hospitals and primary care. Health care utilisation was estimated for respondents and non-respondents, and the difference was explained by a decomposition method of bias components. RESULTS: The surveys produced the same pattern of non-response, but with slight differences in non-response bias. Response rates for the interview and telephone surveys were 75 and 69%, respectively. Refusal was the most frequent reason for non-response (22 and 20% of those sampled, respectively), whereas illness, non-contact, and other reasons were less frequent. Respondents used 3-6% less health care than non-respondents at the aggregate level, but the opposite was true for some specific types of health care. Non-response due to illness was the main contributor to non-response bias. CONCLUSIONS: Different types of non-response have different bias effects. However, the magnitude of the bias encourages the continued use of interview health surveys.     

Phone respondents reported less mental health problems whereas mail interviewee gave higher physical health ratings

OBJECTIVE: This study examines the effects of telephone and mail interview methods on the measurement of health-related quality of life. STUDY DESIGN AND SETTING: One thousand six hundred ninety individuals aged 25-66 were interviewed randomly either by telephone or by mail. Health-related quality of life was assessed with the German SF-8. RESULTS: Although respondents in the telephone survey were more willing to participate (77.4% vs. 47.5%) the difference in the accessibility of publicly available address and telephone records meant that overall more people were interviewed by mail than by telephone (53.2% vs. 46.8%). No differences occurred in terms of the sociodemographic makeup. Telephone respondents gave a more positive account of the mental dimension of their health-related quality of life; whereas mail interviews led to a better rating of the respondents' physical well-being. Gender-specific analyses indicate a slight discrepancy in the influence of the method of interviewing on men and women. Further differences were identified concerning the variance in the existence of ceiling and floor effects and the correlation between items. CONCLUSION: Found differences are small but at least to be valued as relevant in certain settings. Therefore, we approve the use and development of factors of amendment.     

Collecting mental health self-evaluations. Effectiveness and cost of three third-party techniques

The effectiveness and cost of three techniques for obtaining client ratings of their own levels of functioning were assessed in a private, outpatient, mental health clinic. Seventy-five clients were asked to rate the adequacy of their own functioning on vocational, family, social, personal, and overall dimensions. Clients were randomly assigned to (a) mail, (b) telephone, or (c) mail followed (if necessary) by telephone administration of the ratings questionnaire. Mail followed if necessary by phone administration was found to be more cost effective than either mail alone or telephone alone.     

Effects of phone versus mail survey methods on the measurement of health-related quality of life and emotional and behavioural problems in adolescents

Background  

Telephone interviews have become established as an alternative to traditional mail surveys for collecting epidemiological data in public health research. However, the use of telephone and mail surveys raises the question of to what extent the results of different data collection methods deviate from one another. We therefore set out to study possible differences in using telephone and mail survey methods to measure health-related quality of life and emotional and behavioural problems in children and adolescents.     

A Comparison of Mail and Telephone Administration of District-Level Questionnaires for the School Health Policies and Programs Study (SHPPS) 2006: Effects on Estimates and Data Quality

BACKGROUND: The School Health Policies and Programs Study (SHPPS) is a national study periodically conducted to assess school health policies and programs at the state, district, school, and classroom levels. For SHPPS 2006, district-level questionnaires were designed for telephone administration, but mixed-mode data collection that also used paper-and-pencil mail questionnaires was required to obtain an acceptable response rate. Because most mode effect research has involved person-level rather than institution-level data, little is known about the effects of mixed-mode data collection on data quality and prevalence estimates obtained through surveys of school personnel. METHODS: SHPPS 2006 used 1-stage stratified cluster sampling to select a nationally representative sample of public school districts. Personnel in about half of the 538 responding districts completed paper questionnaires and returned them via mail. Analyses were performed comparing data quality and prevalence estimates for mail and telephone administration. RESULTS: Prevalence estimates for only 7.0% (39) of 554 questions tested across the 7 questionnaires differed significantly by response mode at the p < .01 level. Regarding data quality, use of the “don't know” response was higher for telephone administration. CONCLUSIONS: The results of this study demonstrate that SHPPS 2006 successfully used a mixed-mode approach, allowing the data to be used without concern about the mixed-mode administration. The results may also be useful to other researchers interested in using surveys to collect data on schools or school districts or other data that is not person level.     

Alternative modes for health surveillance surveys: an experiment with web, mail, and telephone

BACKGROUND: Web and mail surveys as complements to telephone surveys may help resolve concerns about declining participation in telephone surveys for public health surveillance. Little is known, however, about how responses obtained in Web surveys compare with those from mail or telephone surveys. METHODS: The Behavioral Risk Factor Surveillance System 2003 core interview was conducted in 3 survey modes: Web (n = 1143), mail (n = 836), and telephone (n = 2072). All 3 samples were drawn randomly. We compared respondent demographics and responses to 8 key questions on health conditions and risk behaviors (including asthma, diabetes, obesity, and HIV testing) across the 3 survey modes. RESULTS: Demographic characteristics of mail and Web respondents varied considerably from those interviewed by telephone. The unadjusted prevalence of outcomes varied by survey mode. After adjustment for respondent demographic characteristics, there were still differences among survey modes in several of the health conditions and risk behaviors, although for some of these, the pattern was different for the unadjusted and adjusted results. CONCLUSIONS: As health surveys take advantage of new technologies and moved towards mixed-mode designs, researchers need to test for and, if necessary, account for the effect of mode in the estimates they produce.     

Design of the 1986 National Mortality Followback Survey: considerations on collecting data on decedents

The first National Mortality Followback Survey in 18 years was conducted by the National Center for Health Statistics on a national probability sample of adult deaths in the United States in 1986. Data were collected on (a) socioeconomic differentials in mortality, (b) prevention of premature death by inquiring into the association of risk factors and cause of death, (c) health care services provided in the last year of life, and (d) the reliability of certain items reported on the death certificate. In addition to demographic characteristics of the decedent available from the death certificate and the questionnaire, information was secured on cigarette smoking practices, alcohol use, food consumption patterns, use of hospital, nursing home, and hospice care, sources of payment for care, duration of disability, and assistance with activities of daily living. A rich body of data was collected for analysis. In a large pretest, response was received from 87.3 percent of the next of kin of the decedents. The pretest included several methodologic studies to increase the level and quality of response in the main survey. Response rates were compared for data collection by mail, telephone, and personal interview. A test of certified mail and first class mail was conducted. Response to two forms of different lengths was compared. An experiment was also conducted on the effect of inclusion of boxes for a "don't know" response. A public use data tape is available from the National Center for Health Statistics.     

Results of the socio-dental survey of people aged 50 and older living in East York, Ontario

We determined the oral health status and treatment needs of the current and future elderly in East York and tested four enrollment strategies for community-based surveys. We selected our sample from the Municipal Composite Report and obtained the data using a multi-item questionnaire and a clinical examination. The examination protocol was based on World Health Organization (WHO) methodology but was expanded to provide tooth surface specific information. Of those contacted by telephone and repeated mailings, 35% participated. The sample appeared representative of the community as determined by a follow-up mail survey of non-respondents. 76% of those examined were dentate and 71% had visited the dentist within the last two years. Nonetheless, 27% had chewing impairment, 15% had discomfort with their teeth and 15% needed care urgently. Treatment needs were highest for the 187 dentate. They required 504 extractions or restorations and 113 denture services. Periodontal care for calculus and pockets was needed by 86% of the dentate. This report shows an apparent contradiction of high dental treatment needs remaining in the context of reported high care-seeking rates.     

Using the Internet to conduct surveys of health professionals: a valid alternative?

OBJECTIVE: The purpose of this study was to examine whether Internet-based surveys of health professionals can provide a valid alternative to traditional survey methods. METHODS: (i) Systematic review of published Internet-based surveys of health professionals focusing on criteria of external validity, specifically sample representativeness and response bias. (ii) Internet-based survey of GPs, exploring attitudes about using an Internet-based decision support system for the management of familial cancer. RESULTS: The systematic review identified 17 Internet-based surveys of health professionals. Whilst most studies sampled from professional e-directories, some studies drew on unknown denominator populations by placing survey questionnaires on open web sites or electronic discussion groups. Twelve studies reported response rates, which ranged from nine to 94%. Sending follow-up reminders resulted in a substantial increase in response rates. In our own survey of GPs, a total of 268 GPs participated (adjusted response rate = 52.4%) after five e-mail reminders. A further 72 GPs responded to a brief telephone survey of non-respondents. Respondents to the Internet survey were more likely to be male and had significantly greater intentions to use Internet-based decision support than non-respondents. CONCLUSIONS: Internet-based surveys provide an attractive alternative to postal and telephone surveys of health professionals, but they raise important technical and methodological issues which should be carefully considered before widespread implementation. The major obstacle is external validity, and specifically how to obtain a representative sample and adequate response rate. Controlled access to a national list of NHSnet e-mail addresses of health professionals could provide a solution.     

Cohort reconstruction: which infants can be restudied at school age?

Longer-term follow-up of infants with specific health concerns, such as low birthweight, is critical to assessing the effect of medical interventions. This report examines the approach of reconstructing previously studied cohorts in terms of the factors discriminating between respondents and non-respondents. Follow-up was attempted during 1987-1988 for 1875 children born during a 6-month period in 1978 in three geographically defined regions in the United States, for whom 1-year assessments of health and developmental status were obtained at 1 year of age as part of a previous study. For a 25% sample, participation involved a clinic visit for developmental assessments; for the remainder an interview by telephone or home visit. Follow-up was obtained for 72.5% of the cohort. Refusal rates were low (7%); most non-response was due to an inability to locate the families. Predictors of non-response reflected primarily low socio-economic status; completion rates were not influenced by mode of assessment. The role of a tracing agency is discussed. We conclude that cohort reconstruction is feasible with response rates comparable to some prospective studies with ongoing cohort maintenance.