The experience of using random digit dialling methods in a population-based chronic pain study

OBJECTIVES: To describe response rates and call characteristics using random digit dialling (RDD) methods in a population-based prevalence study of chronic pain; to compare respondent characteristics according to telephone number listing status. METHODS: A RDD telephone study of chronic pain was conducted using computer-assisted telephone interview (CATI) methods in the Northern Sydney Area in 1998. RESULTS: Unlisted number respondents (18.5%) were younger, had a higher proportion of males, and were less likely to live in households with three or more eligible adults. There was no difference in chronic pain status between the two groups. The overall response rate was 73.4% (unlisted number group 66.3%; listed number group 75.0%). Answering machine messages boosted response rates (79.7%). 10.9% of unlisted numbers used in the study resulted in completed interviews, compared with 31.3% of the listed numbers used. CONCLUSIONS: The relatively high response rate obtained, differences in characteristics associated with listing status and reduction in sampling bias provided by using RDD methods have to be balanced against the differential response rates between listed and unlisted number groups and higher costs (including opportunity costs). IMPLICTIONS: Published data on the experience of using RDD methods can assist public health researchers in deciding whether to use these methods in telephone surveys.     

Random digit dialling and Electronic White Pages samples compared: demographic profiles and health estimates

OBJECTIVE: To compare the methodologies of and health estimates derived from two telephone household survey methods. In particular, to establish if White Pages telephone listings provide a relatively unbiased sampling frame for population health surveys. METHOD: In South Australia in 1998, a health survey questionnaire was administered by telephone to two randomly selected population samples. The first method used EWP (Electronic White Pages, n = 6,012), which contains all listed residential telephone numbers as the sampling frame. The results were compared to a RDD (random digit dialling, n = 3,080) sample where all listed and unlisted telephone numbers were included in the sampling frame. Demographic variables and health estimates were compared between the surveys, and then compared to a 'gold standard' door-to-door household survey conducted concurrently. RESULTS: The response rate for EWP (83.8%) exceeded that of RDD (65.4%). More than four times as many calls were required per completed interview in RDD. Demographic profiles and health estimates were substantially similar. CONCLUSIONS: EWP requires fewer telephone calls and enables approach letters establishing the bona fides of the survey to be sent to each selected address before calling, increasing the response rate. RDD is a more inclusive sampling frame but also includes non-connected and business numbers, and offers no significant advantages in providing health estimates. IMPLICATIONS: There are substantial methodological and cost advantages in using EWP over RDD as the sampling, frame for population health surveys, without introducing significant bias into health estimates.     

An Experimental Study Using Opt-in Internet Panel Surveys for Behavioral Health Surveillance

Objective

To present the design and preliminary results of a pilot study to investigate the use of opt-in Internet panel surveys for behavioral health surveillance.

Introduction

Today, surveyors in both the private and public sectors are facing considerable challenges with random digit dialed (RDD) landline telephone samples. The population coverage rates for landline telephone surveys are being eroded by wireless-only households, portable telephone numbers, telecommunication barriers (e.g., call forwarding, call blocking and pager connections), technological barriers (call-blocking, busy circuits) and increased refusal rates and privacy concerns. Addressing these issues increasingly drives up the costs associated with dual-frame telephone surveys designed to be representative of the target population as well as hinders their ability to be fully representative of the adult population of each state and territory in the United States.In an effort to continue to meet these challenges head on and assist state and territorial public health professionals in the continued collection of data that are representative of their respective populations, novel approaches to behavioral health surveillance need continued examination. Both private and public sector researchers are evaluating the use of Internet opt-in panels to augment dual-frame RDD survey methods. Compared to dual-frame RDD, opt-in Internet panels offer lower costs, quick data collection and dissemination, and the ability to gather additional data on panelists over time. However, as with dual-frame RDD, this mode has similar challenges with coverage error and non-response. Nevertheless, survey methodologists are moving forward and exploring ways to reduce or eliminate biases between the sample and the target population.

Methods

A collaborative pilot project was designed to assess the feasibility and accuracy of opt-in Internet panel surveys for behavioral health surveillance. This pilot project is a collaboration between the CDC, four state departments of health, opt-in Internet panel providers and the leads of several large surveys and systems such as the Patient-Reported Outcome Measures Information System (PROMIS) and the Cooperative Congressional Election Study (CCES). Pilot projects were conducted in four states (GA, IL, NY, and TX) and four Metropolitan Statistical Areas (Atlanta, Chicago, New York City, and Houston). Data were collected using three different opt-in Internet panels and sampling methods that differ with respect to recruitment strategy, sample selection and sample matching to the adult population of each geography. A question bank consisting of 80 questions was developed to benchmark with other existing surveys used to assess various public health surveillance measures (e.g., the Behavioral Risk Factor Surveillance System, the PROMIS, National Survey on Drug Use and Health, and the CCES).

Results

We present comparative analyses that assess the advantages and disadvantages of different opt-in Internet panels sampling methodologies across a range of parameters including cost, geography, timeliness, usability, and ease of use for technology transfer to states and local communities. Recommendations for future efforts in behavioral health surveillance are given based on these results.     

Address-based versus random-digit-dial surveys: comparison of key health and risk indicators

Use of random-digit dialing (RDD) for conducting health surveys is increasingly problematic because of declining participation rates and eroding frame coverage. Alternative survey modes and sampling frames may improve response rates and increase the validity of survey estimates. In a 2005 pilot study conducted in six states as part of the Behavioral Risk Factor Surveillance System, the authors administered a mail survey to selected household members sampled from addresses in a US Postal Service database. The authors compared estimates based on data from the completed mail surveys (n = 3,010) with those from the Behavioral Risk Factor Surveillance System telephone surveys (n = 18,780). The mail survey data appeared reasonably complete, and estimates based on data from the two survey modes were largely equivalent. Differences found, such as differences in the estimated prevalences of binge drinking (mail = 20.3%, telephone = 13.1%) or behaviors linked to human immunodeficiency virus transmission (mail = 7.1%, telephone = 4.2%), were consistent with previous research showing that, for questions about sensitive behaviors, self-administered surveys generally produce higher estimates than interviewer-administered surveys. The mail survey also provided access to cell-phone-only households and households without telephones, which cannot be reached by means of standard RDD surveys.     

The future of random-digit-dial surveys for injury prevention and violence research

A central issue facing injury prevention research today is how to collect self-reported data on injury and violence from a geographically dispersed public, quickly, cost effectively, and with a reasonable degree of confidence in the quality of the results. Questions about eroding frame coverage, declining participation rates, and increasing potential for bias have raised doubts about the long-term viability of random-digit-dial (RDD) telephone surveys for injury prevention research. So where does the future lie? The four articles in this volume, as well as other research, point down two paths: (1) continued reliance on RDD, or (2) adoption of alternative survey designs. Continued use of RDD methodology will require additional research in the areas of response rate improvement, techniques for enhancing post-survey adjustments, and cost-effective approaches to nonresponse bias analysis. Moving away from a strict reliance on RDD methodology, injury prevention research could adopt mixed-mode approaches (such as combining telephone, mail, and web-based surveys) or make use of address-based sampling frames as a method for reaching sample members currently missed by most RDD approaches. Either way, the future of collecting self-reports of injury and injury prevention data will be more complex and require considerable resources.     

Encouraging physicians to respond to surveys through the use of fax technology

High response rates in surveys of physicians are difficult to achieve. One possible strategy to improve physicians' survey participation is to offer the option of receiving and returning the survey by fax. This study describes the success of the option of fax communication in a survey of general practitioners, family physicians, and pediatricians in Arkansas with regard to pediatric asthma. Eligible physicians were given the choice of receiving the survey by telephone, mail, or fax. In this observational study, physicians' preferences, response rates, and biases for surveys administered by fax were compared with mail and telephone surveys. The overall survey response rate was 59%. For the 96 physicians completing an eligibility screener survey, the largest percentage requested to be surveyed by fax (47%) rather than by telephone (28%) or mail (25%). Faxing may be one strategy to add to the arsenal of tools to increase response rates in surveying physicians.     

Growing Cell-Phone Population and Noncoverage Bias in Traditional Random Digit Dial Telephone Health Surveys

Objective. Examine the effect of including cell‐phone numbers in a traditional landline random digit dial (RDD) telephone survey. Data Sources. The 2007 California Health Interview Survey (CHIS). Data Collection Methods. CHIS 2007 is an RDD telephone survey supplementing a landline sample in California with a sample of cell‐only (CO) adults. Study Design. We examined the degree of bias due to exclusion of CO populations and compared a series of demographic and health‐related characteristics by telephone usage. Principal Findings. When adjusted for noncoverage in the landline sample through weighting, the potential noncoverage bias due to excluding CO adults in landline telephone surveys is diminished. Both CO adults and adults who have both landline and cell phones but mostly use cell phones appear different from other telephone usage groups. Controlling for demographic differences did not attenuate the significant distinctiveness of cell‐mostly adults. Conclusions. While careful weighting can mitigate noncoverage bias in landline telephone surveys, the rapid growth of cell‐phone population and their distinctive characteristics suggest it is important to include a cell‐phone sample. Moreover, the threat of noncoverage bias in telephone health survey estimates could mislead policy makers with possibly serious consequences for their ability to address important health policy issues.     

The effect of non-response on estimates of health care utilisation: linking health surveys and registers

BACKGROUND: Non-response in health surveys may lead to bias in estimates of health care utilisation. The magnitude, direction and composition of the bias are usually not well known. When data from health surveys are merged with data from registers at the individual level, analyses can reveal non-response bias. Our aim was to estimate the composition, direction and magnitude of non-response bias in the estimation of health care costs in two types of health interview surveys. METHODS: The surveys were (1) a national personal interview survey of 22 484 Danes (2) a telephone interview survey of 5000 Danes living in Funen County. Data were linked with register information on health care utilisation in hospitals and primary care. Health care utilisation was estimated for respondents and non-respondents, and the difference was explained by a decomposition method of bias components. RESULTS: The surveys produced the same pattern of non-response, but with slight differences in non-response bias. Response rates for the interview and telephone surveys were 75 and 69%, respectively. Refusal was the most frequent reason for non-response (22 and 20% of those sampled, respectively), whereas illness, non-contact, and other reasons were less frequent. Respondents used 3-6% less health care than non-respondents at the aggregate level, but the opposite was true for some specific types of health care. Non-response due to illness was the main contributor to non-response bias. CONCLUSIONS: Different types of non-response have different bias effects. However, the magnitude of the bias encourages the continued use of interview health surveys.     

Increasing response rates in community health surveys administered by telephone

This paper reports the effects of two methods used to increase response rates in a community health survey administered by telephone. Converting refusals resulted in an increase of 3.7% in the final response rate, while the investigation and identification of indeterminate telephone numbers increased the response rate by an additional 6.1%. Together, these methods resulted in an increase of 9.8%, from an initial response rate of 70.1% to a final lower-bound response rate of 79.9%. The use of these methods helped to reduce non-response bias at a minimal cost.     

Alternative strategies for selecting population controls: comparison of random digit dialing and targeted telephone calls

PURPOSE: This project was carried out to identify a valid framework for selecting controls to be used in a population-based case-control study of breast cancer, and to compare participation rates and characteristics between women contacted using a standard random digit dialing (RDD) strategy and those who were sent a letter of presentation prior to telephone contact (targeted telephone calls, TTC). METHODS: Twelve hundred women, ages 20-74, were sampled from the Department of Motor Vehicles (DMV) and Health Care Financing Administration (HCFA) records. Women for whom telephone numbers were obtained (N = 771) were randomly assigned to RDD or TTC. The respondents participated in a brief telephone interview. Odd ratios (OR) and their 95% confidence intervals (CI) were used to estimate differences in characteristics of the respondents between the two contact strategies. RESULTS: Telephone numbers were obtained for 79% of women aged > or = 55 years and for only 38% of women aged < 55 years. Interviews were obtained for 48% of women for whom we obtained telephone numbers, and for 77% of women for whom eligibility was confirmed via telephone contact. Participation of target women appeared to be higher for the TTC than the RDD group (42% vs. 35%, p = 0.054). Among respondents who were > or = 55 years old, those in the TTC group were 80% more likely (OR = 1.8, 95% CI: 0.9-3.4) to report a serious medical condition than women in the RDD group, 60% less likely (OR = 0.4, 95% CI: 0.2-1.0) to report having used oral contraceptives, and 80% less likely (OR = 0.2, 95% CI: 0.1-0.5) to report having had breast surgery. CONCLUSIONS: Characteristics of respondents differed according to method of contact. These differences, along with the sampling frame used, should be considered when interpreting findings of case-control studies.     

Nonresponse error in injury-risk surveys

BACKGROUND: Nonresponse is a potentially serious source of error in epidemiologic surveys concerned with injury control and risk. This study presents the findings of a records-matching approach to investigating the degree to which survey nonresponse may bias indicators of violence-related and unintentional injuries in a random-digit-dialed (RDD) telephone survey. METHODS: Data from a statewide RDD survey of 4155 individuals aged 16 years and older conducted in Illinois in 2003 were merged with ZIP code-level data from the 2000 Census. Using hierarchical linear models, ZIP code-level indicators were used to predict survey response propensity at the individual level. Additional models used the same ZIP code measures to predict a set of injury-risk indicators. RESULTS: Several ZIP code measures were found to be predictive of both response propensity and the likelihood of reporting partner violence. For example, people residing in high-income areas were less likely to participate in the survey and less likely to report forced sex by partner, processes that suggest an over-estimation of this form of violence. In contrast, estimates of partner isolation may be under-estimated, as those residing in geographic areas with smaller-sized housing were less likely to participate in the survey but more likely to report partner isolation. No ZIP code-level correlates of survey response propensity, however, were found also to be associated with driving-under-the-influence (DUI) indicators. CONCLUSIONS: There is evidence of a linkage between survey response propensity and one variety of injury prevention measure (partner violence) but not another (DUI). The approach described in this paper provides an effective and inexpensive tool for evaluating nonresponse error in surveys of injury prevention and other health-related conditions.     

Complementing random-digit-dial telephone surveys with other approaches to collecting sensitive data

Surveys of sensitive topics, such as the Injury Control and Risk Surveys (ICARIS) or the Behavioral Risk Factors Surveillance System (BRFSS), are often conducted by telephone using random-digit-dial (RDD) sampling methods. Although this method of data collection is relatively quick and inexpensive, it suffers from growing coverage problems and falling response rates. In this paper, several alternative methods of data collection are reviewed, including audio computer-assisted interviews as part of personal visit surveys, mail surveys, web surveys, and interactive voice response surveys. Their strengths and weaknesses are presented regarding coverage, nonresponse, and measurement issues, and compared with RDD telephone surveys. The feasibility of several mixed mode designs is discussed; none of them stands out as clearly the right choice for surveys on sensitive issues, which implies increased need for methodologic research.     

Effects of enhanced calling efforts on response rates,estimates of health behavior,and costs in a telephone health survey using random-digit dialing.

Public health researchers frequently rely on random-digit dialing (RDD) telephone surveys in monitoring trends in health behavior and evaluating health promotion interventions. RDD response rates have declined during the past decade, and cost-effective methods to increase response rates are needed. The authors evaluated two levels of enhanced calling efforts in an RDD survey of cancer-related health behavior in the State of Washington. The first level of enhanced calling effort was 1 month after 11 original calling attempts to a household, when the authors attempted up to 11 recalls. The second level was 6 months after the first answered call, when the authors recalled those persons who could not be interviewed. Enhanced calling efforts increased the overall survey response rate by 11 percent. Nine percentage points of the increase were attributable to call backs. There were demographic differences among the participants reached at different levels of calling effort, but no consistent associations of level of calling effort with health behavior related to alcohol use, smoking, diet, or health screening. Marginal costs for interviews completed with enhanced calling efforts were about 50 percent higher than costs for interviews reached in the first 11 calls. The authors concluded that enhanced calling efforts may be justified, because they increase confidence in the generalizability of survey results. However, the authors found very little change in survey results by including interviews from persons who were difficult to reach and to interview.     

On weighting the rates in non-response weights

A basic estimation strategy in sample surveys is to weight units inversely proportional to the probability of selection and response. Response weights in this method are usually estimated by the inverse of the sample-weighted response rate in an adjustment cell, that is, the ratio of the sum of the sampling weights of respondents in a cell to the sum of the sampling weights for respondents and non-respondents in that cell. We show by simulations that weighting the response rates by the sampling weights to adjust for design variables is either incorrect or unnecessary. It is incorrect, in the sense of yielding biased estimates of population quantities, if the design variables are related to survey non-response; it is unnecessary if the design variables are unrelated to survey non-response. The correct approach is to model non-response as a function of the adjustment cell and design variables, and to estimate the response weight as the inverse of the estimated response probability from this model. This approach can be implemented by creating adjustment cells that include design variables in the cross-classification, if the number of cells created in this way is not too large. Otherwise, response propensity weighting can be applied.     

Comparisons of the costs and quality of patient data collection by mail versus telephone versus in-person interviews

The costs and quality of three modes of patient data collectionwere compared in a survey on the quality of health care services.Nine hundred and forty-eight non-institutionalized participantsfrom four patient categories (asthma, rheumatic diseases, disabledand dependent elderly) were divided into three different groupsfor the allocation of either a self-administered mail survey,a computer-assisted telephone interview or an in-person interview.The three modes were compared in terms of the total survey costs,the survey response rates and the data quality. The resultsshow that the mail mode was the most cost-effective and yieldedconsistently high response rates. The telephone interviews yieldedfar more complete data than the other interview modes. Althoughin-person interviews have traditionally been considered to yieldhigher response rates, less non-response bias and better dataquality, the results of this study show different figures infavour of mail surveys.     

Propagation of Nonresponse Weights for Censoring in Multi-Phase Screening in Complex Sample Designs

We explore an analogy between survival analysis and nonresponse analysis for complex sample designs that include multi-phase screening procedures to determine the eligibility of sampled units. Units excluded due to ineligibility are analogous to mortality; nonresponse that occurs at multiple phases of the screening procedure is analogous to censoring. Based on this analogy, we discuss methods for estimating the response rate, the nonresponse weights, the eligible population size and the eligibility rate. In particular, we propose a method that estimates nonresponse weights at each phase of screening, and propagates these weights forward as the analysis weights for units that screen eligible at each phase. The analysis weights are used in nonresponse analyses at later phases of screening and for the analyses on the final sample of eligible units. This method is analogous to using a life-table estimator to estimate the probability of eligibility; the sum of the analysis weights for the final sample estimates the total number of eligible units in the population.     

Peer Reviewed: Integrating a Multimode Design Into a National Random-Digit–Dialed Telephone Survey

The Behavioral Risk Factor Surveillance System (BRFSS) was originally conducted by using a landline telephone survey mode of data collection. To meet challenges of random-digit–dial (RDD) surveys and to ensure data quality and validity, BRFSS is integrating multiple modes of data collection to enhance validity. The survey of adults who use only cellular telephones is now conducted in parallel with ongoing, monthly landline telephone BRFSS data collection, and a mail follow-up survey is being implemented to increase response rates and to assess nonresponse bias. A pilot study in which respondents'' physical measurements are taken is being conducted to assess the feasibility of collecting these data for a subsample of adults in 2 states. Physical measures would allow for the adjustment of key self-reported risk factor and health condition estimates and improve the accuracy and usefulness of BRFSS data. This article provides an overview of these new modes of data collection.     

Improving public health surveillance using a dual-frame survey of landline and cell phone numbers

To meet challenges arising from increasing rates of noncoverage in US landline-based telephone samples due to cell-phone-only households, the Behavioral Risk Factor Surveillance System (BRFSS) expanded a traditional landline-based random digit dialing survey to a dual-frame survey of landline and cell phone numbers. In 2008, a survey of adults with cell phones only was conducted in parallel with an ongoing landline-based health survey in 18 states. The authors used the optimal approach to allocate samples into landline and cell-phone-only strata and used a new approach to weighting state-level landline and cell phone samples. They developed logistic models for each of 16 health indicators to examine whether exclusion of adults with cell phones only affected estimates after adjustment for demographic characteristics. The extents of the potential biases in landline telephone surveys that exclude cell phones were estimated. Biases resulting from exclusion of adults with cell phones only from the landline-based survey were found for 9 out of the 16 health indicators. Because landline noncoverage rates for adults with cell phones only continue to increase, these biases are likely to increase. Use of a dual-frame survey of landline and cell phone numbers assisted the BRFSS efforts in obtaining valid, reliable, and representative data.     

Does telephone follow-up predict patient satisfaction and readmission?

Nursing telephone calls after hospital discharge are commonly adopted as a tool to improve patient satisfaction and continuity of care. Previous research, however, has been inconclusive on the impact of telephone follow-up. The purpose of this study was to comparatively examine patients who received telephone follow-up for response differences on a mail satisfaction survey and 30-day readmission rates for a large health system in southeast Texas. Telephone follow-up, patient satisfaction, and administrative billing data from 2008 to 2009 were retrospectively examined across 10 nursing units that routinely performed calls after patient discharge. Patients eligible to receive a nursing call (N = 10,559) were categorized based on responses to nursing questions or if no contact was made. Logistic regression was used to evaluate whether call data significantly predicted survey response and 30-day readmission rates. Nonparametric analysis was used to evaluate whether survey ratings varied between groups. Completion of telephone follow-up was a significant (P < 0.01) predictor of patient response to the mail survey, with 62% more patients returning surveys after contact. Completion of a nursing call with a patient who reported a physician appointment was a significant predictor (P = 0.04) of lower 30-day readmissions. Readmission rates were 10.8% for patients who did not receive telephone follow-up compared to 9.5% for patients who received a call and who had a scheduled physician appointment. Mean nursing and overall satisfaction scores varied minimally between groups and telephone follow-up was not a significant predictor of patient satisfaction. Telephone follow-up shows significant predictive value for mail survey response and 30-day readmission rates but does not correlate with patient satisfaction scores in the hospital setting.     

Factors impacting questionnaire response in a Dutch retrospective cohort study

PURPOSE: To study questionnaire length, type of consent, approach to recruitment, and subject characteristics on participation in epidemiologic studies. METHODS: As part of a health survey among Dutch subjects treated for ear, nose, and throat disorders in childhood, we conducted a pilot study of 200 individuals who were randomly assigned to one of four categories, defined by length of questionnaire (long vs. short) and type of consent form (basic vs. multi-option). In addition, among 8402 subjects eligible to be in the main study (average age 41 years in 1997), we examined the effect of approach to recruitment and subject characteristics on participation rates. RESULTS: The pilot study showed a non-significant 10% increase in participation rate using the shorter questionnaire, but no differences by type of consent form. In the full survey, the participation rate was 49% after the first mailing. Response increased by 15% after a written reminder and by 10% after a telephone survey. The total participation rate was 74%. Attained age, sex, exposure status, age at exposure, and response to an earlier survey were determinants of participation rates. Among male non-participants, outright refusal was less frequent than non-response. The refusal rate, unlike the non-response rate, was positively associated with older age at time of survey. CONCLUSIONS: Health survey participation is influenced by questionnaire length, frequency of contact, and subject characteristics.