The effect of non-response on estimates of health care utilisation: linking health surveys and registers

BACKGROUND: Non-response in health surveys may lead to bias in estimates of health care utilisation. The magnitude, direction and composition of the bias are usually not well known. When data from health surveys are merged with data from registers at the individual level, analyses can reveal non-response bias. Our aim was to estimate the composition, direction and magnitude of non-response bias in the estimation of health care costs in two types of health interview surveys. METHODS: The surveys were (1) a national personal interview survey of 22 484 Danes (2) a telephone interview survey of 5000 Danes living in Funen County. Data were linked with register information on health care utilisation in hospitals and primary care. Health care utilisation was estimated for respondents and non-respondents, and the difference was explained by a decomposition method of bias components. RESULTS: The surveys produced the same pattern of non-response, but with slight differences in non-response bias. Response rates for the interview and telephone surveys were 75 and 69%, respectively. Refusal was the most frequent reason for non-response (22 and 20% of those sampled, respectively), whereas illness, non-contact, and other reasons were less frequent. Respondents used 3-6% less health care than non-respondents at the aggregate level, but the opposite was true for some specific types of health care. Non-response due to illness was the main contributor to non-response bias. CONCLUSIONS: Different types of non-response have different bias effects. However, the magnitude of the bias encourages the continued use of interview health surveys.     

Estimation of non-response bias in the Medicare FFS HOS

We examined non-response bias in physical component summary scores (PCS) and mental component summary scores (MCS) in the Medicare fee-for-service (FFS) Health Outcomes Survey (HOS) using two alternative methods, response propensity weighting and imputation for non-respondents. The two approaches gave nearly identical estimates of non-response bias. PCS scores were 0.74 points lower and MCS scores 0.51 points lower after adjustment for non-response through imputation and 0.63 and 0.46 lower after adjustment for propensity weighting. These levels are small for component scores suggesting that survey non-response to the FFS HOS does not adversely affect estimates of average health status for this population.     

Comparing oral health care utilization estimates in the United States across three nationally representative surveys

OBJECTIVE: To compare estimates of dental visits among adults using three national surveys. DATA SOURCES/STUDY DESIGN: Cross-sectional data from the National Health Interview Survey (NHIS), National Health and Nutrition Examination Survey (NHANES), and National Health Expenditure surveys (NMCES, NMES, MEPS). STUDY DESIGN: This secondary data analysis assessed whether overall estimates and stratum-specific trends are different across surveys. DATA COLLECTION: Dental visit data are age standardized via the direct method to the 1990 population of the United States. Point estimates, standard errors, and test statistics are generated using SUDAAN. PRINCIPAL FINDINGS: Sociodemographic, stratum-specific trends are generally consistent across surveys; however, overall estimates differ (NHANES III [364-day estimate] versus 1993 NHIS: -17.5 percent difference, Z = 7.27, p value < 0.001; NHANES III [365-day estimate] vs. 1993 NHIS: 5.4 percent difference, Z = -2.50, p value = 0.006; MEPS vs. 1993 NHIS: -29.8 percent difference, Z = 16.71, p value < 0.001). MEPS is the least susceptible to intrusion, telescoping, and social desirability. CONCLUSIONS: Possible explanations for discrepancies include different reference periods, lead-in statements, question format, and social desirability of responses. Choice of survey should depend on the hypothesis. If trends are necessary, choice of survey should not matter however, if health status or expenditure associations are necessary, then surveys that contain these variables should be used, and if accurate overall estimates are necessary, then MEPS should be used. A validation study should be conducted to establish "true" utilization estimates.     

On weighting the rates in non-response weights

A basic estimation strategy in sample surveys is to weight units inversely proportional to the probability of selection and response. Response weights in this method are usually estimated by the inverse of the sample-weighted response rate in an adjustment cell, that is, the ratio of the sum of the sampling weights of respondents in a cell to the sum of the sampling weights for respondents and non-respondents in that cell. We show by simulations that weighting the response rates by the sampling weights to adjust for design variables is either incorrect or unnecessary. It is incorrect, in the sense of yielding biased estimates of population quantities, if the design variables are related to survey non-response; it is unnecessary if the design variables are unrelated to survey non-response. The correct approach is to model non-response as a function of the adjustment cell and design variables, and to estimate the response weight as the inverse of the estimated response probability from this model. This approach can be implemented by creating adjustment cells that include design variables in the cross-classification, if the number of cells created in this way is not too large. Otherwise, response propensity weighting can be applied.     

Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of GPs

BACKGROUND: Non-response is an important potential source of bias in survey research. With evidence of falling response rates from GPs, it is of increasing importance when undertaking postal questionnaire surveys of GPs to seek to maximize response rates and evaluate the potential for non-response bias. OBJECTIVES: Our aim was to investigate the effectiveness of follow-up procedures when undertaking a postal questionnaire study of GPs, the use of publicly available data in assessing non-response bias and the development of regression models predicting responder behaviour. METHOD: A postal questionnaire study was carried out of a random sample of 600 GPs in Wales concerning their training and knowledge in palliative care. RESULTS: A cumulative response rate graph permitted optimal timing of follow-up mailings: a final response rate of 67.6% was achieved. Differences were found between responders and non-responders on several parameters and between sample and population on some parameters: some of these may bias the sample data. Logistic regression analysis indicated medical school of qualification and current membership of the Royal College of General Practitioners to be the only significant predictors of responders. Late responders were significantly more likely to have been qualified for longer. CONCLUSIONS: This study has several implications for future postal questionnaire studies of GPs. The optimal timing of reminders may be judged from plotting the cumulative response rate: it is worth sending at least three reminders. There are few parameters that significantly predict GPs who are unlikely to respond; more of these may be included in the sample, or they may be targeted for special attention. Publicly available data may be used readily in the analysis of non-response bias and generalizability.     

Association between variables used in the field substitution and post-stratification adjustment in the Belgian health interview survey and non-response

Objectives

Field substitution and post-stratification adjustment have been proposed to reduce non-response bias in population surveys. We investigated if variables involved in those techniques in the Belgian health interview survey 2004 are associated with non-response and assessed the impact of field substitution and post-stratification adjustment on the survey results.

Methods

Data were obtained from all selected households (n = 12.204). The association between non-response and the selected variables was explored through multilevel logistic regression models with municipality and statistical sector as random effects.

Results

All investigated variables were significantly related with non-response. Especially households that could not be contacted differed substantially from those who participated. Only post-stratification had a clear impact on the survey results.

Conclusions

Even if variables used in the field substitution procedure of health surveys are strongly associated with non-response, the impact of field substitution on the survey results may be minimal, either because there was no bias of relevance or it was not captured. The usefulness of field substitution to correct for non-response bias in population health surveys seems to be quite limited.     

Health insurance coverage during a 24-month period: a comparison of estimates from two national health surveys

National estimates of the uninsured are available from multiple surveys and differ across surveys. Previous efforts to better understand reasons for differences among these estimates have primarily focused on annual estimates. This study compares national estimates of health insurance coverage over generally comparable 24-month time periods using two integrated Federal health-related surveys, the Medical Expenditure Panel Survey (MEPS) and the National Health Interview Survey (NHIS) for the years 2002–2003 and replicated analyses for 2001–2002. We examine survey participants insurance status in year 1 and year 2 based on the NHIS linked with the MEPS and also for MEPS year 1 and year 2 participants. We also examine characteristics associated with 24-month coverage status. National estimates of the percents continuously insured did not differ significantly between the two data sources. In contrast, the MEPS longitudinal estimate of the percent continuously uninsured was higher than the NHIS-MEPS linked estimate whereas the MEPS longitudinal estimate of the discontinuously insured was lower than that derived from the NHIS-MEPS linked data. Factors that help explain these differences include the non-equivalence of the time periods covered by the data sources, modest differences in the length of time covered by the MEPS and NHIS survey instruments, and length of recall. Regression analyses yielded highly consistent correlates of being continuously uninsured versus continuously insured for both data sources. Regression results for discontinuous versus continuous coverage were also generally similar for both data sources. Gaining a better understanding of the alignment in findings based on alternative data sources that support comparable analyses of health insurance coverage helps policymakers to make the most appropriate use of resultant estimates. The views expressed in this paper are those of the authors and no official endorsement by the Department of Health and Human Services or the Agency for Healthcare Research and Quality or the Centers for Disease Control and Prevention is intended or should be inferred.     

New outcome rates for random sampling telephone surveys

BackgroundSurvey non-response rates are important quality indicators. Refusal rates can induce non-response bias in health survey estimates. However, comparisons across surveys highlight inconsistencies in the use of survey outcome categories and in the calculation of response rates. In this paper we discuss the relevance of these indicators and suggest other survey quality indicators.MethodsOutcome rates from two French random-digit dialing (RDD) telephone surveys are compared : the Nicolle survey on infectious diseases of 4112 individuals conducted in 2006, and the HIV knowledge, attitude, belief and practices (KABP) survey of 5071 individuals in 2004. Based on the same protocol, we describe in details the way the two RDD samples were drawn and how non-response rates were estimated.ResultsNon-response rates were different: 36% in Nicolle survey and 18% in KABP survey. However, the quantity of telephone numbers required to obtain one interview was higher in the KABP survey: 2.8 telephone numbers versus 2.1 in the Nicolle survey. The participation rates, aggregating together refusals, break-off and non-reachable numbers, were equivalent for the two surveys. This result occurred because of a greater proportion of unreached calls in the KABP surveys, which is not integrated into the non-response rates commonly used.ConclusionSurvey non-response rate is insufficient to estimate the quality of a survey. The need for other indicators has been previously stressed in the literature, notably with the adoption and utilization of the American Association for Public Opinion Research (AAPOR) standard definitions of four indicators. But these indicators are quite complex for evaluating non-response bias between surveys. In addition to the classical refusal rate, two other indicators are proposed in this paper: participation rate (number of complete interviews divided by the number of eligible and of unknown eligibility units) and a liking contact rate (number of unreachable units because of a long absence, break-off or non-answer divided by the number of eligible and of unknown eligibility units). The sum of these three indicators is equal to 100% and thus easier to manipulate when comparing surveys.     

Survey non-response in the Netherlands: effects on prevalence estimates and associations

PURPOSE: Differences in respondent characteristics may lead to bias in prevalence estimates and bias in associations. Both forms of non-response bias are investigated in a study on psychosocial factors and cancer risk, which is a sub-study of a large-scale monitoring survey in the Netherlands. METHODS: Respondents of a cross-sectional monitoring project (MORGEN; N = 22,769) were also asked to participate in a prospective study on psychosocial factors and cancer risk (HLEQ; N = 12,097). To investigate diverse aspects of non-response in the HLEQ on prevalence estimates and associations are studied, based on information gathered in the MORGEN-project. RESULTS: A response percentage of 45% was obtained in the MORGEN-project. Response rates were found to be lower among men and younger people. The HLEQ showed a response percentage of 56%, and respondents reported higher socioeconomic status, better subjective health and healthier lifestyle behaviors than non-respondents. However, associations between smoking status and either socioeconomic status or subjective health based on respondents only were not statistically different from those based on the entire MORGEN-population. CONCLUSION: Non-response leads to bias in prevalence estimates of current smoking, current alcohol intake, and low physical activity or poor subjective health. However, non-response did not cause bias in the examined associations.     

A Nonresponse Bias Analysis of the Health Information National Trends Survey (HINTS)

We conducted a nonresponse bias analysis of the Health Information National Trends Survey (HINTS) 4, Cycles 1 and 3, collected in 2011 and 2013, respectively, using three analysis methods: comparison of response rates for subgroups, comparison of estimates with weighting adjustments and external benchmarks, and level-of-effort analysis. Areas with higher concentrations of low socioeconomic status, higher concentrations of young households, and higher concentrations of minority and Hispanic populations had lower response rates. Estimates of health information seeking behavior were higher in HINTS compared to the National Health Interview Survey (NHIS). The HINTS estimate of doctors always explaining things in a way that the patient understands was not significantly different from the same estimate from the Medical Expenditure Panel Survey (MEPS); however, the HINTS estimate of health professionals always spending enough time with the patient was significantly lower than the same estimate from MEPS. A level-of-effort analysis found that those who respond later in the survey field period were less likely to have looked for information about health in the past 12 months, but found only small differences between early and late respondents for the majority of estimates examined. There is some evidence that estimates from HINTS could be biased toward finding higher levels of health information seeking.     

Selection bias due to non-response in a health survey among patients with rheumatoid arthritis

BACKGROUND: Non-response may lead to bias in health(care) outcomes. METHODS: We compared respondents (n = 334) to a questionnaire survey among patients with rheumatoid arthritis with non-respondents (n = 68) and determined predictors of (non-)response. The bias in prevalence estimates of health characteristics and health care use was quantified. RESULTS: Self-reported pain and health care utilization were the most important predictors of (non-)response with respondents experiencing pain more often and more often using specific health care services. Bias concerned especially an underestimation of 'never having pain' (60%) and 'no contact with health care services' (51%). CONCLUSION: More insight into the phenomenon of non-response is important to assess disease burden and health care burden more precisely.     

Moscow Health Survey 2004 – social surveying under difficult circumstances

OBJECTIVES: The aim of this paper is to present the Moscow Health Survey 2004, which was designed to examine health inequalities in Moscow. In particular we want to discuss social survey problems, such as non-response, in Moscow and Russia. METHODS: Interviews, covering social and economic circumstances, health and social trust, of a stratified random sample of the greater Moscow population, aged 18+. Reasons for nonresponse were noted down with great care. Odds ratios (ORs) for self-rated health by gender and by six social dimensions were estimated separately for districts with low and high response rates. Bias due to non-response is discussed. RESULTS AND CONCLUSIONS: About one in two (53.1 %) of approached individuals could not be interviewed, resulting in 1190 completed interviews. Non-response in most Russian surveys, but perhaps particularly in Moscow, is large, partly due to fear of strangers and distrust of authorities. ORs for poor health vary significantly by gender, occupational class, education and economic hardship. We find no significant differences in these ORs when comparing districts with low and high response rates. Non-response may be a problem when estimating prevalence rates or population means, but much less so when estimating odds ratios in multivariate analyses.     

The impact of survey attrition on health insurance coverage estimates in a National Longitudinal Health Care Survey

Timely, accurate and reliable estimates of the population’s health insurance status are essential inputs to policymakers to inform assessments of the population’s access to medical care and analyses of associated health care expenditures. Alternative criteria that have been used to produce annual estimates of the uninsured include the following specifications: those uninsured for a full-year, those ever uninsured during a year, and those uninsured at a specific point in time. The Medical Expenditure Panel Survey (MEPS), one of the core health care surveys in the United States, supports all three types of estimates. In this paper, a summary is provided of the survey operations, informational materials, the interviewer training and experience of the field force, and the refusal conversion techniques employed in the MEPS to maintain respondent cooperation for five rounds of interviewing, to help minimize sample attrition. The impact of nonresponse attributable to survey attrition is also assessed with respect to the national health insurance coverage estimates derived from the MEPS. The study includes an examination of the quality of the nonresponse adjustments employed to adjust for potential nonresponse bias attributable to survey attrition. The overlapping panel design of the MEPS survey is particularly well suited to inform these studies. The presentation concludes with a discussion of strategies under consideration that may yield additional improvements in the accuracy for these critical policy relevant survey estimates.     

Nonresponse error in injury-risk surveys

BACKGROUND: Nonresponse is a potentially serious source of error in epidemiologic surveys concerned with injury control and risk. This study presents the findings of a records-matching approach to investigating the degree to which survey nonresponse may bias indicators of violence-related and unintentional injuries in a random-digit-dialed (RDD) telephone survey. METHODS: Data from a statewide RDD survey of 4155 individuals aged 16 years and older conducted in Illinois in 2003 were merged with ZIP code-level data from the 2000 Census. Using hierarchical linear models, ZIP code-level indicators were used to predict survey response propensity at the individual level. Additional models used the same ZIP code measures to predict a set of injury-risk indicators. RESULTS: Several ZIP code measures were found to be predictive of both response propensity and the likelihood of reporting partner violence. For example, people residing in high-income areas were less likely to participate in the survey and less likely to report forced sex by partner, processes that suggest an over-estimation of this form of violence. In contrast, estimates of partner isolation may be under-estimated, as those residing in geographic areas with smaller-sized housing were less likely to participate in the survey but more likely to report partner isolation. No ZIP code-level correlates of survey response propensity, however, were found also to be associated with driving-under-the-influence (DUI) indicators. CONCLUSIONS: There is evidence of a linkage between survey response propensity and one variety of injury prevention measure (partner violence) but not another (DUI). The approach described in this paper provides an effective and inexpensive tool for evaluating nonresponse error in surveys of injury prevention and other health-related conditions.     

The impact of survey design modifications on health insurance coverage estimates in a National Longitudinal Health Care Survey

National health insurance coverage estimates for the overall population and specific population subgroups are critical to policymakers and others concerned with access to medical care and the cost and sources of payment for that care. The Medical Expenditure Panel Survey (MEPS) is one of the core health care surveys in the United States that serves as a primary source for these essential national health insurance coverage estimates. The survey is designed to provide annual national estimates of the health care use, medical expenditures, sources of payment and insurance coverage for the U.S. civilian non-institutionalized population. In 2007, the survey experienced two dominant survey design modifications: (1) a new sample design attributable to the sample redesign of the National Health Interview Survey, and (2) an upgrade to the CAPI platform for the survey instrument, moving from a DOS to a Windows based environment. This study examines the impact of these survey design modifications on the national estimates of insurance coverage. The overlapping panel design of the MEPS survey and its longitudinal features are particularly well suited to assess the impact of survey redesign modifications on estimates. Since two independent nationally representative samples are pooled to produce calendar year estimates, one has the capacity to compare estimates based on the “original survey design” in contrast to those derived from the “survey redesign.” This paper examines the correlates of nonresponse incorporated in the estimation techniques and adjustment methods employed in the survey, and the measures utilized for post-stratification overall and by panel. Particular attention is given to assessing the level of convergence in coverage estimates based on the alternative designs as well as the alignment of model based analyses that discern which factors are associated with health insurance classifications. The paper concludes with a discussion of strategies under consideration that may yield additional improvements in the accuracy for these critical policy relevant survey estimates.     

Predicting risk among non-respondents in prospective studies

Potential non-response bias was investigated in a follow-up study of 2,011 chronically disabled patients. 82.5% and 73.3% of the study subjects responded to self-administered mail questionnaires respectively at 6-month and 1-year follow-up. Information on employment status, the outcome of interest, of approximately 90% of the non-respondents was obtained from indirect sources. Employment rate was lower among the non-respondents than the respondents. Non-response was associated with age, social class, previous employment record, and the type of disability; but none of these characteristics were associated with the outcome. Out of the five known independent risk factors for unemployment, only one (incompletion of rehabilitation course) was associated with non-response. The employment rate among the respondents was also assessed according to the delay in response, that is the number of reminders sent to achieve response. The outcome among- the late respondents was similar to that among the nonrespondents. These data suggest that (a) risk estimates may be biased even when the response rate is greater than 80%, (b) the prevalence of risk factors among non-respondents may not indicate the presence or the degree of non-response bias, but (c) reliable estimates can be obtained from extrapolations of the rates among the respondents according to the delay in response.     

Estimating Non-Response Bias in Family Studies: Application to Mental Health and Lifestyle

Non-response to mailed surveys reduces the effective sample size and may introduce bias. Non-response has been studied by (1) comparison to available data in population based registers, (2) directly contacting non-respondents by telephone or single-item reply cards, and (3) longitudinal repetition of the survey. The goal of this paper was to propose an additional method to study non-response bias: when the variable of interest has a familial component, data from respondents can be used as proxy for the data from their non-responding family members. This approach was used with data on smoking, alcohol consumption, physical activity, coffee- and tea-use, education, body mass index, religion, burnout, life events, personality and mental health in large number of siblings and DZ twins registered with the Netherlands Twin Register. In addition, for smoking behavior, we also used the second strategy by sending a reply card. Results show that scores of members from less cooperative families or incomplete twin pairs tended to be more unfavorable than the scores from highly cooperative families or complete twin pairs. For example, family members from less cooperative families cycled less often and scored higher on anxious depression and neuroticism. For smoking, both the results of the reply card and the results of the additional method suggested a higher percentage smokers among the non-respondents but this was only significant with reply card method. In general, differences between highly/less cooperative families and complete/incomplete DZ twins were small. Results suggest that, even for studies with moderate response rates, data collected on health, personality and lifestyle are relatively unbiased.     

A comparison of mixed-effects quantile stratification propensity adjustment strategies for longitudinal treatment effectiveness analyses of continuous outcomes

The propensity adjustment is used to reduce bias in treatment effectiveness estimates from observational data. We show here that a mixed-effects implementation of the propensity adjustment can reduce bias in longitudinal studies of non-equivalent comparison groups. The strategy examined here involves two stages. Initially, a mixed-effects ordinal logistic regression model of propensity for treatment intensity includes variables that differentiate subjects who receive various doses of time-varying treatments. Second, a mixed-effects linear regression model compares the effectiveness of those ordinal doses on a continuous outcome over time. Here, a simulation study compares bias reduction that is achieved by implementing this propensity adjustment through various forms of stratification. The simulations demonstrate that bias decreased monotonically as the number of quantiles used for stratification increased from two to five. This was particularly pronounced with stronger effects of the confounding variables. The quartile and quintile strategies typically removed in excess of 80-90 per cent of the bias detected in unadjusted models; whereas a median-split approach removed from 20 to 45 per cent of bias. The approach is illustrated in an evaluation of the effectiveness of somatic treatments for major depression in a longitudinal, observational study of affective disorders.     

Generalizability of valuations on health states collected with the EuroQolc-questionnaire

Problem: Non-response and non-usable response were found in population surveys on valuation of health states. If non-response is selective regarding valuations, then generalization of the resulting values to the whole survey population is not permitted. This could limit the use of empirical utility values in resource allocation in health care. Methods: Response behaviour of a sample of 1400 from the Dutch general population to the mailed EuroQol^c-questionnaire was analyzed by four methods. I. Phoning resolute non-respondents; II. comparison of zip code characteristics of respondents and non-respondents (because individual data on background characteristics were not available for the non-respondents); III. analysis of response over time (wave-analysis); IV: comparison of background variables of successful (less than two valuations missing) and unsuccessful respondents, combined with analysis of the effect of these background variables on valuations. Results: No indications for selective non-response were found, although the phenomenon appeared hard to investigate. The successful response came from a slightly younger and better educated subsample. However, a general influence of age and educational level on valuations could not be shown. This finding is consistent with the literature. Conclusion: Although the existence of selective non-response cannot be excluded, its relevance can be considered to be small. This finding is encouraging for the use of empirical utility values in allocative decisions.     

Medical consumption of respondents and non-respondents to a mailed health survey

Non-response bias can distort the results of health surveys.The occurrence of selective non-response can be assessed whendata are available for both respondents and non-respondents.The objective of this study was to compare the medical consumptionof respondents and non-respondents to a mailed health survey.A mailed health survey was conducted among approximately 13,500adults and among parents of approximately 1,500 children aged5–15 years. The net response rate was 70.4%. A panel dataset that could be matched with the health survey data was availablefor all eligible persons. This data set comprises administrativeinformation on hospitalizations, annual health care expendituresand demographic variables. The results of this study show thatresponse was associated with age, sex, degree of urbanizationand type of insurance. After correcting for differences in demographicvariables, respondents and non-respondents differ in the utilizationof several types of care. Relatively more users than non-usersresponded. Response was not associated with the utilizationof care related to severe conditions such as in-patient hospitalcare. The conclusion from this study is that when a mailed healthsurvey is used to measure medical consumption, the non-responsebias will result in a small overestimation of utilization.