Methods for Constructing and Assessing Propensity Scores

Objectives

To model the steps involved in preparing for and carrying out propensity score analyses by providing step-by-step guidance and Stata code applied to an empirical dataset.

Study Design

Guidance, Stata code, and empirical examples are given to illustrate (1) the process of choosing variables to include in the propensity score; (2) balance of propensity score across treatment and comparison groups; (3) balance of covariates across treatment and comparison groups within blocks of the propensity score; (4) choice of matching and weighting strategies; (5) balance of covariates after matching or weighting the sample; and (6) interpretation of treatment effect estimates.

Empirical Application

We use data from the Palliative Care for Cancer Patients (PC4C) study, a multisite observational study of the effect of inpatient palliative care on patient health outcomes and health services use, to illustrate the development and use of a propensity score.

Conclusions

Propensity scores are one useful tool for accounting for observed differences between treated and comparison groups. Careful testing of propensity scores is required before using them to estimate treatment effects.     

The Impact of Electronic Health Record Implementation and Use on Performance of the Surgical Care Improvement Project Measures

Objective

To examine the impact of electronic health record (EHR) deployment on Surgical Care Improvement Project (SCIP) measures in a tertiary-care teaching hospital.

Data Sources

SCIP Core Measure dataset from the CMS Hospital Inpatient Quality Reporting Program (March 2010 to February 2012).

Study Design

One-group pre- and post-EHR logistic regression and difference-in-differences analyses.

Principal Findings

Statistically significant short-term declines in scores were observed for the composite, postoperative removal of urinary catheter and post–cardiac surgery glucose control measures. A statistically insignificant improvement in scores for these measures was noted 3 months after EHR deployment.

Conclusion

The transition to an EHR appears to be associated with a short-term decline in quality. Implementation strategies should be developed to preempt or minimize this initial decline.     

Separated at birth: statisticians, social scientists, and causality in health services research

Objective

Health services research is a field of study that brings together experts from a wide variety of academic disciplines. It also is a field that places a high priority on empirical analysis. Many of the questions posed by health services researchers involve the effects of treatments, patient and provider characteristics, and policy interventions on outcomes of interest. These are causal questions. Yet many health services researchers have been trained in disciplines that are reluctant to use the language of causality, and the approaches to causal questions are discipline specific, often with little overlap. How did this situation arise? This paper traces the roots of the division and some recent attempts to remedy the situation.

Data Sources and Settings

Existing literature.

Study Design

Review of the literature.     

Accounting for haplotype uncertainty in matched association studies: a comparison of simple and flexible techniques

Population-based case-control studies measuring associations between haplotypes of single nucleotide polymorphisms (SNPs) are increasingly popular, in part because haplotypes of a few "tagging" SNPs may serve as surrogates for variation in relatively large sections of the genome. Due to current technological limitations, haplotypes in cases and controls must be inferred from unphased genotypic data. Using individual-specific inferred haplotypes as covariates in standard epidemiologic analyses (e.g., conditional logistic regression) is an attractive analysis strategy, as it allows adjustment for nongenetic covariates, provides omnibus and haplotype-specific tests of association, and can estimate haplotype and haplotype x environment interaction effects. In principle, some adjustment for the uncertainty in inferred haplotypes should be made. Via simulation, we compare the performance (bias and mean squared error of haplotype and haplotype x environment interaction effect estimates) of several analytic strategies using inferred haplotypes in the context of matched case-control data. These strategies include using only the most likely haplotype assignment, the expectation substitution approach described by Stram et al. ([2003b] Hum. Hered. 55:179-190) and others, and an improper version of multiple imputation. For relatively uncomplicated haplotype structures and moderate haplotype relative risks (/=5). An application to progesterone-receptor haplotypes and endometrial cancer further illustrates that the performance of all these methods depends on how well the observed haplotypes "tag" the unobserved causal variant.     

The design and evaluation of a preschool enrichment package for black South African children in day care

In this paper the rationale behind, and effectiveness of, an early enrichment package for young South African children in day care are examined. Results indicate that there are significant gains in children's competence with objects, and in their psychometric performance. However, some attention needs to be paid to the fact that the enrichment package increased children's levels of solitary play. The study illustrates the degree to which enrichment design can be informed and refined as a result of evaluative research.     

Sources of validity evidence for an internal medicine student evaluation system: an evaluative study of assessment methods

Background  Medical students' final clinical grades in internal medicine are based on the results of multiple assessments that reflect not only the students' knowledge, but also their skills and attitudes.
Objective  To examine the sources of validity evidence for internal medicine final assessment results comprising scores from 3 evaluations and 2 examinations.
Methods  The final assessment scores of 8 cohorts of Year 4 medical students in a 6-year undergraduate programme were analysed. The final assessment scores consisted of scores in ward evaluations (WEs), preceptor evaluations (PREs), outpatient clinic evaluations (OPCs), general knowledge and problem-solving multiple-choice questions (MCQs), and objective structured clinical examinations (OSCEs). Sources of validity evidence examined were content, response process, internal structure, relationship to other variables, and consequences.
Results  The median generalisability coefficient of the OSCEs was 0.62. The internal consistency reliability of the MCQs was 0.84. Scores for OSCEs correlated well with WE, PRE and MCQ scores with observed (disattenuated) correlation of 0.36 (0.77), 0.33 (0.71) and 0.48 (0.69), respectively. Scores for WEs and PREs correlated better with OSCE than MCQ scores. Sources of validity evidence including content, response process, internal structure and relationship to other variables were shown for most components.
Conclusion  There is sufficient validity evidence to support the utilisation of various types of assessment scores for final clinical grades at the end of an internal medicine rotation. Validity evidence should be examined for any final student evaluation system in order to establish the meaningfulness of the student assessment scores.     

Effects of an electronic health record-based mobility assessment and automated referral for inpatient physical therapy on patient outcomes: A quasi-experimental study

Objective

To assess the effectiveness of a hospital physical therapy (PT) referral triggered by scores on a mobility assessment embedded in the electronic health record (EHR) and completed by nursing staff on hospital admission.

Data Sources

EHR and billing data from 12 acute care hospitals in a western Pennsylvania health system (January 2017–February 2018) and 11 acute care hospitals in a northeastern Ohio health system (August 2019–July 2021).

Study Design

We utilized a regression discontinuity design to compare patients admitted to PA hospitals with stroke who reached the mobility score threshold for an EHR-PT referral (treatment) to those who did not (control). Outcomes were hospital length of stay (LOS) and 30-day readmission or mortality. Control variables included demographics, insurance, income, and comorbidities. Hospital systems with EHR-PT referrals were also compared to those without (OH hospitals as alternative control). Subgroup analyses based on age were also conducted.

Data Extraction

We identified adult patients with a primary or secondary diagnosis of stroke and mobility assessments completed by nursing (n = 4859 in PA hospitals, n = 1749 in OH hospitals) who completed their inpatient stay.

Principal Findings

In the PA hospitals, patients with EHR-PT referrals had an 11.4 percentage-point decrease in their 30-day readmission or mortality rates (95% CI −0.57, −0.01) relative to the control. This effect was not observed in the OH hospitals for 30-day readmission (β = 0.01; 95% CI −0.25, 0.26). Adults over 60 years old with EHR-PT referrals in PA had a 26.2 percentage-point (95% CI −0.88, −0.19) decreased risk of readmission or mortality compared to those without. Unclear relationships exist between EHR-PT referrals and hospital LOS in PA.

Conclusions

Health systems should consider methodologies to facilitate early acute care hospital PT referrals informed by mobility assessments.     

Indigenous health program evaluation design and methods in Australia: a systematic review of the evidence

Objective: Indigenous Australians experience a disproportionately higher burden of disease compared to non‐Indigenous Australians. High‐quality evaluation of Indigenous health programs is required to inform health and health services improvement. We aimed to quantify methodological and other characteristics of Australian Indigenous health program evaluations published in the peer‐reviewed literature. Methods: Systematic review of peer‐reviewed literature (November 2009–2014) on Indigenous health program evaluation. Results: We identified 118 papers describing evaluations of 109 interventions; 72.0% were university/research institution‐led. 82.2% of evaluations included a quantitative component; 49.2% utilised quantitative data only and 33.1% used both quantitative and qualitative data. The most common design was a before/after comparison (30.5%, n=36/118). 7.6% of studies (n=9/118) used an experimental design: six individual‐level and three cluster‐randomised controlled trials. 56.8% (67/118) reported on service delivery/process outcomes (versus health or health risk factor outcomes) only. Conclusions: Given the number of Indigenous health programs that are implemented, few evaluations overall are published in the peer‐reviewed literature and, of these, few use optimal methodologies such as mixed methods and experimental design. Implications for public health: Multiple strategies are required to increase high‐quality, accessible evaluation in Indigenous health, including supporting stronger research‐policy‐practice partnerships and capacity building for evaluation by health services and government.     

A comparison of synthetic control approaches for the evaluation of policy interventions using observational data: Evaluating the impact of redesigning urgent and emergency care in Northumberland

Objective

To compare the original synthetic control (OSC) method with alternative approaches (Generalized [GSC], Micro [MSC], and Bayesian [BSC] synthetic control methods) and re-evaluate the impact of a significant restructuring of urgent and emergency care in Northeast England, which included the opening of the UK's first purpose-built specialist emergency care hospital.

Data Sources

Simulations and data from Secondary Uses Service data, a single comprehensive repository for patient-level health care data in England.

Study Design

Hospital use of individuals exposed and unexposed to the restructuring is compared. We estimate the impact using OSC, MSC, BSC, and GSC applied at the general practice level. We contrast the estimation methods' performance in a Monte Carlo simulation study.

Data Collection/Extraction Methods

Hospital activity data from Secondary Uses Service for patients aged over 18 years registered at a general practice in England from April 2011 to March 2019.

Principal Findings

None of the methods dominated all simulation scenarios. GSC was generally preferred. In contrast to an earlier evaluation that used OSC, GSC reported a smaller impact of the opening of the hospital on Accident and Emergency (A&E) department (also known as emergency department or casualty) visits and no evidence for any impact on the proportion of A&E patients seen within 4 h.

Conclusions

The simulation study highlights cases where the considered methods may lead to biased estimates in health policy evaluations. GSC was found to be the most reliable method of those considered. Considering more disaggregated data over a longer time span and applying GSC indicates that the specialist emergency care hospitals in Northumbria had less impact on A&E visits and waiting times than suggested by the original evaluation which applied OSC to more aggregated data.     

某县人民医院门诊综合楼扩建项目竣工验收 卫生学评价

摘要:目的　评价医院建筑设计、设施设备及环境条件是否符合相关法律标准要求,以减少或消除医院内公共环境危害因素对诊疗患者和医护人员的不良影响。方法　采用现场卫生学调查和卫生监测等方法,对施工设计、空气质量、集中空调、污水处理和消毒供应等内容进行审查和评价。结果　医院设计布局合理,功能分区隔开设置;集中空调、供排水设计、防护能力能够满足实际需要和标准要求;医院消毒与感染、饮用水、医院污水和集中空调送风、风管内表面监测结果均符合相应的卫生标准要求;但是空调新风机组未安装净化处理装置,3处医生办公室中甲醛浓度超标,不合格率为30%。结论　该医院扩建项目设计和功能布局合理,供排水、医院消毒等均能满足要求,但在室内环境甲醛污染和新风净化处理方面有待进一步完善。     

The Boomers Are Coming: A Total Cost of Care Model of the Impact of Population Aging on Health Care Costs in the United States by Major Practice Category

Objective. To project the impact of population aging on total U.S. health care per capita costs from 2000 to 2050 and for the range of clinical areas defined by Major Practice Categories (MPCs).
Data Sources. Secondary data: HealthPartners health plan administrative data; U.S. Census Bureau population projections 2000–2050; and MEPS 2001 health care annual per capita costs.
Study Design. We calculate MPC-specific age and gender per capita cost rates using cross-sectional data for 2002–2003 and project U.S. changes by MPC due to aging from 2000 to 2050.
Data Collection Methods. HealthPartners data were grouped using purchased software. We developed and validated a method to include pharmacy costs for the uncovered.
Principal Findings. While total U.S. per capita costs due to aging from 2000 to 2050 are projected to increase 18 percent (0.3 percent annually), the impact by MPC ranges from a 55 percent increase in kidney disorders to a 12 percent decrease in pregnancy and infertility care. Over 80 percent of the increase in total per capita cost will result from just seven of the 22 total MPCs.
Conclusions. Understanding the differential impact of aging on costs at clinically specific levels is important for resource planning, to effectively address future medical needs of the aging U.S. population.     

The Impact of Regression to the Mean on Economic Evaluation in Quasi‐Experimental Pre–Post Studies: The Example of Total Knee Replacement Using Data from the Osteoarthritis Initiative

Many treatments are evaluated using quasi‐experimental pre–post studies susceptible to regression to the mean (RTM). Ignoring RTM could bias the economic evaluation. We investigated this issue using the contemporary example of total knee replacement (TKR), a common treatment for end‐stage osteoarthritis of the knee. Data (n = 4796) were obtained from the Osteoarthritis Initiative database, a longitudinal observational study of osteoarthritis. TKR patients (n = 184) were matched to non‐TKR patients, using propensity score matching on the predicted hazard of TKR and exact matching on osteoarthritis severity and health‐related quality of life (HrQoL). The economic evaluation using the matched control group was compared to the standard method of using the pre‐surgery score as the control. Matched controls were identified for 56% of the primary TKRs. The matched control HrQoL trajectory showed evidence of RTM accounting for a third of the estimated QALY gains from surgery using the pre‐surgery HrQoL as the control. Incorporating RTM into the economic evaluation significantly reduced the estimated cost effectiveness of TKR and increased the uncertainty. A generalized ICER bias correction factor was derived to account for RTM in cost‐effectiveness analysis. RTM should be considered in economic evaluations based on quasi‐experimental pre–post studies. Copyright © 2017 John Wiley & Sons, Ltd.     

Putting context centre stage: evidence from a systems evaluation of an area based empowerment initiative in England

It is now widely accepted that context matters in evaluations of the health inequalities impact of community-based social initiatives. Systems thinking offers a lens for uncovering the dynamic relationship between such initiatives and their social contexts. However, there are very few examples that show how a systems approach can be applied in practice and what kinds of evidence are produced when this happens. In this paper, we use data from ethnographic fieldwork embedded within a multi-site mixed method evaluation to demonstrate how a systems approach can be applied in practice to evaluate the early stages of an area-based empowerment initiative – Big Local (funded by the Big Lottery Fund and delivered by Local Trust). Taking place in 150 different local areas in England and underpinned by an ethos of resident-led collective action, Big Local offers an illustration of the applicability of a systems approach to better understand the change processes that emerge as social initiatives embed and co-evolve within a series of local contexts. Findings reveal which parts of the social system are likely to be changed, by what mechanisms, and with what implications. They also raise some salient considerations for knowledge generation and methods development in public health evaluation, particularly for the evaluation of social initiatives where change does not necessarily happen in linear or predictable ways. We suggest future evaluations of such initiatives require the use of more flexible designs, encompassing qualitative approaches capable of capturing the complexity of relational systems processes, alongside more traditional quantitative methods.