Liver transplant center risk tolerance

Recent changes in Center for Medicare & Medicaid Services (CMS) condition for participation, using benchmark volume/outcomes requirements for certification, have been implemented. Consequently, the ability of a transplant center to assess its risk tolerance is important in successful management. An analysis of SRTR data was performed to determine donor/recipient risk factors for graft loss or patient death in the first year. Each transplant performed was then assigned a prospective relative risk (RR) of failure. Using a Monte-Carlo simulation, transplants were selected at random that met the centers' acceptable risk tolerance. Transplant center volume was fixed and its risk tolerance was adjusted to determine the impact on outcomes. The model was run 1000 times on centers with varying volume. The modeling demonstrates that centers with smaller annual volumes must use a more risk taking strategy than larger volume centers to avoid being flagged for CMS volume requirements. The modeling also demonstrates optimal risk taking strategies for centers based upon volume to minimize the probability of being flagged for not meeting volume or outcomes benchmarks. Small volume centers must perform higher risk transplants to meet current CMS requirements and are at risk for adverse action secondary to chance alone.     

Rates of False Flagging Due to Statistical Artifact in CMS Evaluations of Transplant Programs: Results of a Stochastic Simulation

The recent CMS conditions of participation are based on risk‐adjusted models produced by the Scientific Registry for Transplant Recipients (SRTR). The accuracy of these models in identifying poor‐performing centers is unknown. In this stochastic simulation study, 1‐year mortality outcomes were simulated in virtual transplant centers, and used to flag centers according to the methods used by CMS, evaluating nine overlapping 2.5‐year periods of simulated data. In a simulation where all centers had the same underlying risk, 10.2% were falsely flagged at least once during the 4.5 years of simulated evaluations. The probability of false‐positive flagging was lowest in low‐volume centers (2.5%) and highest in high‐volume centers (16.2%). In another simulation where 5% of centers were assigned twofold risk (“poor‐performing centers”), only 32% of poor‐performing centers were correctly flagged. In a final simulation where each center was assigned a unique mortality risk, 94% of flagged centers had greater‐than‐median risk, but only 32% of flagged centers were among the 5% with highest risk. Even after disregarding known covariate limitations to the risk adjustment models, statistical noise alone leads to spurious flagging of many adequately‐performing transplant centers, yet the methods used by CMS fail to flag most centers with true elevated risk.     

Regulatory change for organ procurement organizations: certifying the success of transplantation in America

In June 2006, the Centers for Medicare & Medicaid Services released the final rule for the conditions of participation for organ procurement organizations in the United States. The new guidelines change the way OPOs are measured, shifting from a population-based set of performance measures to a system that is based on eligible donors and conversion rates. In addition to the change in measurement philosophy, CMS has included statements, regarding organizational structure, staffing, and research as standard measures for OPOs. The change from solely donation and transplantation measures is significant for the community, and will result in changes in the way OPOs structure their organizations and governing and clinical boards.     

Scientific Registry of Transplant Recipients: Collecting,analyzing, and reporting data on transplantation in the United States

Founded in 1987, the Scientific Registry of Transplant Recipients (SRTR) operates under a contract from the US government administered by the Health Resources and Services Administration (HRSA). SRTR maintains a database of comprehensive information on all solid organ transplantation in the US. The registry supports the ongoing evaluation of the clinical status of solid organ transplantation, including kidney, heart, liver, lung, intestine, pancreas, and multi-organ transplants. Data in the registry are from multiple sources, but most are collected by the Organ Procurement and Transplantation Network (OPTN) from hospitals, organ procurement organizations, and immunology laboratories. The data include information on current and past organ donors, transplant candidates, transplant recipients, transplant outcomes, and outcomes of living donors. SRTR uses these data to create reports and analyses for HRSA, OPTN committees that make organ allocation policy, and the Centers for Medicare & Medicaid Services to carry out quality assurance surveillance activities; SRTR also creates standard analysis files for scientific investigators. In addition, SRTR and OPTN produce an Annual Data Report and provide information upon request for the general public. Thus, SRTR supports the transplant community with information services and statistical analyses to improve patient access to and outcomes of organ transplant.     

Potential Implications of Recent and Proposed Changes in the Regulatory Oversight of Solid Organ Transplantation in the United States

Every 6 months, the Scientific Registry of Transplant Recipients (SRTR) publishes evaluations of every solid organ transplant program in the United States, including evaluations of 1‐year patient and graft survival. The Centers for Medicare & Medicaid Services (CMS) and the Organ Procurement and Transplantation Network (OPTN) Membership and Professional Standards Committee (MPSC) use SRTR's 1‐year evaluations for regulatory review of transplant programs. Concern has been growing that the regulatory scrutiny of transplant programs with lower‐than‐expected outcomes is harmful, causing programs to undertake fewer high‐risk transplants and leading to unnecessary organ discards. As a result, CMS raised its threshold for a “Condition‐Level Deficiency” designation of observed relative to expected 1‐year graft or patient survival from 1.50 to 1.85. Exceeding this threshold in the current SRTR outcomes report and in one of the four previous reports leads to scrutiny that may result in loss of Medicare funding. For its part, OPTN is reviewing a proposal from the MPSC to also change its performance criteria thresholds for program review, to review programs with “substantive clinical differences.” We review the details and implications of these changes in transplant program oversight.     

Organ Procurement Organizations and the Electronic Health Record

The adoption of electronic health records (EHRs) has adversely affected the ability of organ procurement organizations (OPOs) to perform their federally mandated function of honoring the donation decisions of families and donors who have signed the registry. The difficulties gaining access to potential donor medical record has meant that assessment, evaluation, and management of brain dead organ donors has become much more difficult. Delays can occur that can lead to potential recipients not receiving life‐saving organs. For over 40 years, OPO personnel have had ready access to paper medical records. But the widespread adoption of EHRs has greatly limited the ability of OPO coordinators to readily gain access to patient medical records and to manage brain dead donors. Proposed solutions include the following: (1) hospitals could provide limited access to OPO personnel so that they could see only the potential donor's medical record; (2) OPOs could join with other transplant organizations to inform regulators of the problem; and (3) hospital organizations could be approached to work with Center for Medicare and Medicaid Services (CMS) to revise the Hospital Conditions of Participation to require OPOs be given access to donor medical records.     

SRTR Center-Specific Reporting Tools: Posttransplant Outcomes

Measuring and monitoring performance—be it waiting list and posttransplant outcomes by a transplant center, or organ donation success by an organ procurement organization and its partnering hospitals—is an important component of ensuring good care for people with end-stage organ failure. Many parties have an interest in examining these outcomes, from patients and their families to payers such as insurance companies or the Centers for Medicare and Medicaid Services; from primary caregivers providing patient counseling to government agencies charged with protecting patients.
The Scientific Registry of Transplant Recipients produces regular, public reports on the performance of transplant centers and organ procurement organizations. This article explains the statistical tools used to prepare these reports, with a focus on graft survival and patient survival rates of transplant centers—especially the methods used to fairly and usefully compare outcomes of centers that serve different populations. The article concludes with a practical application of these statistics—their use in screening transplant center performance to identify centers that may need remedial action by the OPTN/UNOS Membership and Professional Standards Committee.     

The Centers for Medicare and Medicaid Services’ proposed metrics for recertification of organ procurement organizations: Evaluation by the Scientific Registry of Transplant Recipients

On December 23, 2019, the US Centers for Medicare and Medicaid Services proposed 2 new standards that organ procurement organizations (OPOs) must meet for recertification. An OPO’s organ donation rate (deceased donors/potential donors) and organ transplant rate (organs transplanted/potential donors) must not fall significantly below the 75th percentile for rates among all OPOs. We examined how OPOs would have fared under the proposed performance standards in 2016‐2017. Data on donors and transplants were from the Organ Procurement and Transplantation Network; donor potential was estimated from Detailed Multiple Cause of Death data collected by the Centers for Disease Control and Prevention. In 2017, 31 (53%) OPOs failed to meet the proposed donation rate standard, 36 (62%) failed to meet the proposed organ transplant rate standard, and 37 (64%) failed at least 1 standard. We found that adjusting for age, race, and Hispanic ethnicity altered the evaluation: 8 OPOs changed their pass/fail status for the donation rate and 5 for the proposed organ transplant rate standard. We conclude that the proposed new standards may result in over half of OPOs facing decertification, and risk adjustment suggests that underlying characteristics of deaths vary regionally such that decertification decisions may be affected.     

Time for reform in transplant program–specific reporting: AST/ASTS transplant metrics taskforce

In accordance with the National Organ Transplant Act and Department of Health and Human Services’ Final Rule, the Scientific Registry of Transplant Recipients (SRTR) publicly releases biannual program‐specific reports that include analyses of transplant centers’ risk‐adjusted waitlist mortality, organ acceptance ratios, transplant rates, and graft and patient survival . Since the inception of these center metrics, 1‐year posttransplant graft and patient survival have improved, and center variation has decreased, casting uncertainty on their clinical relevance. The SRTR has recently modified center evaluations by ranking centers into 5 tiers rather than 3 tiers in an attempt to discriminate between programs performing within a tight range, further exacerbating this uncertainty. The American Society of Transplantation/American Society of Transplant Surgeons convened an expert taskforce to examine both the utility and unintended consequences of transplant center metrics. Estimates of center variation in outcomes in adjacent tiers are imprecise and fleeting, but can result in consequential changes in clinician and center behavior. The taskforce has concerns that current metrics, based principally on 1‐year graft and patient survival, provide minimal if any benefit in informing patient choice and access to transplantation, with the untoward effect of decreased utilization of organs and restriction of research and innovation.     

The Prognostic Value of Kidney Transplant Center Report Cards

SRTR report cards provide the basis for quality measurement of US transplant centers. There is limited data evaluating the prognostic value of report cards, informing whether they are predictive of prospective patient outcomes. Using national SRTR data, we simulated report cards and calculated standardized mortality ratios (SMR) for kidney transplant centers over five distinct eras. We ranked centers based on SMR and evaluated outcomes for patients transplanted the year following reports. Recipients transplanted at the 50th, 100th and 200th ranked centers had 18% (AHR = 1.18, 1.13?1.22), 38% (AHR = 1.38, 1.28?1.49) and 91% (AHR = 1.91, 1.64?2.21) increased hazard for 1‐year mortality relative to recipients at the top‐ranked center. Risks were attenuated but remained significant for long‐term outcomes. Patients transplanted at centers meeting low‐performance criteria in the prior period had 40% (AHR = 1.40, 1.22?1.68) elevated hazard for 1‐year mortality in the prospective period. Centers' SMR from the report card was highly predictive (c‐statistics > 0.77) for prospective center SMRs and there was significant correlation between centers' SMR from the report card period and the year following (ρ = 0.57, p < 0.001). Although results do not mitigate potential biases of report cards for measuring quality, they do indicate strong prognostic value for future outcomes. Findings also highlight that outcomes are associated with center ranking across a continuum rather than solely at performance margins.

     

What patients and members of their support networks ask about transplant program data

Transplant patients often seek specific data and statistics to inform medical decision making; however, for many relevant measures, patient‐friendly information is not available. Development of patient‐centered resources should be informed by patient needs. This study used qualitative document research methods to review 678 detailed Scientific Registry of Transplant Recipients (SRTR) entries and summary counts of 55 362 United Network for Organ Sharing (UNOS) entries to provide a better understanding of what was asked and what requests were most common. Incoming call and email logs maintained by SRTR and UNOS were reviewed for 2010‐2015. Patients sought a wide range of information about outcomes, waiting times, program volumes, and willingness to perform transplants in candidates with specific diseases or demographics. Patients and members of their support networks requested explanation of complex information, such as actual‐vs‐expected outcomes, and of general transplant processes, such as registering on the waiting list or becoming a living donor. They sought transplant program data from SRTR and UNOS, but encountered gaps in the information they wanted and occasionally struggled to interpret some data. These findings were used to identify potential gaps in providing program‐specific data and to enhance the SRTR website ( www.srtr.org ) with more patient‐friendly information.     

Organ Donation and Utilization in the United States: 1998–2007

Organ transplantation remains the only life-saving therapy for many patients with organ failure. Despite the work of the Organ Donation and Transplant Collaboratives, and the marked increases in deceased donors early in the effort, deceased donors only rose by 67 from 2006 and the number of living donors declined during the same time period. There continues to be increases in the use of organs from donors after cardiac death (DCD) and expanded criteria donors (ECD). This year has seen a major change in the way organs are offered with increased patient safety measures in those organ offers made by OPOs using DonorNet^©. Unfortunately, the goals of 75% conversion rates, 3.75 organs transplanted per donor, 10% of all donors from DCD sources and 20% growth of transplant center volume have yet to be reached across all donation service areas (DSAs) and transplant centers; however, there are DSAs that have not only met, but exceeded, these goals. Changes in organ preservation techniques took place this year, partly due to expanding organ acceptance criteria and increasing numbers of ECDs and DCDs. Finally, the national transplant environment has changed in response to increased regulatory oversight and new requirements for donation and transplant provider organizations.     

Importance of incorporating standardized,verifiable, objective metrics of organ procurement organization performance into discussions about organ allocation

Identifying and supporting specific organ procurement organizations (OPOs) with the greatest opportunity to increase donation rates could significantly increase the number of organs available for transplant. Accomplishing this is complicated by current Scientific Registry of Transplant Recipients/Centers for Medicare & Medicaid Services metrics of donation rates and OPO performance that rely on eligible deaths. These data are self‐reported and unverifiable and have been shown to underestimate potential organ donors. We examine the limitations of current OPO performance/donation metrics to inform discussions related to strategies to increase donation. We propose changing to a simple, verifiable, and uniformly applied donation metric. This would allow the transplant community to (1) better understand inherent differences in donor availability based on geography and (2) identify underperforming areas that would benefit from systems improvement agreements to increase donation rates.     

Association of Candidate Removals From the Kidney Transplant Waiting List and Center Performance Oversight

Approximately 59 000 kidney transplant candidates have been removed from the waiting list since 2000 for reasons other than transplantation, death, or transfers. Prior studies indicate that low‐performance (LP) center evaluations by the Scientific Registry of Transplant Recipients (SRTR) are associated with reductions in transplant volume. There is limited information to determine whether performance oversight impacts waitlist management. We used national SRTR data to evaluate outcomes of 315 796 candidates on the kidney transplant waiting list (2007–2014). Compared to centers without LP, rates of waitlist removal (WLR) were higher at centers with LP evaluations (44.6/1000 follow‐up years, 95% confidence interval [CI] 44.0, 45.1 versus 68.0/1000 follow‐up years, 95% CI 66.6, 69.4), respectively, which was consistent after risk adjustment (adjusted hazard ratio [AHR] = 1.59, 95% CI 1.55, 1.63). Candidate mortality following waitlist removal was lower at LP centers (AHR = 0.90, 95% CI 0.87, 0.94). Analyses limited to LP centers indicated a significant increase in WLR (+28.6 removals/1000 follow‐up years, p < 0.001), a decrease in transplant rates (?11.9/1000 follow‐up years, p < 0.001) and a decrease in mortality after removal (?67.5 deaths/1000 follow‐up years, p < 0.001) following LP evaluation. There is a significant association between LP evaluations and transplant center processes of care for waitlisted candidates. Further understanding is needed to determine the impact of performance oversight on transplant center quality of care and patient outcomes.

     

Seeking new answers to old questions about public reporting of transplant program performance in the United States

The Scientific Registry of Transplant Recipients (SRTR) is mandated by the National Organ Transplant Act, the Final Rule, and the SRTR contract with the Health Resources and Services Administration to report program‐specific information on the performance of transplant programs. Following a consensus conference in 2012, SRTR developed a new version of the public website to improve public reporting of often complex metrics, including changing from a 3‐tier to a 5‐tier summary metric for first‐year posttransplant survival. After its release in December 2016, the new presentation was moved to a “beta” website to allow collection of additional feedback. SRTR made further improvements and released a new beta website in May 2018. In response to feedback, SRTR added 5‐tier summaries for standardized waitlist mortality and deceased donor transplant rate ratios, along with an indicator of which metric most affects survival after listing. Presentation of results was made more understandable with input from patients and families from surveys and focus groups. Room for improvement remains, including continuing to make the data more useful to patients, deciding what additional data elements should be collected to improve risk adjustment, and developing new metrics that better reflect outcomes most relevant to patients.     

Kidney offer acceptance at programs undergoing a Systems Improvement Agreement

In the United States, the Centers for Medicare and Medicaid Services (CMS) use Systems Improvement Agreements (SIAs) to require transplant programs repeatedly flagged for poor‐performance to improve performance or lose CMS funding for transplants. We identified 14 kidney transplant (KT) programs with SIAs and 28 KT programs without SIAs matched on waitlist volume and characterized kidney acceptance using SRTR data from 12/2006‐3/2015. We used difference‐in‐differences linear regression models to identify changes in acceptance associated with an SIA independent of program variation and trends prior to the SIA. SIA programs accepted 26.9% and 22.1% of offers pre‐ and post‐SIA, while non‐SIA programs accepted 33.9% and 44.4% of offers in matched time periods. After adjustment for donor characteristics, time‐varying waitlist volume, and secular trends, SIAs were associated with a 5.9 percentage‐point (22%) decrease in kidney acceptance (95% CI: ?10.9 to ?0.8, P = .03). The decrease in acceptance post‐SIA was more pronounced for KDPI 0‐40 kidneys (12.3 percentage‐point decrease, P = .007); reductions in acceptance of higher KDPI kidneys occurred pre‐SIA. Programs undergoing SIAs substantially reduced acceptance of kidney offers for waitlisted candidates. Attempts to improve posttransplant outcomes might have the unintended consequence of reducing access to transplantation as programs adopt more restrictive organ selection practices.     

SRTR Program-Specific Reports on Outcomes: A Guide for the New Reader

Differences in outcomes indeed exist among transplant programs and organ procurement organizations (OPO). A growing set of tools are available from the Scientific Registry of Transplant Recipients (SRTR) to measure and assess these outcomes in the different phases of the transplant process. These tools are not intended to compare two individual programs, rather to help identify programs whose practices may need further scrutiny, to be either avoided, corrected or emulated.
To understand which differences in outcomes might be due to underlying differences in populations served and which might be due to differences in treatment, it is important to compare outcomes to 'risk-adjusted' expected values. Further, it is important to recognize and assess the role that random chance may play in these outcomes by considering the p-value or confidence interval of each estimate. We present the reader with a basic explanation of these tools and their interpretation in the context of reading the SRTR Program-Specific Reports.
We describe the intended audience of these reports, including patients, monitoring and process improvement bodies, payers and others such as the media. Use of these statistics in a way that reflects a basic understanding of these concepts and their limitations is beneficial for all audiences.     

Effect of Comorbidity Adjustment on CMS Criteria for Kidney Transplant Center Performance

The Centers for Medicare & Medicaid Services (CMS) uses kidney transplant outcomes, unadjusted for standard comorbidity, to identify centers with sufficiently higher than expected rates of graft failure or patient death (underperforming centers) that they may be denied Medicare participation. To examine whether comorbidity adjustment would affect this determination, we identified centers that would have failed to meet 1-year graft survival criteria, 1992–2005, with and without adjustment using the Elixhauser Comorbidity Index. Adjustment was performed for each U.S. center for 24 consecutive (overlapping) 30-month intervals, including 102 176 adult deceased-donor and living-donor kidney transplant patients with Medicare as primary payer 6 months pretransplant. For each interval, we determined percent positive agreement (PPA) (number of centers underperforming both before and after adjustment, divided by number underperforming either before or after adjustment). Overall PPA was 80.8%, with no evidence of a trend over time. Among deceased-donor recipients, 10 of 31 comorbid conditions were predictors of graft failure in at least half of the intervals, as were six conditions among living-donor recipients. Lack of comorbidity adjustment may disadvantage centers willing to accept higher risk patients. Risk of jeopardizing Medicare funding may give centers incentive to deny transplantation to higher risk patients.     

The Association of Center Performance Evaluations and Kidney Transplant Volume in the United States

Report cards evaluating transplant center performance have received significant attention in recent years corresponding with the Centers for Medicare and Medicaid Services issue of the 2007 Conditions of Participation. Our primary aim was to evaluate the association of report card evaluations with transplant center volume. We utilized data from the Scientific Registry of Transplant Recipients (SRTR) along with six consecutive program‐specific reports from January 2007 to July 2009 for adult kidney transplant centers. Among 203 centers, 46 (23%) were low performing (LP) with statistically significantly lower than expected 1‐year graft or patient survival at least once during the study period. Among LP centers, there was a mean decline in transplant volume of 22.4 cases compared to a mean increase of 7.8 transplants among other centers (p = 0.001). Changes in volume between LP and other centers were significant for living, standard and expanded criteria deceased donor (ECD) transplants. LPs had a reduction in use of donors with extended cold ischemia time (p = 0.04) and private pay recipients (p = 0.03). Centers without low performance evaluations were more likely to increase the proportion of overall transplants that were ECDs relative to other centers (p = 0.04). Findings indicate a significant association between reduced kidney transplant volume and low performance report card evaluations.