Evaluation Guidelines for the Clinical and Translational Science Awards (CTSAs)

The National Center for Advancing Translational Sciences (NCATS), a part of the National Institutes of Health, currently funds the Clinical and Translational Science Awards (CTSAs), a national consortium of 61 medical research institutions in 30 states and the District of Columbia. The program seeks to transform the way biomedical research is conducted, speed the translation of laboratory discoveries into treatments for patients, engage communities in clinical research efforts, and train a new generation of clinical and translational researchers. An endeavor as ambitious and complex as the CTSA program requires high‐quality evaluations in order to show that the program is well implemented, efficiently managed, and demonstrably effective. In this paper, the Evaluation Key Function Committee of the CTSA Consortium presents an overall framework for evaluating the CTSA program and offers policies to guide the evaluation work. The guidelines set forth are designed to serve as a tool for education within the CTSA community by illuminating key issues and practices that should be considered during evaluation planning, implementation, and utilization. Additionally, these guidelines can provide a basis for ongoing discussions about how the principles articulated in this paper can most effectively be translated into operational reality.     

Clinical and Translational Scientist Career Success: Metrics for Evaluation

Despite the increased emphasis on formal training in clinical and translational research and the growth in the number and scope of training programs over the past decade, the impact of training on research productivity and career success has yet to be fully evaluated at the institutional level. In this article, the Education Evaluation Working Group of the Clinical and Translational Science Award Consortium introduces selected metrics and methods associated with the assessment of key factors that affect research career success. The goals in providing this information are to encourage more consistent data collection across training sites, to foster more rigorous and systematic exploration of factors associated with career success, and to help address previously identified difficulties in program evaluation. Clin Trans Sci 2012; Volume 5: 400–407     

NIH Grant Awards as a Metric of Clinical and Translational Research Training Effectiveness

The number of clinical research training programs has increased over the past 5–10 years, but few studies have quantitatively evaluated the effectiveness of these programs. The goal of this study was to evaluate the clinical and translational research training program at the University of Cincinnati by comparing the number of National Institutes of Health grants awarded to pediatric fellows who graduated from the MS degree program between 1995 and 2013 versus fellows who did not pursue an MS degree. Among 394 pediatric fellows, 16 of 81 (20%) MS alumni were awarded at least one NIH grant, as compared with 28 of 313 (9%) fellows who did not obtain an MS degree (p < 0.02). In multivariable analysis, MS alumni were more than three times as likely to have received at least one grant than were non‐MS fellows (OR = 3.5, 95% CI [1.7–7.2]; C‐statistic = 0.71) and MS alumni were more likely to obtain at least one K‐series (OR = 4.1, 95% CI [1.6–10.2]; C‐statistic = 0.74), M‐series (OR = 11.8, 95% CI [3.4–41.4]; C‐statistic = 0.81), or R‐series (OR = 10.1, 95% CI [2.4–42.8]; C‐statistic = 0.74) grant than were non‐MS fellows. These findings suggest that graduate training in clinical and translational research prepares graduates for the highly competitive field of clinical and translational research.     

Developing the Translational Research Workforce: A Pilot Study of Common Metrics for Evaluating the Clinical and Translational Award KL2 Program

Purpose

This pilot study describes the career development programs (i.e., NIH KL2 awards) across five Clinical and Translational Science Award (CTSA) institutions within the University of California (UC) system, and examines the feasibility of a set of common metrics for evaluating early outcomes.

Methods

A survey of program administrators provided data related to the institutional environment within which each KL2 program was implemented. Application and progress report data yielded a combined data set that characterized KL2 awardees, their initial productivity, and early career outcomes.

Results

The pilot project demonstrated the feasibility of aggregating common metrics data across multiple institutions. The data indicated that KL2 awardees were an accomplished set of investigators, both before and after the award period, representing a wide variety of disciplines. Awardees that had completed their trainee period overwhelmingly remained active in translational research conducted within an academic setting. Early indications also suggest high rates of success with obtaining research funding subsequent to the KL2 award.

Conclusion

This project offers a model for how to collect and analyze common metrics related to the education and training function of the CTSA Consortium. Next steps call for expanding participation to other CTSA sites outside of the University of California system.     

A Distributed Model: Redefining a Robust Research Subject Advocacy Program at the Harvard Clinical and Translational Science Center

The Harvard Clinical and Translational Science Center (“Harvard Catalyst”) Research Subject Advocacy (RSA) Program has reengineered subject advocacy, distributing the delivery of advocacy functions through a multi‐institutional, central platform rather than vesting these roles and responsibilities in a single individual functioning as a subject advocate. The program is process‐oriented and output‐driven, drawing on the strengths of participating institutions to engage local stakeholders both in the protection of research subjects and in advocacy for subjects’ rights. The program engages stakeholder communities in the collaborative development and distributed delivery of accessible and applicable educational programming and resources. The Harvard Catalyst RSA Program identifies, develops, and supports the sharing and distribution of expertise, education, and resources for the benefit of all institutions, with a particular focus on the frontline: research subjects, researchers, research coordinators, and research nurses.     

The Quantitative Evaluation of the Clinical and Translational Science Awards (CTSA) Program Based on Science Mapping and Scientometric Analysis

The Clinical and Translational Science Awards (CTSA) program is one of the most important initiatives in translational medical funding. The quantitative evaluation of the efficiency and performance of the CTSA program has a significant referential meaning for the decision making of global translational medical funding. Using science mapping and scientometric analytic tools, this study quantitatively analyzed the scientific articles funded by the CTSA program. The results of the study showed that the quantitative productivities of the CTSA program had a stable increase since 2008. In addition, the emerging trends of the research funded by the CTSA program covered clinical and basic medical research fields. The academic benefits from the CTSA program were assisting its members to build a robust academic home for the Clinical and Translational Science and to attract other financial support. This study provided a quantitative evaluation of the CTSA program based on science mapping and scientometric analysis. Further research is required to compare and optimize other quantitative methods and to integrate various research results.     

Evaluating Various Areas of Process Improvement in an Effort to Improve Clinical Research: Discussions from the 2012 Clinical Translational Science Award (CTSA) Clinical Research Management Workshop

Emphasis has been placed on assessing the efficiency of clinical and translational research as part of the National Institutes of Health (NIH) goal to “improve human health.” Improvements identified and implemented by individual organizations cannot address the research infrastructure needs of all clinical and translational research conducted. NIH''s National Center for Advancing Translational Sciences (NCATS) has brought together 61 Clinical and Translational Science Award (CTSA) sites creating a virtual national laboratory that reflects the diversity and breadth of academic medical centers to collectively improve clinical and translational science. The annual Clinical Research Management workshop is organized by the CTSA consortium with participation from CTSA awardees, NIH, and others with an interest in clinical research management. The primary objective of the workshop is to disseminate information that improves clinical research management although the specific objectives of each workshop evolve within the consortium. The fifth annual workshop entitled “Learning by doing; applying evidence‐based tools to re‐engineer clinical research management” took place in June 2012. The primary objective of the 2012 workshop was to utilize data to evaluate, modify, and improve clinical research management. This report provides a brief summary of the workshop proceedings and the major themes discussed among the participants.     

Graduate Education for the Future: New Models and Methods for the Clinical and Translational Workforce

This paper is the third in a five‐part series on the clinical and translational science educational pipeline, and it focuses on strategies for enhancing graduate research education to improve skills for interdisciplinary team science. Although some of the most cutting edge science takes place at the borders between disciplines, it is widely perceived that advancements in clinical and translational science are hindered by the “siloed” efforts of researchers who are comfortable working in their separate domains, and reluctant to stray from their own discipline when conducting research. Without appropriate preparation for career success as members and leaders of interdisciplinary teams, talented scientists may choose to remain siloed or to leave careers in clinical and translational science all together, weakening the pipeline and depleting the future biomedical research workforce. To address this threat, it is critical to begin at what is perhaps the most formative moment for academics: graduate training. This paper focuses on designs for graduate education, and contrasts the methods and outcomes from traditional educational approaches with those skills perceived as essential for the workforce of the future, including the capacity for research collaboration that crosses disciplinary boundaries.     

The Critical Need for Academic Health Centers to Assess the Training,Support, and Career Development Requirements of Clinical Research Coordinators: Recommendations from the Clinical and Translational Science Award Research Coordinator Taskforce

Clinical Research Coordinators (CRCs) are a vital component of the clinical research enterprise providing a pivotal role in human subject protection through the numerous activities and responsibilities assigned to them. In 2006, the National Institutes of Health’s National Center for Research resources (NCRR) implemented the Clinical and Translational Science Awards program (CTSA) to advance biomedical research. As a part of this endeavor, many workgroups were formed among the Consortium to support translational research. The Research Coordinator Taskforce was created as part of the Regulatory Knowledge group of the Clinical Research Innovation Key Function Committee, and focuses on enhancing CTSA capabilities to provide support and training for CRCs. In the spring of 2008, this taskforce conducted two surveys of the then 24 CTSA Consortium members to better understand the current expectations and responsibilities of research coordinators in addition to the mechanism for providing education, training, and support in order for CRCs to successfully meet the study responsibilities placed upon them. The results of these surveys are summarized in this article and provide context to the recommendations of the Research Coordinator Taskforce for institutional considerations, approaches, and best practices for providing education, training, and support the expanding role of CRCs in fulfilling their responsibilities delegated to them by investigators. Clin Trans Sci 2012; Volume 5: 470–475     

Training the Next Generation of Research Mentors: The University of California,San Francisco,Clinical & Translational Science Institute Mentor Development Program

Mentoring is a critical component of career development and success for clinical translational science research faculty. Yet few programs train faculty in mentoring skills. We describe outcomes from the first two faculty cohorts who completed a Mentor Development Program (MDP) at UCSF. Eligibility includes having dedicated research time, expertise in a scientific area and a desire to be a lead research mentor. A post‐MDP survey measured the program''s impact on enhancement of five key mentoring skills, change in the Mentors‐in‐Training (MIT) self‐rated importance of being a mentor to their career satisfaction, and overall confidence in their mentoring skills. Since 2007, 29 MITs participated in and 26 completed the MDP. Only 15% of the MITs reported any previous mentor training. Overall, 96% of MITs felt that participation in the MDP helped them to become better mentors. A majority reported a significant increase in confidence in mentoring skills and most reported an increased understanding of important mentoring issues at UCSF. MITs reported increased confidence in overall and specific mentoring skills after completion of the MDP. The MDP can serve as a model for other institutions to develop the next generation of clinical‐translational research mentors.     

Facilitating Emergency Care Research Networks: Integration into the Clinical Translational and Science Award (CTSA) Infrastructure

Emergency care research (ECR) does not fit neatly into the traditional National Institutes of Health (NIH) funding model, because emergency research involves undifferentiated disease presentations involving multiple disciplines and time-sensitive interventions. A task force of emergency care researchers and other stakeholders was convened to discuss the present and future state of clinical research networks. Integration of ECR with the Clinical Translational and Science Award (CTSA) program through a multidisciplinary emergency care research network (ECRN) would obviate the duplication of research efforts by disease-specific or institute-specific multicenter networks and reduce startup and maintenance costs. Strategies to enhance integration must include the training of emergency physician investigators in biostatistical and epidemiologic methods, as well as educating collaborative investigators in emergency care–related methodologies. Thus, an ECRN would be of great benefit to CTSA awardees and applicants and should be considered a priority.     

Applying Process Improvement Methods to Clinical and Translational Research: Conceptual Framework and Case Examples

There is growing appreciation that process improvement holds promise for improving quality and efficiency across the translational research continuum but frameworks for such programs are not often described. The purpose of this paper is to present a framework and case examples of a Research Process Improvement Program implemented at Tufts CTSI. To promote research process improvement, we developed online training seminars, workshops, and in‐person consultation models to describe core process improvement principles and methods, demonstrate the use of improvement tools, and illustrate the application of these methods in case examples. We implemented these methods, as well as relational coordination theory, with junior researchers, pilot funding awardees, our CTRC, and CTSI resource and service providers. The program focuses on capacity building to address common process problems and quality gaps that threaten the efficient, timely and successful completion of clinical and translational studies.