Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned

The National Institutes of Health (NIH) Roadmap for Medical Research initiative, funded by the NIH Common Fund and offered through the Clinical and Translational Science Award (CTSA) program, developed more than 60 unique models for achieving the NIH goal of accelerating discoveries toward better public health. The variety of these models enabled participating academic centers to experiment with different approaches to fit their research environment. A central challenge related to the diversity of approaches is the ability to determine the success and contribution of each model. This paper describes the effort by the Evaluation Key Function Committee to develop and test a methodology for identifying a set of common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics. The project involved more than one‐fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.     

NIH Grant Awards as a Metric of Clinical and Translational Research Training Effectiveness

The number of clinical research training programs has increased over the past 5–10 years, but few studies have quantitatively evaluated the effectiveness of these programs. The goal of this study was to evaluate the clinical and translational research training program at the University of Cincinnati by comparing the number of National Institutes of Health grants awarded to pediatric fellows who graduated from the MS degree program between 1995 and 2013 versus fellows who did not pursue an MS degree. Among 394 pediatric fellows, 16 of 81 (20%) MS alumni were awarded at least one NIH grant, as compared with 28 of 313 (9%) fellows who did not obtain an MS degree (p < 0.02). In multivariable analysis, MS alumni were more than three times as likely to have received at least one grant than were non‐MS fellows (OR = 3.5, 95% CI [1.7–7.2]; C‐statistic = 0.71) and MS alumni were more likely to obtain at least one K‐series (OR = 4.1, 95% CI [1.6–10.2]; C‐statistic = 0.74), M‐series (OR = 11.8, 95% CI [3.4–41.4]; C‐statistic = 0.81), or R‐series (OR = 10.1, 95% CI [2.4–42.8]; C‐statistic = 0.74) grant than were non‐MS fellows. These findings suggest that graduate training in clinical and translational research prepares graduates for the highly competitive field of clinical and translational research.     

The Quantitative Evaluation of the Clinical and Translational Science Awards (CTSA) Program Based on Science Mapping and Scientometric Analysis

The Clinical and Translational Science Awards (CTSA) program is one of the most important initiatives in translational medical funding. The quantitative evaluation of the efficiency and performance of the CTSA program has a significant referential meaning for the decision making of global translational medical funding. Using science mapping and scientometric analytic tools, this study quantitatively analyzed the scientific articles funded by the CTSA program. The results of the study showed that the quantitative productivities of the CTSA program had a stable increase since 2008. In addition, the emerging trends of the research funded by the CTSA program covered clinical and basic medical research fields. The academic benefits from the CTSA program were assisting its members to build a robust academic home for the Clinical and Translational Science and to attract other financial support. This study provided a quantitative evaluation of the CTSA program based on science mapping and scientometric analysis. Further research is required to compare and optimize other quantitative methods and to integrate various research results.     

Developing the Translational Research Workforce: A Pilot Study of Common Metrics for Evaluating the Clinical and Translational Award KL2 Program

Purpose

This pilot study describes the career development programs (i.e., NIH KL2 awards) across five Clinical and Translational Science Award (CTSA) institutions within the University of California (UC) system, and examines the feasibility of a set of common metrics for evaluating early outcomes.

Methods

A survey of program administrators provided data related to the institutional environment within which each KL2 program was implemented. Application and progress report data yielded a combined data set that characterized KL2 awardees, their initial productivity, and early career outcomes.

Results

The pilot project demonstrated the feasibility of aggregating common metrics data across multiple institutions. The data indicated that KL2 awardees were an accomplished set of investigators, both before and after the award period, representing a wide variety of disciplines. Awardees that had completed their trainee period overwhelmingly remained active in translational research conducted within an academic setting. Early indications also suggest high rates of success with obtaining research funding subsequent to the KL2 award.

Conclusion

This project offers a model for how to collect and analyze common metrics related to the education and training function of the CTSA Consortium. Next steps call for expanding participation to other CTSA sites outside of the University of California system.     

Clinical and Translational Scientist Career Success: Metrics for Evaluation

Despite the increased emphasis on formal training in clinical and translational research and the growth in the number and scope of training programs over the past decade, the impact of training on research productivity and career success has yet to be fully evaluated at the institutional level. In this article, the Education Evaluation Working Group of the Clinical and Translational Science Award Consortium introduces selected metrics and methods associated with the assessment of key factors that affect research career success. The goals in providing this information are to encourage more consistent data collection across training sites, to foster more rigorous and systematic exploration of factors associated with career success, and to help address previously identified difficulties in program evaluation. Clin Trans Sci 2012; Volume 5: 400–407     

The CTSA Consortium's Catalog of Assets for Translational and Clinical Health Research (CATCHR)

The 61 CTSA Consortium sites are home to valuable programs and infrastructure supporting translational science and all are charged with ensuring that such investments translate quickly to improved clinical care. Catalog of Assets for Translational and Clinical Health Research (CATCHR) is the Consortium''s effort to collect and make available information on programs and resources to maximize efficiency and facilitate collaborations. By capturing information on a broad range of assets supporting the entire clinical and translational research spectrum, CATCHR aims to provide the necessary infrastructure and processes to establish and maintain an open‐access, searchable database of consortium resources to support multisite clinical and translational research studies. Data are collected using rigorous, defined methods, with the resulting information made visible through an integrated, searchable Web‐based tool. Additional easy‐to‐use Web tools assist resource owners in validating and updating resource information over time. In this paper, we discuss the design and scope of the project, data collection methods, current results, and future plans for development and sustainability. With increasing pressure on research programs to avoid redundancy, CATCHR aims to make available information on programs and core facilities to maximize efficient use of resources.     

Mentor Training within Academic Health Centers with Clinical and Translational Science Awards

Multiple studies highlight the benefits of effective mentoring in academic medicine. Thus, we sought to quantify and characterize the mentoring practices at academic health centers (AHCs) with Clinical and Translational Science Awards (CTSA). Here we report findings pertaining specifically to mentor training at the level of the KL2 mentored award program, and at the broader institutional level. We found only four AHCs did not provide any form of training. One‐time orientation was most prevalent at the KL2 level, whereas formal face‐to‐face training was most prevalent at the institutional level. Despite differences in format usage, there was general consensus at both the KL2 and institutional level about the topics of focus of face‐to‐face training sessions. Lower‐resource training formats utilized at the KL2 level may reveal a preference for preselection of qualified mentors, while institutional selection of resource‐heavy formats may be an attempt to raise the mentoring qualifications of the academic community as a whole. The present work fits into the expanding landscape of academic mentoring literature and sets the framework for future longitudinal, outcome studies focused on identifying the most efficient strategies to develop effective mentors.     

Faculty Participation in and Needs around Community Engagement within a Large Multiinstitutional Clinical and Translational Science Awardee

Community engagement is recommended to ensure the public health impact of NIH‐funded science. To understand the prevalence of community‐engaged research and faculty interest in and needs around this, from 2012 to 2013, an online survey (n = 3,022) was sent to UCLA Clinical and Translational Science Institute faculty. Among respondents, 45% reported community‐engaged project participation in the last year and 64% an interest in learning about community‐engaged research. Over 50% indicated career development and pilot grants would increase participation in community‐engaged research. A greater percentage of pretenure than tenured faculty (pretenure 54.9%, tenured 42.2%, p = 0008) noted faculty promotion criteria incentivizing community‐engaged research would increase participation. In adjusted analyses, African American (OR 4.06, CI 1.68–9.82, p = 0.002) and Latino (OR 1.91, CI 1.10–3.33, p = 0.022) faculty had higher odds of prior participation in community‐engaged projects than Whites. Female faculty had greater odds of interest (OR 1.40, CI 1.02–1.93, p = 0.038) in learning about community‐engaged research than males. African American (OR 4.31, CI 1.42–13.08, p = 0.010) and Asian/Pacific Islander (OR 2.24, CI 1.52–3.28, p < 0.001) faculty had greater interest in learning about community‐engaged research than Whites. To build community‐engaged faculty research capacity, CTSAs’ may need to focus resources on female and minority faculty development.     

Facilitating Emergency Care Research Networks: Integration into the Clinical Translational and Science Award (CTSA) Infrastructure

Emergency care research (ECR) does not fit neatly into the traditional National Institutes of Health (NIH) funding model, because emergency research involves undifferentiated disease presentations involving multiple disciplines and time-sensitive interventions. A task force of emergency care researchers and other stakeholders was convened to discuss the present and future state of clinical research networks. Integration of ECR with the Clinical Translational and Science Award (CTSA) program through a multidisciplinary emergency care research network (ECRN) would obviate the duplication of research efforts by disease-specific or institute-specific multicenter networks and reduce startup and maintenance costs. Strategies to enhance integration must include the training of emergency physician investigators in biostatistical and epidemiologic methods, as well as educating collaborative investigators in emergency care–related methodologies. Thus, an ECRN would be of great benefit to CTSA awardees and applicants and should be considered a priority.     

Clinical and Translational Research Capacity Building Needs in Minority Medical and Health Science Hispanic Institutions

A preliminary needs assessment was conducted among faculty and students of three minority medical and health science institutions comprising the Puerto Rico Clinical and Translational Research Consortium (PRCTRC). The Web‐based survey was focused on evaluating the training interests in the clinical and translational research core areas and competencies developed by the National Institutes of Health‐Clinical and Translational Sciences Award. The survey was the result of a team effort of three PRCTRC key function''s leaderships: Multidisciplinary Training and Career Development, Tracking and Evaluation and Community Research and Engagement. The questionnaire included 45 items distributed across five content areas including demographics, research training needs, training activities coordination and knowledge about the services offered by the PRCTRC. Analysis of research needs includes a sample distribution according to professor, assistant/associate professor and graduate students. The thematic area with highest response rate among the three groups was: “Identify major clinical/public health problems and relevant translational research questions,” with the competency “Identify basic and preclinical studies that are potential testable clinical research hypothesis.” These preliminary results will guide the training and professional development of the new generation of clinical and translational researchers needed to eliminate health disparities.     

Models of Interinstitutional Partnerships between Research Intensive Universities and Minority Serving Institutions (MSI) across the Clinical Translational Science Award (CTSA) Consortium

Health disparities are an immense challenge to American society. Clinical and Translational Science Awards (CTSAs) housed within the National Center for Advancing Translational Science (NCATS) are designed to accelerate the translation of experimental findings into clinically meaningful practices and bring new therapies to the doorsteps of all patients. Research Centers at Minority Institutions (RCMI) program at the National Institute on Minority Health and Health Disparities (NIMHD) are designed to build capacity for biomedical research and training at minority serving institutions. The CTSA created a mechanism fostering formal collaborations between research intensive universities and minority serving institutions (MSI) supported by the RCMI program. These consortium‐level collaborations activate unique translational research approaches to reduce health disparities with credence to each academic institutions history and unique characteristics. Five formal partnerships between research intensive universities and MSI have formed as a result of the CTSA and RCMI programs. These partnerships present a multifocal approach; shifting cultural change and consciousness toward addressing health disparities, and training the next generation of minority scientists. This collaborative model is based on the respective strengths and contributions of the partnering institutions, allowing bidirectional interchange and leveraging NIH and institutional investments providing measurable benchmarks toward the elimination of health disparities.     

Evaluating Various Areas of Process Improvement in an Effort to Improve Clinical Research: Discussions from the 2012 Clinical Translational Science Award (CTSA) Clinical Research Management Workshop

Emphasis has been placed on assessing the efficiency of clinical and translational research as part of the National Institutes of Health (NIH) goal to “improve human health.” Improvements identified and implemented by individual organizations cannot address the research infrastructure needs of all clinical and translational research conducted. NIH''s National Center for Advancing Translational Sciences (NCATS) has brought together 61 Clinical and Translational Science Award (CTSA) sites creating a virtual national laboratory that reflects the diversity and breadth of academic medical centers to collectively improve clinical and translational science. The annual Clinical Research Management workshop is organized by the CTSA consortium with participation from CTSA awardees, NIH, and others with an interest in clinical research management. The primary objective of the workshop is to disseminate information that improves clinical research management although the specific objectives of each workshop evolve within the consortium. The fifth annual workshop entitled “Learning by doing; applying evidence‐based tools to re‐engineer clinical research management” took place in June 2012. The primary objective of the 2012 workshop was to utilize data to evaluate, modify, and improve clinical research management. This report provides a brief summary of the workshop proceedings and the major themes discussed among the participants.     

Characterization of Investigators' Approach to Translational Research: A Qualitative Study

Background

Little is known about how investigators approach their research programs along the translational research continuum. Many consider the translational continuum to be linear, with research beginning at the bench and concluding with research at the bedside or in the community. We aimed to understand if translational investigators approach and view their research in this fashion.

Methods

We conducted semistructured individual interviews with 16 graduates of the University of Pittsburgh''s Multidisciplinary Clinical Research Scholars Program (KL2) in 2012.

Results

Our research revealed three characteristic models. The first model we called “linear” and represented the traditional approach. The second we called “holistic”; these investigators began with central research questions and sought to explore them in every direction of translation, not necessarily taking linear steps. The third model we called “technical”; in this model, investigators focused on a unique technology or methodology and applied it across multiple research contexts.

Conclusion

This study found that there are multiple ways that translational investigators approach their research program. Better understanding of these models can help educators and mentors guide investigators so that they can be more productive in their clinical or translational research career.     

Alignment of Emergency Medicine Research Efforts with Clinical and Translational Science Awards

The Clinical and Translational Science Awards (CTSA) represent a major new funding pathway for health science investigators seeking National Institutes of Health (NIH) funds. This new pathway provides institutional-level support for clinical and translational research and is not tied to one organ system or disease process, fitting well with emergency medicine (EM) research needs. These awards open unique opportunities for advancing EM research. The CTSA mechanism provides institutional support from the NIH to promote both clinical and translational science. Of the 60 expected awards, 38 sites are currently funded. EM investigators can benefit the institutions applying for these awards and simultaneously gain from involvement. Some opportunities for participation provided by the CTSA include research training programs, joining multidisciplinary research teams, seed grant funding, and use of the CTSA-developed research infrastructure. Involvement of EM can benefit institutions by enhancing acute care research collaboration both within and among institutions. Emergency medicine researchers at institutions either planning to submit a CTSA application or with funded CTSA grants are encouraged to become actively involved in CTSA-related research programs.