Twelve tips for improvement-oriented evaluation of competency-based medical education

Abstract

The shift to competency-based medical education (CBME) requires a new approach to program evaluation. CBME implementers need to embed evaluation in their programs to ensure their CBME adapts to the changing demands of the healthcare system. This 12 tips paper proposes that those advancing CBME use an improvement-oriented, utilization-focused approach to program evaluation. This will yield information that can help CBME implementers to continually examine the context, process and early outcomes of their programs. The paper uses examples from the College of Family Physicians of Canada’s (CFPC’s) evaluation of the implementation of the Triple C Competency-based curriculum in family medicine residency programs across Canada. These practical tips will be useful to medical educators looking to integrate evaluation into their CBME programs and to those considering other curriculum reform in health professions education.     

Eight-year outcomes of a competency-based residency training program in orthopedic surgery

Abstract

Background: The Division of Orthopaedic Surgery at the University of Toronto implemented a pilot residency training program that used a competency-based framework in July of 2009. The competency-based curriculum (CBC) deployed an innovative, modularized approach that dramatically intensified both the structured learning elements and the assessment processes.

Methods: This paper discusses the initial curriculum design of the CBC pilot program; the refinement of the curriculum using curriculum mapping that allowed for efficiencies in educational delivery; details of evaluating resident competence; feedback from external reviews by accrediting bodies; and trainee and program outcomes for the first eight years of the program’s implementation.

Results: Feedback from the residents, the faculty, and the postgraduate residency training accreditation bodies on the CBC has been positive and suggests that the essential framework of the program may provide a valuable tool to other programs that are contemplating embarking on transition to competency-based education.

Conclusions: While the goal of the program was not to shorten training per se, efficiencies gained through a modular, competency-based program have resulted in shortened time to completion of residency training for some learners.     

Evidence-based medicine (EBM) for undergraduate medical students: A six-step,integrative approach

Abstract

Background: Nowadays, there is an increasing interest in practicing evidence-based medicine (EBM) to provide valid and up-to-date information about health problems and procedures for solving these problems. However, studying EBM among other medical education disciplines remains unsatisfactory.

Aim: To design and pilot a basic course on EBM for undergraduate medical students in order to raise the awareness of the Saudi medical education community about this discipline.

Subjects and methods: The developed course targeted undergraduate Year 4 medical students at Ibn Sina National College for Medical Studies in Jeddah, Saudi Arabia. The course was integrated longitudinally into ?Integrated Multisystem Module? of Year 4. Students and faculty perceptions were evaluated for program evaluation purposes. Course design was based on Kern et al. six-step approach for curriculum development.

Results: Students overall perception of this course was positive except for some points related to the time allocated and their training on using evidence databases. Faculty members who participated in implementing the course perceived it positively.

Conclusions: Teaching EBM is recommended and applicable. It is not resource-intensive, and can be conducted through developing and integrating EBM courses in the undergraduate medical curriculum. Students and teachers agree upon the need and importance of teaching such discipline.     

Using language of entrustable professional activities to define learning objectives of radiology clerkship: A modified Delphi study

Abstract

Background: A standard undergraduate radiology education is essential to prepare graduates for multidisciplinary clinical practice yet the literature lacks clear guidelines or consensus about the learning objectives of an optimal radiology clerkship.

Aim: To define a competency-based framework for undergraduate radiology education by using language of Entrustable Professional Activities (EPAs).

Methods: A modified Delphi method with three iterative rounds was used as an expert consensus approach. An online questionnaire with Likert scale was formulated incorporating EPAs and their components (competencies, assessment strategies, and supervision level) and distributed to 45 consultant radiologists following pilot study. Items reaching consensus were accepted and rest were resent in round 2. In round 3, a dichotomous scale was used for final approval and to see response stability.

Results: A final set of six EPAs with 87 competencies and respective assessment strategies, all aiming for ‘level 3a’ of supervision was identified. These include recommending cost effective appropriate imaging tests for common pathologies, obtaining informed consent for diagnostic contrast studies, basic interpretation and communication of common pathologies/emergencies on radiographs (chest, abdominal, and skeletal) and on CT brain.

Conclusion: This EPA framework for radiology clerkship is a first step towards a competency-based approach to undergraduate radiology training and assessment.     

School of Medicine of Federal University of Rio Grande Do Norte: A traditional curriculum with innovative trends in medical education

Abstract

Introduction: The Medical School of the Federal University of Rio Grande do Norte (UFRN) is one of the biggest public medical schools in Northeast Brazil. In the last decade, significant investment in faculty development, innovative learning methodologies and student engagement has been key milestones in educational improvement at this medical school, harnessed to recent political changes that strengthened community-based and emergency education. This study describes how curriculum changes in UFRN Medical School have been responsible for major improvements in medical education locally and which impacts such transformations may have on the educational community.

Methods: A group of students and teachers revised the new curriculum and established the key changes over the past years that have been responsible for the local enhancement of medical education. This information was compared and contrasted to further educational evidences in order to define patterns that can be reproduced in other institutions.

Results: Improvements in faculty development have been fairly observed in the institution, exemplified by the participation of a growing number of faculty members in programs for professional development and also by the creation of a local masters degree in health education. Alongside, strong student engagement in curriculum matters enhanced the teaching-learning process.

Conclusions: Due to a deeper involvement of students and teachers in medical education, it has been possible to implement innovative teaching-learning and assessment strategies over the last ten years and place UFRN Medical School at a privileged position in relation to undergraduate training, educational research and professional development of faculty staff.     

The alignment imperative in curriculum renewal

Abstract

Rationale: There are perennial calls for MD curricula to reform in order to meet the changing needs of students, patients, and society. And yet, efforts at renewal have also been suggested to have minimal impact on the pedagogy and outcomes of medical education. One reason may be misalignment between the components of the curriculum during design and implementation. The University of Toronto MD program recently renewed its undergraduate preclinical Foundations curriculum. Mindful of the pitfalls of misalignment, the renewal process focused deliberately on alignment between the various components of the curriculum: instructional methods, student assessment, faculty development, and the larger purpose of serving students and society.

Innovation: Educational evidence was used to drive the alignment process which resulted in three major changes. First, we created a spiral curriculum centered on 72 virtual patient cases designed to integrate content and prepare students for clinical learning. Second, we introduced a novel medical psychiatry component to address a core societal need in mental health. This exposed students early to experiences of complexity, ambiguity, and integrated patient care. Lastly, a shift to assessment for learning and programmatic assessment was designed and implemented concurrently to reinforce the pedagogy of the curriculum. Synchronous faculty development was developed for the new roles required of faculty.

Conclusions: Early program evaluation shows alignment of these curricular components requires ongoing attention and resources in order to be successful. The potential benefits of this alignment are well prepared students who can meet the needs of their patients and society in an increasingly complex health system.     

Transforming medical education in Kazakhstan: Successful case of internationalization from Karaganda State Medical University

Abstract

Background: Medical education in Kazakhstan has been literally transformed in the past 10 years. Kazakhstan inherited the Soviet-time discipline-based teacher-centered system of education when no decisions could be made independently. The curriculum was mostly governed in a traditional way, with lectures being the core, little use of e-learning tools, and assessment through oral exams and multiple-choice questions. Most of the universities still preserve the subject-based curriculum with elements of integrated learning.

Methods: Being the most active member of International Space Education, Karaganda State Medical University (KSMU) took the initiative to adapt the full integrated curriculum mostly based on problem-based, team-based learning, and use of virtual patient cases. The given approach was chosen because of active involvement of our University in nine Tempus and Erasmus+projects including reforming of Public Health and Nursing curriculum, human resources development, active learning, credit mobility, and move towards autonomy of medical schools.

Results: KSMU became the coordinator of two of these projects, taking its active position in internationalization of medical education. We actively use technology-based medical education, pro-actively adapting deliberate practice in acquiring essential practical skills, for which KSMU was recognized by an ASPIRE-to-Excellence Award in simulation.

Conclusions: Kazakhstan hopes to become the leader in medical education in Central Asia and suggests other Universities in the area to adopt its approach to internationalization of medical education.     

Growth of the James Cook University Medical Program: Maintaining quality,continuing the vision,developing postgraduate pathways

Abstract

Background: James Cook University (JCU) enrolled its first cohort of 64 in 2000 into a 6-year undergraduate medical program aimed at producing graduates capable of meeting the needs of North Queensland, Australia, with a focus on rural, remote, Indigenous and tropical health. The school’s 1465 graduates over 13 cohorts who have a pattern of practice likely to meet the region’s health needs. The JCU course was the first new Australian medical program for 25?years. The number of Australian medical schools has since doubled, while enrollments have almost tripled.

Methods: JCU’s course features innovations such as dispersed, community-based education, rurally-focused selection, extended rural placements, and an emphasis on community needs – which are all now mainstream. This paper traces developments at JCU over the past decade, illustrating parallels with the broader Australian scene.

Results: Maintaining quality and educational integrity while numbers grow is challenging. The course has undergone modest curriculum redesign, but the fundamental elements are intact. The focus on meeting the region’s needs remains, with some evolution of its mission to include social accountability and the needs of underserved populations.

Conclusions: Postgraduate pathways are an important priority. Regional training hubs are being developed to support local pipelines into specialty practice. Queensland’s Rural Generalist Pathway provides an incentivised pathway to rural practice while Generalist Medical Training provides a local training pipeline into general practice and rural medicine. As these initiatives mature, communities should benefit as JCU and other Australian programs continue to address local workforce needs.     

Integrating an interprofessional education initiative: Evidence from King Abdulaziz University

Abstract

Purpose: This paper examines current issues with interprofessional education (IPE) at King Abdulaziz University (KAU) and discusses initiatives for integrating IPE into the medical curricula at KAU.

Methods: We reviewed the current body of literature, studied reports from IPE conferences and workshops organized at KAU, and synthesized participants' feedback from the IPE programs, including an online survey.

Results: A total of 506 participants responded to the online survey. Respondents rated Interprofessional Collaborative Learning as the highest category of IPE, followed by Interprofessional Self-Improvement and Interprofessional Relationship. A hybrid conceptual framework is proposed, to tackle the issue of role clarification across all healthcare colleges at KAU. This proposition was found to be necessary due to the current state of the undergraduate curriculum which does not prepare students properly for professional collaboration.

Conclusions: The hybrid model may narrow the gap in IPE by emphasizing professional identity while reducing autonomy. Recommendations toward IPE are presented. Challenges toward IPE reform are discussed in the context of implementation at KAU and at other medical schools in the region.     

Describing medical student curiosity across a four year curriculum: An exploratory study

Background: Intellectual curiosity can be defined as a desire for knowledge that leads to exploratory behavior and consists of an inherent and stable trait (i.e. trait curiosity) and a variable context-dependent state (i.e. state curiosity). Although intellectual curiosity has been considered an important aspect of medical education and practice, its relationship to medical education has not been empirically investigated. The purpose of this exploratory study was to describe medical students’ intellectual curiosity across a four-year undergraduate program.

Methods: We employed a cross-sectional design in which medical students, across a four-year undergraduate program at McGill University, completed the Melbourne Curiosity Inventory as a measure of their state and trait intellectual curiosity. A Mixed Models ANOVA was used to compare students across year of training.

Results: Four hundred and two out of 751 students completed the inventory (53.5%). Trait curiosity was significantly higher than state curiosity (M?=?64.5, SD?=?8.5 versus M?=?58.5, SD?=?11.6) overall, and within each year of training.

Conclusions: This study is the first to describe state and trait intellectual curiosity in undergraduate medical education. Findings suggest that medical students’ state curiosity may not be optimally supported and highlight avenues for further research.     

Empowering future physicians to advocate for health equity: A blueprint for a longitudinal thread in undergraduate medical education

Abstract

In light of a growing body of evidence demonstrating pervasive health disparities, medical schools are increasingly incorporating educational programs on social medicine in undergraduate and graduate medical curricula. In 2015, we significantly restructured the cultural competency instruction for medical students at our institution, focusing on achieving greater health equity through caring for vulnerable populations and acknowledging and addressing bias and stereotyping. In order to facilitate educational sustainability while students were immersed in clinical care, a key element of our approach included extending teaching into the clerkship year. The resulting longitudinal thread, Health Equity and Social Justice, empowers future physicians with the knowledge and skills to work towards greater health equity. This article discusses the lessons learned in the implementation of this novel educational program. Our approach can serve as a model for other institutions considering similar instructional reform.     

Flexibility in individualized,competency-based workplace curricula with EPAs: Analyzing four cohorts of physician assistants in training

Background: Entrustable professional activities (EPAs) were introduced as a principle for individualized physician assistant (PA) workplace curricula at the University of Applied Sciences (UAS) Utrecht in 2008. We studied how the focus on EPAs served the competency-based flexibility intention of the program.

Methods: We analyzed data of those 119 students who enrolled in the program 2010 through 2013, and completed the program before April 2016. We analyzed the number of EPAs per student at start and end of the program, number changed during training and the reasons for change.

Results: Data of 101 students were suitable for evaluation. Excluded were 16 students ending the program prematurely and two with study delay.

Mean number of EPAs per student at the start was 6.8 (range 4–12) and at the end 6.6 (range 3–13). On average 1.5 EPAs were altered (range 0–13). Reasons included extension of the EPA package during training (n?=?10), lack of proficiency at planned moments of summative entrustment decisions (n?=?9) and procedures not being suitable for PAs at closer look (n?=?6). All changes resulted in a curriculum meeting the school’s standards for graduation.

Conclusions: The flexibility of the EPA concept enabled changes in the individualized curriculum of students, according to the intended competency-based nature of the educational program.     

A decade of reform in medical education: Experiences and challenges at Tehran University of Medical Sciences

Abstract

Purpose: In this paper, we present the major curricular reform in MD program of Tehran University of Medical Sciences, the oldest and the largest medical university in Iran, initiated about a decade ago.

Materials and methods: Following a comprehensive program evaluation, many of the basic challenges of the traditional curriculum were revealed, namely, lack of pre-defined competencies for graduates, over-reliance on teacher-centered teaching methods, over-emphasis on knowledge base in student assessments, and focusing solely on biomedical aspects of patient care. In 2010, a vision statement for reform was created and approved by the University Council. The new curriculum was launched in 2011.

Results: The changes included: revising the content of the courses, assimilating horizontal and vertical integration, emphasizing clinical skills, encouraging active involvement in patient management, providing more opportunity for supervised practice, integrating behavioral and psychosocial topics into the curriculum, incorporating interactive teaching methods, assessing students’ higher levels of cognition, and strengthening workplace assessments. To evaluate the changes, data were continuously collected and analyzed from the beginning.

Conclusions: Changing the curriculum of an MD program is a laborious task which should be planned and undertaken carefully and cautiously. It is an endless, yet invaluable and satisfying endeavor toward better future.     

Training disrupted: Practical tips for supporting competency-based medical education during the COVID-19 pandemic

Abstract

The COVID-19 pandemic has disrupted healthcare systems around the world, impacting how we deliver medical education. The normal day-to-day routines have been altered for a number of reasons, including changes to scheduled training rotations, physical distancing requirements, trainee redeployment, and heightened level of concern. Medical educators will likely need to adapt their programs to maximize learning, maintain effective care delivery, and ensure competent graduates. Along with a continued focus on learner/faculty wellness, medical educators will have to optimize existing training experiences, adapt those that are no longer viable, employ new technologies, and be flexible when assessing competencies. These practical tips offer guidance on how to adapt medical education programs within the constraints of the pandemic landscape, stressing the need for communication, innovation, collaboration, flexibility, and planning within the era of competency-based medical education.     

ScribeMD: Mixed-methods analysis of a pen pal program in undergraduate medical education

Abstract

Introduction: A growing emphasis on humanistic medical care has led to the development of programs to imbue more humanistic values into training physicians. The ScribeMD intra-class pen pal program is a narrative medicine tool designed to meet this need with a focus on the journey through medical education. This study intends to evaluate the efficacy of this novel program on the participants’ professional identity formation (PIF), emotional intelligence development (EID), and narrative competency improvement (NCI).

Methods: This program evaluation involved quantitative and qualitative survey solicitations from first and second-year medical student participants and controls. An efficacy-triangulation model was developed as a quantitative outcome-measuring tool linking objective, writing prompts, and survey data.

Results: The quantitative results showed statistically significant improvements in line with the Efficacy Triangulation Model in the participant group. Qualitatively, reflective and insight-driven gains were elicited, as were notable themes of personal and community improvements.

Conclusion: ScribeMD poses a unique avenue for the development of more humanistic physicians during their medical education. We recommend additional development of the Efficacy Triangulation Model as well as the program design itself. We also recommend the piloting of this program in other healthcare-education settings both with medical education and other health professions.     

Educating universal professionals or global physicians? A multi-centre study of international medical programmes design

Abstract

Introduction: Medical schools increasingly offer curricula that specifically aim to prepare students for an international medical career. This is challenging as well as controversial: curriculum designers must balance specific local healthcare requirements with global health competencies doctors need in our globalised world. By investigating how international medical programme designers experience this balancing act, this study aims to contribute insights to the debate on local versus global medical education.

Methods: We conducted a multi-centre instrumental case study across three universities with international medical programmes in three countries. The study involved 26 semi-structured interviews with key curriculum designers recruited through purposive sampling. Additionally, we performed a curriculum document analysis. Data were thematically analysed within a multidisciplinary team.

Results: Participants described two profiles of international medical programme graduates: ‘a global physician’, equipped with specific competencies for international practice, and ‘a universal professional’, an overall high-level graduate fit for future practice anywhere. These perspectives presented different curriculum design challenges.

Conclusions: International medical programmes teach us how we can rethink graduate profiles in a globalising world. Yet, educational standardisation poses risks and securing equity in global health education is challenging, as is preparing students to be adaptable to the requirements of a rapidly changing future local healthcare context.     

Training medical students for the twenty-first century: Rationale and development of the Utrecht curriculum “CRU+”

Abstract

Aim: The aim of this report, written for the 40th anniversary issue of Medical Teacher, is to document 20 years of development of the Utrecht undergraduate medical curriculum, as both to exhibit accountability and to inform the community of the process and choices that can be made in long-term curriculum development.

Methods: We used the SPICES model, created by Medical Teacher’s Editor Ronald Harden and colleagues in 1984.

Results: The Utrecht six-year program, now called “CRU+”, has many distinct features that were introduced, most of which are well documented. A limited selection includes

• ???A new 3+3 years Bachelor-Master structure following the EU Bologna rules leading to MD registration for cohorts of about 300.

• ???Horizontally integrated classroom teaching of basic sciences with clinical disciplines predominantly in groups of 12 and limited lectures.

• ???Mandatory knowledge retention tests, retesting the clinically relevant core knowledge from block tests of semesters one through four.

• ???Vertical integration not only linking clinical experience with background knowledge, but also exemplified by a stepwise increase in health care responsibilities throughout the curriculum.

• ???A final year focussing on growth towards the level of a primary responsible physician in a 12-week sub-internship for a limited number of patients and beds, in a chosen specialty. The student is called a semi-physician in the clerkship of this transitional year to residency.

• ???Teaching skills training for all medical graduates, an elective teaching rotation and various peer-teaching arrangements throughout the curriculum.

• ???Integrated semi-longitudinal clerkships with an assessment focus on entrustment decisions for Entrustable Professional Activities.

Conclusion: UMC Utrecht has made a continuous attempt to both develop its medical curriculum and to study and report on its development in the literature, regarding new methods found and insights derived. UMC Utrecht will remain committed to developing training to meet twenty-first century demands of medical graduates.     

Reforms in VUmc School of Medical Sciences Amsterdam: Student engagement,a Minor elective semester and stakeholder collaboration in improving the quality of assessments

Abstract

Background: At VUmc School of Medical Sciences, major curricular reforms occurred in 2005 and 2015, related to the introduction of a Bachelor-Master structure, a new legislation from the Ministry of Education, the changing societal context, and taking note of students’ and teachers’ needs.

Summary of work: Along with the introduction of the Bachelor-Master system, the period between 2005 and 2009 saw the movement from traditional lecture-based teaching to small group teaching in a competency-based curriculum, in which the students were responsible for their learning. Student engagement grew through students’ designing learning modules and conducting some of the teaching. In the Bachelor program, an elective “Minor”, was designed to broaden and deepen the knowledge of our students beyond the core learning outcomes, in a discipline of their choice. The examination board (EB), responsible for maintaining the quality of assessment, was split into the General EB, which handled overall strategy issues, and the Executive EB, which handled student requests and monitored the quality of assessments.

Lessons learned: Students develop a sense of what education is about if they are provided opportunities in designing teaching and conducting it. A Minor elective in the medical study can provide the students with an opportunity to learn outside the medical field. Collaborative working between different stakeholders in a medical school is crucial for safeguarding the quality of assessments. Curricular reforms need time to be accepted and integrated into the culture of the medical school. The educational vision needs to be refreshed regularly in alignment with the changing societal context.