Entrustment decisions: Implications for curriculum development and assessment

Abstract

With increased interest in the use of entrustable professional activities (EPAs) in undergraduate and postgraduate medical education, comes questions about their implications for curriculum development and assessment. This paper addresses some of those questions, discussed at a symposium of the 2017 conference of AMEE, by presenting the components of an EPA, describing their importance and application, identifying their implications for assessment, and pinpointing some of challenges they pose in undergraduate and postgraduate settings. It defines entrustment, describes the three levels of trust, and presents trainee and supervisor factors that influence it as well as perceived benefits, and risks. Two aspects of EPAs have implications for assessment: units of professional practice and decisions based on entrustment, which impact an assessment’s blueprint, test methods, scores, and standards. In an undergraduate setting EPAs have great appeal, but work is needed to identify and develop a robust assessment system for core EPAs. At the postgraduate level, there is tension between the granularity of the competencies and the integrated nature of the EPAs. Even though work remains, EPAs offer an important step in the evolution of competency-based education.     

Development of core entrustable professional activities linked to a competency-based veterinary education framework

Abstract

Purpose: Despite the adoption of competency-based education in some veterinary schools over the past 15?years, only recently has a concerted effort been directed toward this in veterinary education internationally.

Methods: In 2015, educational leaders from the Association of American Veterinary Medical Colleges (AAVMC) member schools came together with a strong call to action to create shared tools for clinical competency assessment.

Results: This resulted in the formation of the AAVMC Competency-Based Veterinary Education (CBVE) Working Group, which then embarked on the creation of a shared competency framework and the development of eight core entrustable professional activities (EPAs) linked to this framework.

Conclusions: This paper will report on the development of these EPAs and their integration with the concurrently-developed CBVE Framework.     

Flexibility in individualized,competency-based workplace curricula with EPAs: Analyzing four cohorts of physician assistants in training

Background: Entrustable professional activities (EPAs) were introduced as a principle for individualized physician assistant (PA) workplace curricula at the University of Applied Sciences (UAS) Utrecht in 2008. We studied how the focus on EPAs served the competency-based flexibility intention of the program.

Methods: We analyzed data of those 119 students who enrolled in the program 2010 through 2013, and completed the program before April 2016. We analyzed the number of EPAs per student at start and end of the program, number changed during training and the reasons for change.

Results: Data of 101 students were suitable for evaluation. Excluded were 16 students ending the program prematurely and two with study delay.

Mean number of EPAs per student at the start was 6.8 (range 4–12) and at the end 6.6 (range 3–13). On average 1.5 EPAs were altered (range 0–13). Reasons included extension of the EPA package during training (n?=?10), lack of proficiency at planned moments of summative entrustment decisions (n?=?9) and procedures not being suitable for PAs at closer look (n?=?6). All changes resulted in a curriculum meeting the school’s standards for graduation.

Conclusions: The flexibility of the EPA concept enabled changes in the individualized curriculum of students, according to the intended competency-based nature of the educational program.     

Entrustable professional activities and facets of competence in a simulated workplace-based assessment for advanced medical students

Abstract

Background: Competence-based assessment formats in medical education usually focus on individual facets of competence (FOCs). The concept of ‘Entrustable Professional Activities’ (EPAs) encompasses supervisors’ decisions on which level of supervision a trainee requires to perform a professional activity including several FOCs. How the different FOCs as perceived by clinician raters contribute to entrustment decisions is yet unclear.

Objective: How do FOC perceptions relate to entrustment-decisions?

Methods: Sixty-seven advanced medical students participated in an assessment simulating the first day of a resident physician. Participants were rated by supervisors for seven FOCs and twelve EPAs.

Results: There was a positive correlation between FOC and EPA scores. Each EPA displayed a different correlation pattern with FOC ratings.

Discussion: For most EPAs high levels of entrustment were associated with high ratings for selected FOCs. The results are in alignment with the assumption that each EPA encompasses a different set of FOCs.

Conclusions: In our simulated workplace-based assessment, entrustment decisions for EPAs reflect the FOCs observed in a trainee. Thus, assessment of FOCs alongside with EPA ratings could add to the understanding of factors contributing to entrustment decisions.     

Eight-year outcomes of a competency-based residency training program in orthopedic surgery

Abstract

Background: The Division of Orthopaedic Surgery at the University of Toronto implemented a pilot residency training program that used a competency-based framework in July of 2009. The competency-based curriculum (CBC) deployed an innovative, modularized approach that dramatically intensified both the structured learning elements and the assessment processes.

Methods: This paper discusses the initial curriculum design of the CBC pilot program; the refinement of the curriculum using curriculum mapping that allowed for efficiencies in educational delivery; details of evaluating resident competence; feedback from external reviews by accrediting bodies; and trainee and program outcomes for the first eight years of the program’s implementation.

Results: Feedback from the residents, the faculty, and the postgraduate residency training accreditation bodies on the CBC has been positive and suggests that the essential framework of the program may provide a valuable tool to other programs that are contemplating embarking on transition to competency-based education.

Conclusions: While the goal of the program was not to shorten training per se, efficiencies gained through a modular, competency-based program have resulted in shortened time to completion of residency training for some learners.     

Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales

Abstract

Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.

Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017–18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.

Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.

Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.     

Defining a framework for medical teachers’ competencies to teach ethnic and cultural diversity: Results of a European Delphi study

Background: Medical students need to be trained in delivering diversity-responsive health care but unknown is what competencies teachers need. The aim of this study was to devise a framework of competencies for diversity teaching.

Methods: An open-ended questionnaire about essential diversity teaching competencies was sent to a panel. This resulted in a list of 74 teaching competencies, which was sent in a second round to the panel for rating. The final framework of competencies was approved by the panel.

Results: Thirty-four experts participated. The final framework consisted of 10 competencies that were seen as essential for all medical teachers: (1) ability to critically reflect on own values and beliefs; (2) ability to communicate about individuals in a nondiscriminatory, nonstereotyping way; (3) empathy for patients regardless of ethnicity, race or nationality; (4) awareness of intersectionality; (5) awareness of own ethnic and cultural background; (6) knowledge of ethnic and social determinants of physical and mental health of migrants; (7) ability to reflect with students on the social or cultural context of the patient relevant to the medical encounter; (8) awareness that teachers are role models in the way they talk about patients from different ethnic, cultural and social backgrounds; (9) empathy for students of diverse ethnic, cultural and social background; (10) ability to engage, motivate and let all students participate.

Conclusions: This framework of teaching competencies can be used in faculty development programs to adequately train all medical teachers.     

Competencies,milestones, and EPAs – Are those who ignore the past condemned to repeat it?

Background: The idea of competency-based education sounds great on paper. Who wouldn’t argue for a standardized set of performance-based assessments to assure competency in graduating students and residents? Even so, conceptual concerns have already been raised about this new system and there is yet no evidence to refute their veracity.

Aims: We argue that practical concerns deserve equal consideration, and present evidence strongly suggesting these concerns should be taken seriously.

Method: Specifically, we share two historical examples that illustrate what happened in two disparate contexts (K-12 education and the Department of Defense [DOD]) when competency (or outcomes-based) assessment frameworks were implemented. We then examine how observation and assessment of clinical performance stands currently in medical schools and residencies, since these methodologies will be challenged to a greater degree by expansive lists of competencies and milestones.

Results/Conclusions: We conclude with suggestions as to a way forward, because clearly the assessment of competency and the ability to guarantee that graduates are ready for medical careers is of utmost importance. Hopefully the headlong rush to competencies, milestones, and core entrustable professional activities can be tempered before even more time, effort, frustration and resources are invested in an endeavor which history suggests will collapse under its own weight.     

Defining curricular priorities for student-as-teacher programs: A National Delphi Study

Background: “Student-as-Teacher” (SaT) programs have been growing in number to prepare medical students for their teaching roles in residency and beyond, but it remains unknown what content areas should be covered in SaT curricula.

Aim: To determine five to ten “essential” content areas for inclusion in SaT curricula using expert opinion.

Methods: Using a three-round Delphi process, moderators iteratively surveyed a panel of 28 medical educators (25 academy directors and three individuals identified as having expertise in undergraduate medical education) representing 25 medical schools in the United States. This “SaT Delphi Working Group” was tasked with rating topics for inclusion in SaT curricula on a 3-point scale (i.e. 1. “essential,” 2. “important, but not essential” 3. “not important”). Topics achieving ≥70% consensus as “essential,” “important” or “not important” were accepted by the moderators and removed from subsequent rounds.

Results: Hundred per cent response rate (n?=?28) was achieved for all survey rounds. Five content areas reached consensus as “essential” for inclusion in a SaT curriculum: feedback, bedside teaching and clinical precepting, small-group teaching, case-based teaching and professionalism as a medical educator.

Conclusion: This consensus from a group of leaders in medical education is a first step toward the implementation of more developmentally-appropriate SaT competencies.     

Moving a mountain: Practical insights into mastering a major curriculum reform at a large European medical university

Abstract

Aim: Undergraduate medical education is currently in a fundamental transition towards competency-based programs around the globe. A major curriculum reform implies a dual challenge: the change of the curriculum and the delivering organization. Both are closely interwoven. In this article, we provide practical insights into our approach of managing such a fundamental reform of the large undergraduate medical program at the Charité – Universitätsmedizin Berlin.

Methods: Members of the project management team summarized the key features of the process with reference to the literature.

Results: Starting point was a traditional, discipline-based curriculum that was reformed into a fully integrated, competency-based program. This change process went through three phases: initiation, curriculum development and implementation, and sustainability. We describe from a change management perspective, their main characteristics, and the approaches that were employed to manage them successfully.

Conclusions: Our report is intended to provide practical insights and guidance for those institutions which are yet considering or have already started to undergo a major reform of their undergraduate programs towards competency medical education.     

Effects of an in-training assessment programme on supervision of and feedback on competencies in an undergraduate Internal Medicine clerkship

Assessment drives the educational behaviour of students and supervisors. Therefore, an assessment programme targeted at specific competencies may be expected to motivate supervisors and students to pay more attention to those competencies. In-training assessment (ITA) is regarded as a feasible method for assessing a broad range of competencies. Before and after the implementation of an ITA programme in an undergraduate Internal Medicine clerkship we surveyed students on the frequency of unobserved and observed supervision, and the quality of feedback as inferred from the seniority of the person providing it. After the implementation of the ITA programme supervision increased, but the difference was not statistically significant. The quality of feedback showed no significant change either. Inter-student variation in supervision and feedback remained invariably high after the implementation of the ITA programme. Whether these results are attributable to the way the programme was implemented or to the way the results were assessed remains to be clarified.     

A general practice workplace-based assessment instrument: Content and construct validity

Abstract

Introduction: Relatively few general practice (GP) workplace-based assessment instruments have been psychometrically evaluated. This study aims to establish the content validity and internal consistency of the General Practice Registrar Competency Assessment Grid (GPR-CAG).

Methods: The GPR-CAG was constructed as a formative assessment instrument for Australian GP registrars (trainees). GPR-CAG items were determined by an iterative literature review, expert opinion and pilot-testing process. Validation data were collected, between 2014 and 2016, during routine clinical teaching visits within registrars’ first two general practice training terms (GPT1 and GPT2) for registrars across New South Wales and the Australian Capital Territory. Factor analysis and expert consensus were used to refine items and establish GPR-CAG’s internal structure. GPT1 and GPT2 competencies were analysed separately.

Results: Data of 555 registrars undertaking GPT1 and 537 registrars undertaking GPT2 were included in analyses. A four-factor, 16-item solution was identified for GPT1 competencies (Cronbach’s alpha range: 0.71–0.83) and a seven-factor 27-item solution for GPT2 competencies (Cronbach’s alpha: 0.63–0.84). The emergent factor structures were clinically characterisable and resonant with existing medical education competency frameworks.

Discussion: This study establishes initial evidence for the content validity and internal consistency of GPR-CAG. GPR-CAG appears to have utility as a formative GP training WBA instrument.     

Development and alignment of undergraduate medical curricula in a web-based,dynamic Learning Opportunities,Objectives and Outcome Platform (LOOOP)

Introduction: This study presents a web-based method and its interface ensuring alignment of all parts of a curriculum map including competencies, objectives, teaching and assessment methods, workload and patient availability. Needs, acceptance and effectiveness are shown through a nine-year study.

Methods: After a comprehensive needs assessment, the curriculum map and a web-based interface “Learning Opportunities, Objectives and Outcome Platform” (LOOOP) were developed according to Harden’s conceptual framework of 10-steps for curriculum mapping. The outcome was measured by surveys and results of interdisciplinary MCQ-assessments. The usage rates and functionalities were analysed.

Results: The implementation of LOOOP was significantly associated with improved perception of the curriculum structure by teachers and students, quality of defined objectives and their alignment with teaching and assessment, usage by students to prepare examinations and their scores in interdisciplinary MCQ-assessment. Additionally, LOOOP improved the curriculum coordination by faculty, and assisted departments for identifying patient availability for clinical training.

Conclusion: LOOOP is well accepted among students and teachers, has positive effect on curriculum development, facilitates effective utilisation of educational resources and improves student’s outcomes. Currently, LOOOP is used in five undergraduate medical curricula including 85,000 mapped learning opportunities (lectures, seminars), 5000 registered users (students, teachers) and 380,000 yearly page-visits.     

Effects of implementing time-variable postgraduate training programmes on the organization of teaching hospital departments

Abstract

Introduction: As competency-based education has gained currency in postgraduate medical education, it is acknowledged that trainees, having individual learning curves, acquire the desired competencies at different paces. To accommodate their different learning needs, time-variable curricula have been introduced making training no longer time-bound. This paradigm has many consequences and will, predictably, impact the organization of teaching hospitals. The purpose of this study was to determine the effects of time-variable postgraduate education on the organization of teaching hospital departments.

Methods: We undertook exploratory case studies into the effects of time-variable training on teaching departments’ organization. We held semi-structured interviews with clinical teachers and managers from various hospital departments.

Results: The analysis yielded six effects: (1) time-variable training requires flexible and individual planning, (2) learners must be active and engaged, (3) accelerated learning sometimes comes at the expense of clinical expertise, (4) fast-track training for gifted learners jeopardizes the continuity of care, (5) time-variable training demands more of supervisors, and hence, they need protected time for supervision, and (6) hospital boards should support time-variable training.

Conclusions: Implementing time-variable education affects various levels within healthcare organizations, including stakeholders not directly involved in medical education. These effects must be considered when implementing time-variable curricula.     

Evaluation of the role of incentive structure on student participation and performance in active learning strategies: A comparison of case-based and team-based learning

Background: Student participation is important for the success of active learning strategies, but participation is often linked to the level of preparation. At our institution, we use two types of active learning activities, a modified case-based learning exercise called active learning groups (ALG) and team-based learning (TBL). These strategies have different assessment and incentive structures for participation. Non-cognitive skills are assessed in ALG using a subjective five-point Likert scale. In TBL, assessment of individual student preparation is based on a multiple choice quiz conducted at the beginning of each session.

Methods: We studied first-year medical student participation and performance in ALG and TBL as well as performance on course final examinations.

Results: Student performance in TBL, but not in ALG, was strongly correlated with final examination scores. Additionally, in students who performed in the upper 33rd percentile on the final examination, there was a positive correlation between final examination performance and participation in TBL and ALG. This correlation was not seen in students who performed in the lower 33rd percentile on the final examinations.

Conclusions: Our results suggest that assessments of medical knowledge during active learning exercises could supplement non-cognitive assessments and could be good predictors of performance on summative examinations.     

Restructuring Saudi Board in Restorative Dentistry (SBRD) curriculum using CanMEDS competency

Abstract

Objective: The purpose of this paper is to describe the process of adopting the Canadian Medical Education Directions for Specialists (CanMEDS) 2015 competency framework in a dental specialty program to reconstruct the Saudi Board in Restorative Dentistry (SBRD) curriculum and disseminate the lessons learned.

Method and development process: The process of curriculum development was started with the selection of SBRD curriculum committee and review of CanMEDS framework. The Committee conducted needs assessment among the stakeholders and adopted CanMEDS 2015 competencies through a careful process. A modeled curriculum was developed after taking feedback, review of existing literature, and unique context of dentistry.

Curriculum: Several unique features are incorporated. For example, milestones and continuum of learning are developed to enable residents develop competencies at different stages (transition to discipline, foundation of discipline, and core of discipline). Academic activities are restructured to encourage interactive, student-centered approaches, team work, intellectual curiosity, and scholarship. Learning outcomes are integrated throughout within several modules. Many formative assessment tools are adopted to promote learning and evaluate clinical skills.

Conclusions: This is the first published example of adopting CanMEDS competency framework in a dental specialty program. The success of developing SBRD curriculum has encouraged other dental specialties toward adopting CanMEDS 2015 frameworks for their own curricula.     

Does a socially-accountable curriculum transform health professional students into competent,work-ready graduates? A cross-sectional study of three medical schools across three countries

Abstract

Background: Socially-accountable health professional education (SAHPE) is committed to achieving health equity through training health-workers to meet local health needs and serve disadvantaged populations. This research assesses the biomedical and socially-accountable competencies and work-readiness of first year graduates from socially-accountable medical schools in Australia, the United States and Sudan.

Method: A self-administered survey to hospital and community health facility staff closely associated with the training and/or supervision of first year medical graduates from three SAHPE medical schools.

Main outcome measure: Likert scale ratings of key competencies of SAHPE graduates (as a group) employed as first-year doctors, compared to first year doctors from other medical schools in that country (as a group).

Findings: Supervisors rated medical graduates from the 3 SAHPE schools highly for socially-accountable competencies (‘communication skills’, ‘teamwork’, ‘professionalism’, ‘work-readiness’, ‘commitment to practise in rural communities’, ‘commitment to practise with underserved ethnic and cultural populations’), as well as ‘overall performance’ and ‘overall clinical skills’.

Interpretation: These findings suggest SAHPE medical graduates are well regarded by their immediate hospital supervisors, and SAHPE can produce a medical workforce as competent as from more traditional medical schools, but with greater commitment to health equity, working with underserved populations, and addressing local health needs.