Entrustment decisions: Implications for curriculum development and assessment

Abstract

With increased interest in the use of entrustable professional activities (EPAs) in undergraduate and postgraduate medical education, comes questions about their implications for curriculum development and assessment. This paper addresses some of those questions, discussed at a symposium of the 2017 conference of AMEE, by presenting the components of an EPA, describing their importance and application, identifying their implications for assessment, and pinpointing some of challenges they pose in undergraduate and postgraduate settings. It defines entrustment, describes the three levels of trust, and presents trainee and supervisor factors that influence it as well as perceived benefits, and risks. Two aspects of EPAs have implications for assessment: units of professional practice and decisions based on entrustment, which impact an assessment’s blueprint, test methods, scores, and standards. In an undergraduate setting EPAs have great appeal, but work is needed to identify and develop a robust assessment system for core EPAs. At the postgraduate level, there is tension between the granularity of the competencies and the integrated nature of the EPAs. Even though work remains, EPAs offer an important step in the evolution of competency-based education.     

Entrustable professional activities and facets of competence in a simulated workplace-based assessment for advanced medical students

Abstract

Background: Competence-based assessment formats in medical education usually focus on individual facets of competence (FOCs). The concept of ‘Entrustable Professional Activities’ (EPAs) encompasses supervisors’ decisions on which level of supervision a trainee requires to perform a professional activity including several FOCs. How the different FOCs as perceived by clinician raters contribute to entrustment decisions is yet unclear.

Objective: How do FOC perceptions relate to entrustment-decisions?

Methods: Sixty-seven advanced medical students participated in an assessment simulating the first day of a resident physician. Participants were rated by supervisors for seven FOCs and twelve EPAs.

Results: There was a positive correlation between FOC and EPA scores. Each EPA displayed a different correlation pattern with FOC ratings.

Discussion: For most EPAs high levels of entrustment were associated with high ratings for selected FOCs. The results are in alignment with the assumption that each EPA encompasses a different set of FOCs.

Conclusions: In our simulated workplace-based assessment, entrustment decisions for EPAs reflect the FOCs observed in a trainee. Thus, assessment of FOCs alongside with EPA ratings could add to the understanding of factors contributing to entrustment decisions.     

Entrustable professional activities (EPAs) for teachers in medical education: Has the time come?

On a daily basis, patients put their trust in the healthcare system for safe and high-quality healthcare. However, what evidence do we have as an educational community that our supervising faculty members are competent to fulfill this responsibility? Few, if any, requirements exist for faculty members to have continuous professional development in the field of medical education. Many faculty “love to teach”, however, this love of teaching does not make them competent to teach or assess the competence of trainees whom they supervise. Faculty members who have a significant role as a teacher in the clinical setting should be assessed with regards to their baseline competence in applicable teaching EPAs. When competence is reached, an entrustment decision can be made. Once proficient or expert, a statement of awarded responsibility (STAR) may be granted. The time has come to reach beyond the “standards” of the old adage “see one, do one, teach one” in medical education. In this personal view, the authors outline an argument for and list the potential benefits for teachers, learners, and patients when we assess clinical teachers using EPAs within a competency-based medical education framework.     

Flexibility in individualized,competency-based workplace curricula with EPAs: Analyzing four cohorts of physician assistants in training

Background: Entrustable professional activities (EPAs) were introduced as a principle for individualized physician assistant (PA) workplace curricula at the University of Applied Sciences (UAS) Utrecht in 2008. We studied how the focus on EPAs served the competency-based flexibility intention of the program.

Methods: We analyzed data of those 119 students who enrolled in the program 2010 through 2013, and completed the program before April 2016. We analyzed the number of EPAs per student at start and end of the program, number changed during training and the reasons for change.

Results: Data of 101 students were suitable for evaluation. Excluded were 16 students ending the program prematurely and two with study delay.

Mean number of EPAs per student at the start was 6.8 (range 4–12) and at the end 6.6 (range 3–13). On average 1.5 EPAs were altered (range 0–13). Reasons included extension of the EPA package during training (n?=?10), lack of proficiency at planned moments of summative entrustment decisions (n?=?9) and procedures not being suitable for PAs at closer look (n?=?6). All changes resulted in a curriculum meeting the school’s standards for graduation.

Conclusions: The flexibility of the EPA concept enabled changes in the individualized curriculum of students, according to the intended competency-based nature of the educational program.     

Toward a research agenda for competency-based medical education

Competency-based medical education (CBME) is both an educational philosophy and an approach to educational design. CBME has already had a broad impact on medical schools, residency programs, and continuing professional development in health professions around the world. As the CBME movement evolves and CBME programs are implemented, a wide range of emerging research questions will warrant scholarly examination. In this paper, we describe a proposed CBME research agenda developed by the International CBME Collaborators. The resulting framework includes questions about the meaning of key concepts of CBME and their implications for learners, faculty members, and institutional structures. Other research questions relate to the learning process, the meaning of entrustment decisions, fundamental measurement issues, and the nature and definition of standards. The exploration of these questions will help to solidify the theoretical foundation of CBME, but many issues related to implementation also need to be addressed. These pertain to, among other things, nurturing independent learning, assembling and using assessment results to make decisions about competence, structuring feedback, supporting remediation, and how best to evaluate the longer-term outcomes of CBME. High-quality research on these questions will require rigorous outcome measures with strong validity evidence. The complexity of CBME necessitates theoretical and methodological diversity. It also requires multi-institutional studies that examine effects at multiple levels, from the learner to the team, the institution, and the health care system. Such a framework of research questions can guide and facilitate scholarly discourse on the theoretical and practical body of knowledge related to competency-based health professions education.     

Assessment challenges in competency-based education: A case study in health professions education

There is a growing demand for health sciences faculty with formal training in education. Addressing this need, the University of Michigan Medical School created a Master in Health Professions Education (UM-MHPE). The UM-MHPE is a competency-based education (CBE) program targeting professionals. The program is individualized and adaptive to the learner’s situation using personal mentoring. Critical to CBE is an assessment process that accurately and reliably determines a learner’s competence in educational domains. The program’s assessment method has two principal components: an independent assessment committee and a learner repository. Learners submit evidence of competence that is evaluated by three independent assessors. The assessments are presented to an Assessment Committee who determines whether the submission provides evidence of competence. The learner receives feedback on the submission and, if needed, the actions needed to reach competency. During the program’s first year, six learners presented 10 submissions for review. Assessing learners in a competency-based program has created challenges; setting standards that are not readily quantifiable is difficult. However, we argue it is a more genuine form of assessment and that this process could be adapted for use within most competency-based formats. While our approach is demanding, we document practical learning outcomes that assess competence.     

A model for programmatic assessment fit for purpose

We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory.     

Ethical issues in the managed care setting: a new curriculum for primary care physicians

Several challenging ethical issues have been associated with the shift to managed healthcare in the United States. Our objective was to develop, implement, and evaluate a curriculum designed to help physicians identify and examine ethical issues encountered in the managed care setting. The curriculum was developed during a year-long workshop at Johns Hopkins Bayview Medical Center. The content of the curriculum was established through literature review, focus group discussions with physicians, and a needs assessment of targeted learners (primary care physicians practicing in managed care settings). Some of the key issues addressed in the curriculum include: changing professional responsibilities of physicians; fair use of resources; and threats to the doctor-patient relationship as a consequence of the new healthcare delivery system. The 7.5-h curriculum was taught over five sessions using varied teaching methods. Evaluations demonstrated that the curriculum was successful in increasing learner awareness of ethical issues confronted in the managed care environment and improved learner knowledge in these areas. The physician-learners reported that this educational experience would change their teaching of medical students and residents. After completing the curriculum, learners felt that they were at least somewhat better able to cope with ethical challenges encountered in the managed care setting. Future research might examine whether such a curriculum could positively affect physician behavior or enhance physician satisfaction with the managed care setting.     

Using language of entrustable professional activities to define learning objectives of radiology clerkship: A modified Delphi study

Abstract

Background: A standard undergraduate radiology education is essential to prepare graduates for multidisciplinary clinical practice yet the literature lacks clear guidelines or consensus about the learning objectives of an optimal radiology clerkship.

Aim: To define a competency-based framework for undergraduate radiology education by using language of Entrustable Professional Activities (EPAs).

Methods: A modified Delphi method with three iterative rounds was used as an expert consensus approach. An online questionnaire with Likert scale was formulated incorporating EPAs and their components (competencies, assessment strategies, and supervision level) and distributed to 45 consultant radiologists following pilot study. Items reaching consensus were accepted and rest were resent in round 2. In round 3, a dichotomous scale was used for final approval and to see response stability.

Results: A final set of six EPAs with 87 competencies and respective assessment strategies, all aiming for ‘level 3a’ of supervision was identified. These include recommending cost effective appropriate imaging tests for common pathologies, obtaining informed consent for diagnostic contrast studies, basic interpretation and communication of common pathologies/emergencies on radiographs (chest, abdominal, and skeletal) and on CT brain.

Conclusion: This EPA framework for radiology clerkship is a first step towards a competency-based approach to undergraduate radiology training and assessment.     

Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales

Abstract

Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.

Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017–18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.

Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.

Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.     

Twelve tips to promote successful development of a learner performance dashboard within a medical education program

Abstract

Easily accessible and interpretable performance data constitute critical feedback for learners that facilitate informed self-assessment and learning planning. To provide this feedback, there has been a proliferation of educational dashboards in recent years. An educational (learner) dashboard systematically delivers timely and continuous feedback on performance and can provide easily visualized and interpreted performance data. In this paper, we provide practical tips for developing a functional, user-friendly individual learner performance dashboard and literature review of dashboard development, assessment theory, and users’ perspectives. Considering key design principles and maximizing current technological advances in data visualization techniques can increase dashboard utility and enhance the user experience. By bridging current technology with assessment strategies that support learning, educators can continue to improve the field of learning analytics and design of information management tools such as dashboards in support of improved learning outcomes.     

Using entrustable professional activities (EPAs) to assess teaching competence and transfer of training: A personal view

Faculty development (FD) activities aim to improve teaching competencies. Success of these activities is conditioned with the ability of participants to transfer the learned competencies into their teaching practices. Unfortunately, evaluation of the effectiveness of FD rely mostly upon self-reported or verbal feedback, without valid evaluation of their progress in teaching performance. This shortcoming may be attributed to the unavailability of a systematic assessment system to evaluate participants’ performance in the workplace. In this article, we advocate how the concept and principles of entrustable professional activities (EPAs) can be used in assessment of teaching competencies and ensuring the transfer of training to the workplace.     

The adaptive medical curriculum: A model for continuous improvement

Abstract

This paper describes the medical curriculum designed to foster adaptive expertise. Engaging in the formal and informal curriculum, students learn to achieve desired outcomes in novel situations, perform comfortably with uncertainty, and are often recognized for creative problem-solving. Students learn by asking and answering their own and others’ questions. They readily operate at the metacognitive level, anticipating events, self-monitoring, and checking decisions and emotions. A key function of the reflective process is to identify gaps or shortcomings in the thinking process. The adaptive learner shifts into reflective thinking when confronted with complex contextual and situational demands. We are only beginning to understand how to create educational pathways to foster adaptive learning. An essential focus is the adaptive teacher who frames learning and assessment around predictive analytics, reflective spaced practice, and authentic learning material. To be effective, the teacher must engage the learner outside the formal classroom in the parallel curriculum. A major premise is that learning occurs individually and together with peers, teachers, and team members in multiple contexts. During the learning process, the learner readily operates at the metacognitive level, anticipating behavior, self-monitoring and assessing, and checking theirs and others decisions and emotions. The adaptive medical curriculum provides the pathway for such learning.     

Ventilators,Guidelines, Judgment,and Trust

Covid-19 confronts us with tragic choices, in which every option is unacceptable. On the New York State Task Force on Life and the Law, I worked on guidelines for such situations. We did not envision the scale or character of Covid-19. To minimize fear that the decisions made in these situations might be unfair, we all must know what guidelines or mandates inform them. Only with transparency about how decisions will be made, by whom, and according to what requirements can we have confidence that fairness prevails. We now face many related questions about process, goals, leadership, and trust. For example, how might ethical guidelines evolve as scientific understanding advances? Should guidelines vary with different venues? And if, as I have argued, judgment is necessary even with the best of guidelines, how can we prepare clinicians to make good judgments? Are there implications for better training of personnel in nonemergency times for what they might face in the worst of times?     

Meaningful feedback through a sociocultural lens

Abstract

This AMEE guide provides a framework and practical strategies for teachers, learners and institutions to promote meaningful feedback conversations that emphasise performance improvement and professional growth. Recommended strategies are based on recent feedback research and literature, which emphasise the sociocultural nature of these complex interactions. We use key concepts from three theories as the underpinnings of the recommended strategies: sociocultural, politeness and self-determination theories. We view the content and impact of feedback conversations through the perspective of learners, teachers and institutions, always focussing on learner growth. The guide emphasises the role of teachers in forming educational alliances with their learners, setting a safe learning climate, fostering self-awareness about their performance, engaging with learners in informed self-assessment and reflection, and co-creating the learning environment and learning opportunities with their learners. We highlight the role of institutions in enhancing the feedback culture by encouraging a growth mind-set and a learning goal-orientation. Practical advice is provided on techniques and strategies that can be used and applied by learners, teachers and institutions to effectively foster all these elements. Finally, we highlight throughout the critical importance of congruence between the three levels of culture: unwritten values, espoused values and day to day behaviours.     

How do clinical competency committees use different sources of data to assess residents’ performance on the internal medicine milestones?A mixed methods pilot study

Purpose: This study examines how Clinical Competency Committees (CCCs) synthesize assessment data to make judgments about residents’ clinical performances.

Methods: Between 2014 and 2015, after four six-month reporting periods to the Accreditation Council for Graduate Medical Education (ACGME), 7 of 16 CCC faculty at Rush University Medical Center completed questionnaires focused on their perspectives about rating residents on their achievement of the milestones and participated in a focus group. Qualitative data were analyzed using grounded theory. Milestones ratings for two six-month ACGME reporting cycles (n?=?100 categorical residents) were also analyzed.

Results: CCC members weighted resident rotation ratings highest (weight?=?37%), followed by faculty rotation comments (weight?=?27%) and personal experience with residents (weight?=?14%) for making judgments about learner’s milestone levels. Three assessment issues were identified from qualitative analyses: (1) “design issues” (e.g. problems with available data or lack thereof); (2) “synthesis issues” (e.g. factors influencing ratings and decision-making processes) and (3) “impact issues” (e.g. how CCC generated milestones ratings are used).

Conclusions: Identifying factors that affect assessment at all stages of the CCC process can contribute to improving assessment systems, including support for faculty development for CCCs. Recognizing challenges in synthesizing first and second-hand assessment data is an important step in understanding the CCC decision-making process.     

Development of core entrustable professional activities linked to a competency-based veterinary education framework

Abstract

Purpose: Despite the adoption of competency-based education in some veterinary schools over the past 15?years, only recently has a concerted effort been directed toward this in veterinary education internationally.

Methods: In 2015, educational leaders from the Association of American Veterinary Medical Colleges (AAVMC) member schools came together with a strong call to action to create shared tools for clinical competency assessment.

Results: This resulted in the formation of the AAVMC Competency-Based Veterinary Education (CBVE) Working Group, which then embarked on the creation of a shared competency framework and the development of eight core entrustable professional activities (EPAs) linked to this framework.

Conclusions: This paper will report on the development of these EPAs and their integration with the concurrently-developed CBVE Framework.