Medical students as feedback assessors in a faculty development program: Implications for the future

Abstract

Background: Little is known about simulated students’ ability in assessing feedback received in Objective Structured Teaching Encounters (OSTEs). We aimed to assess to which extent students’ perceptions matched objective analysis regarding quality of received feedback, to explore what elements of feedback they emphasized and what they learned about feedback.

Methods: In this mixed-method study, 43 medical students participated as simulated residents in five OSTEs at Geneva University Hospitals. They assessed quality of feedback from faculty using a 15-item questionnaire and gave written/oral comments. Videotaped feedbacks were assessed using an 18-item feedback scale. During four focus groups, 25 students were asked about what they learned as feedback assessors.

Results: 453 students’ questionnaires and feedback scale were compared. Correlations were moderate for stimulating self-assessment (0.48), giving a balanced feedback (0.44), checking understanding (0.47) or planning (0.43). Students’ feedback emphasized elements such as faculty’s empathy or ability to give concrete advice. They reported that being a feedback assessor helped them to realize importance of making the learner active and that giving effective feedback required structure and skills.

Conclusion: Medical students may identify quality of feedback. Involving them in OSTEs could be interesting to train them to become valid raters of supervisors’ teaching skills.     

Taking stock of what is known about faculty development in competency-based medical education: A scoping review paper

Abstract

Purpose: The primary objective was to inventory what is currently known about faculty development (FD) for competency-based medical educations (CBME) and identify gaps in the literature.

Methods: A scoping review methodology was employed. Inclusion criteria for article selection were established with two reviewers completing a full-text analysis. Quality checks were included, along with iterative consultation on data collection and consensus decision making via a grounded theory approach.

Results: The review identified 19 articles published between 2009 and 2018. Most articles (N?=?15) offered suggestions as to what should happen with FD in CBME, but few (N?=?4) adopted an experimental design. Six main themes were identified with three main features of FD noted across themes: (1) The importance of direct and timely feedback to faculty members on their teaching and assessment skills. (2) The role of establishing shared mental models for CBME curricula. (3) That FD is thought of longitudinally, not as a one-time bolus.

Conclusion: This work illustrates that there is limited, high quality research in FD for CBME. Future FD activities should consider employing a longitudinal and multi-modal program format that includes feedback for the faculty participants on their teaching and assessments skills, including the development of faculty coaching skills.     

A systematic review of faculty development initiatives designed to enhance teaching effectiveness: A 10-year update: BEME Guide No. 40

Background: This review, which focused on faculty development initiatives designed to improve teaching effectiveness, synthesized findings related to intervention types, study characteristics, individual and organizational outcomes, key features, and community building.

Methods: This review included 111 studies (between 2002 and 2012) that met the review criteria.

Findings: Overall satisfaction with faculty development programs was high. Participants reported increased confidence, enthusiasm, and awareness of effective educational practices. Gains in knowledge and skills, and self-reported changes in teaching behaviors, were frequently noted. Observed behavior changes included enhanced teaching practices, new educational initiatives, new leadership positions, and increased academic output. Organizational changes were infrequently explored. Key features included evidence-informed educational design, relevant content, experiential learning, feedback and reflection, educational projects, intentional community building, longitudinal program design, and institutional support.

Conclusion: This review holds implications for practice and research. Moving forward, we should build on current success, broaden the focus beyond individual teaching effectiveness, develop programs that extend over time, promote workplace learning, foster community development, and secure institutional support. We should also embed studies in a theoretical framework, conduct more qualitative and mixed methods studies, assess behavioral and organizational change, evaluate transfer to practice, analyse key features, and explore the role of faculty development within the larger organizational context.     

Communities of teaching practice in the workplace: Evaluation of a faculty development programme

Background: The focus of faculty development (FD) has recently shifted from individual and formal learning to formal and informal learning by a team of teachers in the workplace where the teaching is actually effected. This study aimed to evaluate the impact of a faculty development programme on teachers' educational workplace environment.

Methods: We invited 23 teachers, who had successfully completed a University Teaching Qualification (UTQ) programme, to evaluate the faculty development programme and participate in focus group discussions. This UTQ programme spanned one year and covered 185 hours of formal and informal learning and training activities and formal coaching.

Results: After having obtained their UTQ, teachers reported that coaching enhances reflection and feedback, to participate more frequently in educational networks, which enhances consultation among teachers, increased awareness of organizational educational policies and more confidence in fulfilling educational tasks and activities.

Conclusion: The evaluation of the UTQ programme demonstrated to enhance the development of a community of teachers at the workplace who share a passion for education and provide each other with support and feedback, which triggered a change in culture enhancing improvement of education. However, this did not hold for all teachers. Inhibiting factors hold sway, such as a prevailing commitment to research over education in some departments and a lack of interest in education by some department chairs.     

Mutual mentoring: Effect on faculty career achievements and experiences

Abstract

Introduction: In an ever-changing academic environment, the traditional model of one senior mentor is no longer sufficient to provide faculty with the interdisciplinary perspective needed for success. We adapted, implemented and examined an interschool/interprofessional Mutual Mentoring Program.

Methods: Participants were expected to expand their mentoring networks, formulate career goals and achievement plans, and, achieve concrete career goal(s). Participants were offered a 1-year mentoring network grant. To examine short-term outcomes, we collected participants’ lists of contacts and accomplished goals, and asked for immediate feedback on the program. To examine intermediate outcomes, we conducted 30-min interviews with participants after 1 year of participation. We used quantitative and qualitative approaches to analyze the data from four program cohorts.

Results: Thirty-seven faculty participated throughout the four academic years. Participants reported 260 contacts made, and achieved goals related to academic products and outcomes. Eleven participants were interviewed and highlighted professional achievements, the power of the established networks, gained career resources, and the accomplishment of significant personal changes.

Conclusions: This approach provides the type of multiple mentoring support that is necessary for today’s academic climate for faculty to obtain their career goals, to demonstrate the institutional commitment to faculty success and to provide avenues for interdisciplinary connections.     

Scholar outcomes for dental internship research program in Saudi Arabia: A qualitative evaluation

Abstract

Objective: To explore the potential, challenges and needs for internship research activities in achieving scholar outcomes among graduates.

Methods: A qualitative general needs assessment and evaluation of an internship research program was conducted at King Abdulaziz University, Faculty of Dentistry (KAUFD), KSA, from December 2014 to February 2015 using focus groups and interviews. The participants included: administrates, faculty, and internship students. Data were transcribed and analyzed following the grounded theory.

Results: The participants were two administrative personnel, 21 faculty members, and 16 internship students. Results were clustered around five main domains; curriculum design, faculty, students, administrative, and institutional domain. Reported potentials included: a multi-faceted educational intervention approach, and building evidence-based skills and inquiry minds among graduates. Time, load, and incentives were major challenges reported by faculty. Interesting and achievable research topics were major challenges reported by students. Areas that needed development included: equipped research personnel, aligned administrative and institutional support, faculty skills, students’ knowledge and skills, aligned curriculum, and clear program goals, objectives, and outcomes.

Conclusion: Curriculum design, faculty and students’ skills; as well as administrative and institutional support were found to play major roles in the success of the current internship research program at KAUFD.     

Twelve tips for developing and implementing curriculum in dedicated ‘collaborative classrooms’

Abstract

Many health professional schools may be investing time and resources on dedicated educational spaces intended to promote collaborative learning. Alone, innovative physical space or technologies are not sufficient to ensure success in this. Lesson plans informed by collaborative praxis, individual motivation, faculty development, learner feedback, and team interactions also play a necessary and substantial role. We have used faculty observations, quantitative and qualitative student evaluation data, and the existing educational literature to provide twelve tips on leveraging curricular content, activity setup, physical space, learner behavior, and faculty facilitation to make the most of collaborative learning spaces.     

Feedback mapping – The curricular cornerstone of an “educational alliance”

Purpose: The “educational alliance” concept articulates a collaborative framework to facilitate effective feedback through transparency of opportunities aligned with learner stage and intended educational outcomes. Using this framework, we evaluated feedback across a 5 year undergraduate medical program to support embedding a successful learner – teacher “educational alliance”.

Method: A comprehensive mapping exercise used an iterative action research process of source documentary analysis, consultations with key curriculum stakeholders and qualitative analysis. The “educational alliance” model provided a critical lens through which to ensure feedback opportunities aligned with intended learning outcomes and developmental progression.

Results: Key information about the type, frequency and timing of feedback opportunities was identified within 188 curricular components. The purpose and intended learning outcomes were mapped, aligning with the stage of learning and with an expectation of supporting learners’ capacity for self-regulation. This focus providing clear articulation of feedback opportunities supported the longitudinal developmental curricular review, and facilitated enhanced awareness of dialogic feedback within the “educational alliance”.

Conclusions: Explicit alignment of learning intentions between learner and educator is key to forming a successful “educational alliance”. The feedback map provides clarity ensuring mutual understanding of intended learning outcomes. The iterative process additionally certified feedback aligned with maturing learner developmental needs across the program.     

Assessing learning in the adaptive curriculum

Abstract

Background: Medical education is a dynamic process that will continuously evolve to respond to changes in the foundations of medicine, the clinical practice of medicine and in health systems science.

Purpose: In this paper, we review how assessing learning in such a dynamic environment requires comprehensive flexible and adaptable methodological approaches designed to assess knowledge attainment and transfer, clinical skills/competency development, and ethical/professional behavior. Adaptive assessments should measure the learner’s ability to observe where changes in health care delivery are needed and how to implement them. Balancing formative and summative assessments will promote reflective learning so that each student will reach her/his highest potential. From the programmatic perspective, measuring the design and delivery of instruction in relation to students? efforts to achieve competency will improve learning and foster continuous professional development of faculty and advance the science of learning.

Approach: We describe how two medical schools are approaching adaptive assessment, including using portfolio systems that encompass teaching and learning experiences while offering real-time longitudinal tracking of digital data toward improving learning and provide curricula continuous improvement cycles. Using latest technologies, portfolios produce actionable data displays with precise guidance for learning and program development.     

What do I do? Developing a competency inventory for postgraduate (residency) program directors

Background: Few new Residency Program Directors (PD) are formally trained for the demands and responsibilities of the leadership aspect of their role. Currently, there are no comprehensive frameworks that describe specific leadership competencies that can inform PD self-reflection or faculty development.

Methods: The authors developed a Postgraduate Program Director Competency Inventory (PPDCI) in order to frame the performance of PDs for a multisource feedback (MSF) program. The development of the PPDCI occurred in five phases which involved: development of an initial inventory, implementation of a key informant survey of national opinion leaders, execution of a validity survey with postgraduate education leaders and committee members and implementation of a further refined inventory with 17 PD and 147 raters as part of a pilot MSF program.

Outcomes: Five distinct domains of leadership competence were identified which included: Communication and relationship management, leadership, professionalism and self-management, environmental engagement, and management skills and knowledge. The content validity of the PPDCI was endorsed by 85% of the key informants. The validity survey indicated strong endorsement of the PPDCI domains and recognition of its utility for both orientation of new PD as well as a frame for self-assessment. The pilot MSF program yielded a further refined and reduced inventory of 26 items of competence as well as recommendations for its utility.

Conclusions: Use of this leadership inventory has the potential to ensure effective leadership of postgraduate programs.     

Teaching,reflecting, and learning: The value of an intercalated medical education programme

Abstract

Objectives: Teaching programmes within medicine focus primarily on pathways of a shorter length with little regard to teaching lasting longer than a month. This study is different from other studies as it examines the benefits of a nine month-long medical education degree programme and its impact beyond graduation. This study set out to explore the impact of a medical education intercalated degree programme for its graduates and their careers.

Methods: A small scale, exploratory qualitative case study was conducted with 10 graduates of an intercalated degree programme.

Results: The findings highlight the longer term value of an intercalated degree programme with particular emphasis on academic and personal skills; research and teaching skills; independence and confidence; its impact on future practice and the notion that ‘student as teacher’ programmes are a valuable asset to medical education as a whole. Participants advocated more teaching opportunities as a core longitudinal teaching component in preparation for the teaching responsibilities in their working lives.

Conclusions: The programme enables the development of a range of academic and personal skills, with particular emphasis on research and teaching skills, independence and confidence.     

A holistic model for evaluating the impact of individual technology-enhanced learning resources

Background: The use of technology within education has now crossed the Rubicon; student expectations, the increasing availability of both hardware and software and the push to fully blended learning environments mean that educational institutions cannot afford to turn their backs on technology-enhanced learning (TEL). The ability to meaningfully evaluate the impact of TEL resources nevertheless remains problematic.

Aims: This paper aims to establish a robust means of evaluating individual resources and meaningfully measure their impact upon learning within the context of the program in which they are used.

Methods: Based upon the experience of developing and evaluating a range of mobile and desktop based TEL resources, this paper outlines a new four-stage evaluation process, taking into account learner satisfaction, learner gain, and the impact of a resource on both the individual and the institution in which it has been adapted.

Results: A new multi-level model of TEL resource evaluation is proposed, which includes a preliminary evaluation of need, learner satisfaction and gain, learner impact and institutional impact. Each of these levels are discussed in detail, and in relation to existing TEL evaluation frameworks.

Conclusions: This paper details a holistic, meaningful evaluation model for individual TEL resources within the specific context in which they are used. It is proposed that this model is adopted to ensure that TEL resources are evaluated in a more meaningful and robust manner than is currently undertaken.     

Attending rounds: What do the all-star teachers do?

Aim: To examine differences in the types of teaching activities performed during rounds between the most effective and least effective inpatient teaching attendings.

Methods: Participants included 56 attending physicians supervising 279 trainees. Trained observers accompanied teams during rounds and recorded the frequencies of educational activities that occurred. Students and residents then rated their satisfaction with the teaching on rounds.

Results: Attending physicians with the highest learner satisfaction scores performed significantly more teaching activities per patient than attending physicians who were rated as average or less-effective (2.1 vs. 1.4 vs. 1.5; p?=?.03). There were significant differences in the frequencies of 3 out of the 9 specific teaching activities observed, including answering specific patient-care related questions (77% vs. 66% vs. 47%; p?=?.003), teaching on learner chosen topics (8% vs. 2% vs. 2%; p?=?.02), and providing feedback (31% vs. 10% vs. 0.1%; p?=?.001).

Conclusions: Specific categories of teaching activities—patient-specific teaching, teaching on learner-identified topics, and providing real-time feedback—are performed more frequently by the highest-rated attending physicians, which can guide faculty development.     

A multi-source feedback tool for measuring a subset of Pediatrics Milestones

Background: The Pediatrics Milestones Assessment Pilot employed a new multisource feedback (MSF) instrument to assess nine Pediatrics Milestones among interns and subinterns in the inpatient context.

Objective: To report validity evidence for the MSF tool for informing milestone classification decisions.

Methods: We obtained MSF instruments by different raters per learner per rotation. We present evidence for validity based on the unified validity framework.

Results: One hundred and ninety two interns and 41 subinterns at 18 Pediatrics residency programs received a total of 1084 MSF forms from faculty (40%), senior residents (34%), nurses (22%), and other staff (4%). Variance in ratings was associated primarily with rater (32%) and learner (22%). The milestone factor structure fit data better than simpler structures. In domains except professionalism, ratings by nurses were significantly lower than those by faculty and ratings by other staff were significantly higher. Ratings were higher when the rater observed the learner for longer periods and had a positive global opinion of the learner. Ratings of interns and subinterns did not differ, except for ratings by senior residents. MSF-based scales correlated with summative milestone scores.

Conclusion: We obtain moderately reliable MSF ratings of interns and subinterns in the inpatient context to inform some milestone assignments.     

Faculty development: From rubies to oak

Abstract

Introduction: Faculty development has played a significant role in health professions education over the last 40?years. The goal of this perspective is to present a portrait of faculty development in Medical Teacher since its inception and to highlight emerging trends moving forward.

Method: All issues of Medical Teacher were reviewed, using the search terms faculty development, staff development, professional development, or in-service training for faculty. The search yielded 286 results of which 145 focused specifically on faculty development initiatives, reviews, or frameworks.

Findings: This review demonstrated a significant growth in publications related to faculty development in Medical Teacher over the last 40?years, with a primary focus on teaching improvement and traditional approaches to faculty development, including workshops, short courses and other structured, group activities. The international nature of faculty development was also highlighted.

Recommendations: Moving forward, it is suggested that we: broaden the scope of faculty development from teaching to academic development; expand our approaches to faculty development, to include peer coaching, workplace learning and communities of practice; utilize a competency-based framework to guide the development of faculty development curricula; support teachers’ professional identities through faculty development; focus on organizational development and change; and rigorously promote research and scholarship in faculty development.     

Clinical teaching performance improvement of faculty in residency training: A prospective cohort study

Purpose: The purpose of this study is to investigate how aspects of a teaching performance evaluation system may affect faculty’s teaching performance improvement as perceived by residents over time.

Methods: Prospective multicenter cohort study conducted in The Netherlands between 1 September 2008 and 1 February 2013. Nine hundred and one residents and 1068 faculty of 65 teaching programs in 16 hospitals were invited to annually (self-) evaluate teaching performance using the validated, specialty-specific System for Evaluation of Teaching Qualities (SETQ). We used multivariable adjusted generalized estimating equations to analyze the effects of (i) residents’ numerical feedback, (ii) narrative feedback, and (iii) faculty’s participation in self-evaluation on residents’ perception of faculty’s teaching performance improvement.

Results: The average response rate over three years was 69% for faculty and 81% for residents. Higher numerical feedback scores were associated with residents rating faculty as having improved their teaching performance one year following the first measurement (regression coefficient, b: 0.077; 95% CI: 0.002–0.151; p?=?0.045), but not after the second wave of receiving feedback and evaluating improvement. Receiving more suggestions for improvement was associated with improved teaching performance in subsequent years.

Conclusions: Evaluation systems on clinical teaching performance appear helpful in enhancing teaching performance in residency training programs. High performing teachers also appear to improve in the perception of the residents.     

Eight-year outcomes of a competency-based residency training program in orthopedic surgery

Abstract

Background: The Division of Orthopaedic Surgery at the University of Toronto implemented a pilot residency training program that used a competency-based framework in July of 2009. The competency-based curriculum (CBC) deployed an innovative, modularized approach that dramatically intensified both the structured learning elements and the assessment processes.

Methods: This paper discusses the initial curriculum design of the CBC pilot program; the refinement of the curriculum using curriculum mapping that allowed for efficiencies in educational delivery; details of evaluating resident competence; feedback from external reviews by accrediting bodies; and trainee and program outcomes for the first eight years of the program’s implementation.

Results: Feedback from the residents, the faculty, and the postgraduate residency training accreditation bodies on the CBC has been positive and suggests that the essential framework of the program may provide a valuable tool to other programs that are contemplating embarking on transition to competency-based education.

Conclusions: While the goal of the program was not to shorten training per se, efficiencies gained through a modular, competency-based program have resulted in shortened time to completion of residency training for some learners.