Direct Observation of Residents in the Emergency Department: A Structured Educational Program

Objectives:  The objective was to describe the implementation of a program of structured direct observation of emergency medicine (EM) residents during clinical shifts in the emergency department (ED).
Methods:  The authors developed a program in which an observer spent 4 to 5 hours with each resident, without intervening in the clinical encounters. A structured data form was developed to document the resident's performance in a number of defined clinical areas relevant to patient care and mastery of the core competencies. Individual strengths and weaknesses were noted, and the observer provided directed feedback at the end of the session.
Results:  Over an 18-month period, 32 EM residents were observed during their ED shifts. The sessions not only provided specific information on individual residents' performances, but also identified areas where the residency program curriculum could be enhanced and provided a means of assessing mastery of the core competencies. In addition, the program provided an opportunity to give detailed and timely directed feedback to residents. Both residents and attending staff found the sessions acceptable and useful.
Conclusions:  Implementation of a structured direct observation program was feasible and well received and provided insight into the strengths and weaknesses of residents both individually and as a group.     

Assessing clinical competence in physical medicine & rehabilitation residency programs

OBJECTIVE: Evaluation of resident clinical competence is a complex task. A multimodal approach is necessary to capture all of the dimensions of competence. Recent guidelines from the Accreditation Council for Graduate Medical Education delineate six general competencies that physicians should posses. Application of these guidelines presents challenges to residency program directors in defining educational experiences and evaluation methods. DESIGN: We surveyed 81 physical medicine and rehabilitation program directors regarding assessment tools used in their programs. Seventy-five percent responded. The most frequently used assessment tools included: In-training self-assessment examinations, faculty evaluations, direct observation, and conference participation. Program directors assigned the highest values to direct observation, faculty evaluations, self-assessment examinations, and oral examinations. RESULTS: Of the general competencies, more than 90% of program directors believed they did an adequate job rating dimensions of patient care, medical knowledge, professionalism, and communication skills. Approximately one-third, however, thought they did a less than fair job rating practice-based learning and improvement and systems-based practice. The majority of programs reported that they were able to identify a resident with difficulties during the first year of training, 44% within the first 6 months. Program directors reported that their residents spend a significant amount of their time with nurses and therapists during their inpatient rotations; however, this was not reflected in their evaluation practices, in which only one-fourth of programs reported the use of nurses and therapists in evaluating residents. CONCLUSIONS: Survey results indicate that physical medicine and rehabilitation program directors apply a variety of assessment tools in evaluating resident clinical competence. Although perceptions about the relative value of these tools vary, most programs report a high value to direct observation of residents by faculty. Of the six general competencies, program directors struggle the most with their evaluation of practice-based learning and improvement and systems-based practice.     

Predictive Value of Letters of Recommendation vs Questionnaires for Emergency Medicine Resident Performance

Objective: To evaluate the predictive value of standard letters of recommendation (LORs) vs preprinted questionnaires (PPQs) for resident performance at one emergency medicine (EM) residency program.
Methods: A retrospective association of LORs and PPQs with intraining resident performance ratings was done at one EM residency program. The residency application files of EM residents who completed the program were reviewed to locate files that had LORs and PPQs written by the same author. Seventeen resident files contained 32 LOR/PPQ pairs. These LORs and PPQs were submitted in a blinded fashion to 3 outside EM residency directors. Each LOR and PPQ was evaluated for the applicant's suitability for the speciality of EM, medical knowledge, procedural skills, interpersonal skills, motivation, and overall rank. The scores given by the outside reviewers were compared with resident performance ratings determined by 5 EM attending physicians who evaluated the residents along the same 6 dimensional ratings.
Results: Statistically, no differences were found between the LORs and PPQs in predicting resident performance.
Conclusions: PPQs may substitute for LORs in the evaluation of resident applicants.     

Off‐service Resident Education in the Emergency Department: Outline of a National Standardized Curriculum

Although many residency programs mandate at least one rotation in emergency medicine (EM), to the best of our knowledge, a standardized curriculum for emergency department (ED) rotations for “off‐service” residents has not been developed. As a result, the experiences of these residents in the ED tend to vary during their rotations. To design an off‐service EM curriculum, we adopted Kern’s six‐step approach to curriculum development as a conceptual framework. The resulting program encompasses clinical experience and didactic sessions through which residents are trained in core topics and skills. This knowledge will be applicable in the clinical settings in which residents will continue to train and ultimately practice their specialty. It is flexible enough to be applicable and implementable without being limited by resource availability or faculty strengths.     

8 The Observed Teaching Encounter: Providing Residents Feedback on Their Teaching Skills

Introduction:  Emergency medicine residents spend a significant portion of their time teaching junior residents and medical students in the clinical setting. Feedback is an integral component of any teaching curriculum, and therefore, feedback on residents' skill in teaching abilities is an essential part of their learning to teach. We have developed a structured method of providing feedback to senior residents on their teaching competence.
Methods:  Upcoming senior residents receive an 8-hour course on clinical teaching during their useful conference time. In our ED, attending faculty and senior "teaching" residents are matched with medical student learners. The Observed Teaching Encounter (OTE) is used during usual clinical ED shifts to reinforce concepts in teaching. During the OTE, the teaching resident is directly observed by a faculty physician while teaching a student learner. A checklist is completed by both the faculty member and the student learner in order to provide feedback to the teaching resident. Assessed skills correlate with teaching theory provided to residents in their didactic curriculum. Written formative comments are provided to the resident from faculty, as well.
Results:  Attending faculty, senior residents, and student learners have all provided positive feedback on the OTE. Assessment of residents' retention of knowledge on methodology of teaching is presently in progress as a tool to evaluate the efficacy of the OTE.     

临床病例解析培训对提高麻醉住院医师临床思维的作用分析

目的通过比较麻醉住院医师临床技能考核结果,说明麻醉临床病例解析评估的重要性,为进一步提高麻醉住院医师培训质量提供思路和手段。方法选取2011年至2013年北京友谊医院和北京天坛医院麻醉专业住院医师轮转培训25人,以临床病例解析分析要点为考核点,对其专业临床技能进行考核评分,对比不同培训时段的考核结果。结果第一年住院医师的得分比第二年第三年住院医师均低,不合格率也高于第二年第三年住院医师。住院医师参加的临床病例解析培训次数越多,成绩优良率也越高。结论住院医师的临床思维麻醉病例解析培训有利于提高住院医师的临床思维和技能,使学科培训更加高效化、系统化、规范化。     

Interobserver Variability among Faculty in Evaluations of Residents Clinical Skills

Objective: To describe interobserver variability among emergency medicine (EM) faculty when using global assessment (GA) rating scales and performance-based criterion (PBC) checklists to evaluate EM residents clinical skills during standardized patient (SP) encounters. Methods: Six EM residents were videotaped during encounters with SPs and subsequently evaluated by 38 EM faculty at four EM residency sites. There were two encounters in which a single SP presented with headache, two in which a second SP presented with chest pain, and two in which a third SP presented with abdominal pain, resulting in two parallel sets of three. Faculty used GA rating scales to evaluate history taking, physical examination, and interpersonal skills for the initial set of three cases. Each encounter in the second set was evaluated with complaint-specific PBC checklists developed by SAEM's National Consensus Group on Clinical Skills Task Force. Results: Standard deviations, computed for each score distribution, were generally similar across evaluation methods. None of the distributions deviated significantly from that of a Gaussian distribution, as indicated by the Kolmogorov-Smirnov goodness-of-fit test. On PBC checklists, 80% agreement among faculty observers was found for 74% of chest pain, 45% of headache, and 30% of abdominal pain items. Conclusions: When EM faculty evaluate clinical performance of EM residents during videotaped SP encounters, interobserver variabilities are similar, whether a PBC checklist or a GA rating scale is used.     

Emergency Medicine Resident Moonlighting: A Survey of Program Directors

OBJECTIVES: 1) To systematically describe emergency medicine (EM) program directors' perceptions of the benefits and risks of resident moonlighting. 2) To assess moonlighting policies of EM residencies, the degree of compliance with these policies, and the methods of dealing with residents who are out of compliance. METHODS: A written survey was mailed or hand-delivered to all allopathic and osteopathic EM residency directors in the United States in 1992-93. Incomplete and ambiguous surveys were completed by phone. RESULTS: There was a 96% response rate (113/118). The average EM resident clinical workweek ranged from 38 to 50 hours while the resident was assigned to ED rotations. Most (90%) of the program directors believe moonlighting interferes with residency duties to some degree. Few (10%) programs prohibit moonlighting altogether, although 44% limit moonlighting to an average of 41.5 hours per month. Program directors believe residents moonlight primarily for financial reasons. Most (60%) of the program directors believe moonlighting offers experience not available in the residency, primarily related to autonomous practice. Fifteen programs reported residents who had been sued for malpractice while moonlighting, with one program director named along with the resident. One third of program directors have penalized residents for abuse of moonlighting privileges. CONCLUSIONS: EM residency directors are concerned about the effect of moonlighting on resident education. The directors' concerns regarding litigation, excessive work hours, and interference with residency duties are balanced by a general acceptance of the financial need to supplement residency income.     

Survey of moonlighting practices and work requirements of emergency medicine residents

We conducted an anonymous moonlighting and academic practice survey of all emergency medicine residents enrolled in accredited programs during 1997. Expanding on previous work, this survey included specific details and practice trends of moonlighting emergency medicine residents and for comparison also included academic work requirements. The typical emergency residency program requires residents to work 204 hours monthly. However, the range of required work-hours is strikingly large (120-300). Half of emergency medicine residents moonlight. The typical moonlighting resident works as a solo emergency department practitioner in multiple facilities outside of residency-affiliated institutions. Moonlighting salaries generally double a resident's annual income and are used to pay off student loans and other debt. Residents with higher student debt are more likely to moonlight. Despite the fact that most residency programs restrict moonlighting, a majority of moonlighting residents have violated an Accreditation Council for Graduate Medical Education prohibition restricting work within one period of a regular residency-scheduled shift. Half of all residents surveyed, whether involved in moonlighting practice or not, would violate a ban on the practice. Residents universally felt that moonlighting enhanced residency performance and was a positive educational experience. Use of these data may aid in the development of formal guidelines regarding emergency medicine moonlighting practice.     

Graduate Medical Education Downsizing: Perceived Effects of Participating in the HCFA Demonstration Project in New York State

OBJECTIVE: Financial support for graduate medical education (GME) is shrinking nationally as Medicare cuts GME funds. Thirty-nine hospitals in New York State (NYS) voluntarily participated in a Health Care Financing Administration demonstration project (HCFADP)-the goal of which was to reduce total residency training positions by 4-5%/year over a five-year period, while increasing primary care positions. The objective of this study was to determine the effect of downsizing on emergency department (ED) staffing and emergency medicine (EM) residency training. METHODS: Structured interviews and surveys of NYS program directors (PDs) were conducted in October-December 1999. Simple frequencies are reported. RESULTS: One hundred percent of 17 PDs completed the interviews and seven of 12 participants in the HCFADP returned surveys. Twelve of 17 programs participated in HCFADP and two programs downsized outside HCFADP. Seven of 12 participants lost EM positions. Six of 12 programs were forced to exclude outside residents from rotating in their ED, leading to a need for one participating program and one non-participating program to find alternative sites for trauma. Five of 12 institutions provided resident staffing data, reporting a reduction in ED resident coverage in year 1 of the project of 9-40%. Programs compensated by increasing the number of shifts worked (4/12), increasing shift length (1/12), decreasing pediatric ED shifts (1/12), decreasing elective or research time (2/12), and decreasing off-service rotations (4/12). Six departments hired physician assistants or nurse practitioners, two hired faculty, and two hired resident moonlighters. Six of 12 programs withdrew from HCFADP and returned to previous resident numbers. Eight of 12 PDs thought that they had decreased time for clinical teaching. CONCLUSIONS: A 4-5% reduction in residency positions was associated with a marked reduction in ED resident staffing and EM residency curriculum changes.     

Residents as Members of Intern Selection Committees: Can They Partially Replace Faculty?

Background: Faculty involvement in the residency selection process is important, but increasing pressures on available faculty resources have made their ability to participate more difficult. Residents may be acceptable substitutions for some faculty in the selection process.

Purpose: To test 2 new interview models and explore whether residents could partially replace faculty by examining what role the interview itself plays in the overall assessment of an applicant, comparing faculty and resident ratings of the interview, and asking whether partially substituting residents for faculty had an impact on the match outcome.

Methods: Applicants to an internal medicine residency program were assigned to 1 of 3 interview models: faculty-faculty, resident-faculty, resident-faculty-faculty. The 12 interview days were randomly assigned to a model and all applicants on each day had the same model. Interviewers used an applicant assessment form to assign ratings to 6 components of the portfolio and a final score.

Results: For both residents and faculty, the final score was highly correlated with the formal interview component. Within-model analyses showed residents consistently gave more favorable scores than faculty interviewers. There was no impact of interview model on initial or final rank position. Similarly, there was no difference between models in their match "success" rates.

Conclusions: Residents can be successfully substituted for some faculty in the residency selection process. The use of residents does not impact the match results. Developing a small, committed group of interviewers should benefit both programs and applicants.     

Debriefing in the intensive care unit: a feedback tool to facilitate bedside teaching

OBJECTIVE: To develop an assessment tool for bedside teaching in the intensive care unit (ICU) that provides feedback to residents about their performance compared with clinical best practices. METHOD: We reviewed the literature on the assessment of resident clinical performance in critical care medicine and summarized the strengths and weaknesses of these assessments. Using debriefing after simulation as a model, we created five checklists for different situations encountered in the ICU--areas that encompass different Accreditation Council for Graduate Medical Education core competencies. Checklists were designed to incorporate clinical best practices as defined by the literature and institutional practices as defined by the critical care professionals working in our ICUs. Checklists were used at the beginning of the rotation to explicitly define our expectations to residents and were used during the rotation after a clinical encounter by the resident and supervising physician to review a resident's performance and to provide feedback to the resident on the accuracy of the resident's self-assessment of his or her performance. RESULTS: Five "best practice" checklists were developed: central catheter placement, consultation, family discussions, resuscitation of hemorrhagic shock, and resuscitation of septic shock. On average, residents completed 2.6 checklists per rotation. Use of the cards was fairly evenly distributed, with the exception of resuscitation of hemorrhagic shock, which occurs less frequently than the other encounters in the medical ICU. Those who used more debriefing cards had higher fellow and faculty evaluations. Residents felt that debriefing cards were a useful learning tool in the ICU. CONCLUSIONS: Debriefing sessions using checklists can be successfully implemented in ICU rotations. Checklists can be used to assess both resident performance and consistency of practice with respect to published standards of care in critical care medicine.     

Effect of an Integrated Public Health Curriculum in an Emergency Medicine Residency

Background: Emergency departments (EDs) serve as a central point of interaction between the public and the medical system. Emergency physicians need education in public health in order to optimize their clinical care and their ability to evaluate potential public health interventions in the ED. Methods: As part of the Centers for Disease Control and Prevention (CDC) and the Association of American Medical College's (AAMC) national initiative for "Regional Medicine-Public Health Education Centers-Graduate Medical Education", we designed and implemented a new public health curriculum for the emergency medicine residents. Over four sessions during regular didactic time, we used a modular approach to link a basic public health principle, such as environmental hazard assessment, to a relevant clinical topic, such as violent patients and ED safety. Each session emphasized resident involvement, including small group work and role-plays. Journal clubs and quality assurance projects supplemented the curriculum. We sought resident feedback through focus groups and anonymous online pre- and post-tests for each session. Assessment: Both before and after the curriculum, 76% of responders felt it was important for physicians to receive training in public health. The program appeared to have a positive effect on residents' comfort level with various public health topics, and felt the residency program had taught them the skills necessary to implement public health principles in clinical practice (23.8%, versus 11.5% before; p<0.05). Conclusions: Integration of public health principles into existing clinical curricula in emergency medicine may increase resident interest and knowledge. Combining public health and emergency medicine topics in regular didactic conferences facilitates public health education for residents.     

Role of simulators, educational programs, and nontechnical skills in anesthesia resident selection, education, and competency assessment

Theoretical knowledge for anesthesia residents requires learning from a variety of sources. Technical skills are important and simulators are being used in many centers. Anesthesia nontechnical skills, those are, cognitive, interpersonal, and decision making, may not be uniformly acquired during clinical training and may need to be specifically taught and evaluated. Clinical competency committees usually evaluate the performance of every resident on a 6-month basis; however, a more objective test should be used. Training is evolving and higher standards in this field should create safer anesthesiologists. Simulator-based education and testing and assessment of nontechnical skills should be a priority in anesthesia residency programs.     

Reliability of Faculty Clinical Evaluations of Non-Emergency Medicine Residents during Emergency Department Rotations

Objectives: To assess the reliability of faculty evaluations of non-emergency medicine (non-EM) residents during clinical ED rotations and to determine the effect that the “leniency” of grading by these evaluators had on the residents' final evaluations. Methods: A prospective, observational study of the evaluation patterns of EM faculty was performed in an academic ED (50, 000 visits yearly census). Each resident was evaluated on a daily basis by a board-certified or board-prepared emergency physician. The evaluation form rated 7 characteristics, but only the rating for overall clinical competence was used for data analysis. If an attending evaluated the same resident more than once, only the first evaluation was used to avoid bias from prior exposure. The scoring patterns of the evaluators, both individually and in groups, were analyzed using 1-way analysis of variance. Evaluator leniency was estimated using the mean evaluator score across all residents. Since each resident was evaluated by a different combination of evaluators, evaluator leniency for each resident was estimated from the mean leniency of the evaluators who specifically assessed that resident. Results: During the period of the study, 66 residents rotated through the ED, yielding a total of 401 evaluations. When the scoring patterns of individual evaluators were analyzed, a high degree of variability was found in the mean scores (range 5.23–8.09) and SDs (range 0.45–1.55) across evaluators; p = 0.0001. There was a moderate correlation between the mean overall competence score received by each resident and that resident's evaluators' leniency, r = 0.52; p = 0.0001. Conclusions: There is significant variability in the scoring patterns of individual evaluators. The evaluators in this study showed large variations in both leniency (as measured by their mean score) and range restriction (as measured by their SD). The differences in evaluator scoring leniency have a moderate correlation with the overall score received by the resident.