Direct Observation for Assessing Emergency Medicine Core Competencies: Interpersonal Skills

The American Board of Medical Specialties described six core competencies considered essential elements of medical practice: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. In response, the Accreditation Council for Graduate Medical Education (ACGME) mandated that all residency programs assess trainees for the newly defined core competencies. Despite the mandate for including these six competencies in residency training, neither a specific curriculum nor a method to assess the outlined objectives has been developed by the ACGME. Instead, it is up to individual residency programs to document how they plan to incorporate and assess the core competencies in their programs. This article describes the potential use of direct observation to assess resident performance in the interpersonal skills core competency.     

360-degree Feedback: Possibilities for Assessment of the ACGME Core Competencies for Emergency Medicine Residents

The Accreditation Council for Graduate Medical Education (ACGME) has challenged residency programs to provide documentation via outcomes assessment that all residents have successfully mastered the six core competencies. A variety of assessment "tools" has been identified by the ACGME for outcomes assessment determination. Although rarely cited in the medical literature, 360-degree feedback is currently in widespread use in the business sector. This tool provides timely, consolidated feedback from sources in the resident's sphere of influence (emergency medicine faculty, emergency medicine residents, off-service residents and faculty, nurses, ancillary personnel, patients, out-of-hospital care providers, and a self-assessment). This is a significant deviation from both the peer review process and the resident review process that almost exclusively use physicians as raters. Because of its relative lack of development, utilization, and validation as a method of resident assessment in graduate medical education, a great opportunity exists to develop the 360-degree feedback tool for resident assessment.     

An Evaluation of Emergency Medicine Resident Interaction Time with Faculty in Different Teaching Venues

OBJECTIVES: To measure actual emergency medicine (EM) resident interaction time with faculty and to investigate the potential to use direct observation as an assessment tool for the core competencies. By 2006 all EM residencies must implement resident assessment techniques of the six Accreditation Council for Graduate Medical Education core competencies. Emergency medicine educators recommend direct observation as the optimal evaluation tool for patient care, systems-based practice, interpersonal and communication skills, and professionalism. Continuous faculty presence in the emergency department (ED) is widely believed to facilitate direct observation as an assessment technique. METHODS: Observational study of EM resident-faculty interaction time during two-hour periods. Study venues included two EDs, two trauma services, inpatient medicine, adult and pediatric intensive care units (ICUs), and a pediatric outpatient clinic. Using a priori definitions, the authors categorized faculty-EM resident interaction time as direct observation of patient care, indirect patient care, or non-patient care activities, and calculated total faculty interaction time. Subjects were blinded to the nature of the study, and data gathering was encrypted. RESULTS: Two hundred seventy observation periods of two hours each were conducted, sampling 32 EMR1, 33 EMR2-3, 41 EM, and 38 non-EM faculty. The mean total faculty interaction time ranged from a high of 30% (95% CI = 20% to 41%) in the pediatric ICU to a low of 10% (95% CI = 3% to 16%) on internal medicine wards. Overall, EM faculty interaction time was 20% (95% CI = 18% to 22%). Direct observation by faculty ranged from a high of 6% for EMR2-3s in the critical care areas of the ED (95% CI = 3% to 9%) to a low of 1% (95% CI = 0% to 2%) on internal medicine wards. Overall ED direct observation time was 3.6% (95% CI = 2.6% to 4.7%). Emergency department direct observation did not vary within EM resident training level or by ED site. Direct observation varied by treatment area within the EDs, with the critical care areas being substantially higher (6%) than the noncritical care areas (1%). CONCLUSIONS: Faculty direct observation time of EM residents was low in all training venues studied. Direct observation was the highest in ED critical care areas and lowest on medicine ward rotations. Emergency medicine faculty involved simultaneously in routine ED teaching, supervision, and patient care rarely performed direct observation, despite their continuous physical presence. This finding suggests that alternative strategies may be required to assess core competencies through direct observation in the ED.     

Evaluation of Student Performance: Clinical and Professional Performance

In 1999, the Accreditation Council for Graduate Medical Education (ACGME) endorsed six general competencies for residents as part of an ongoing attempt to emphasize educational outcomes as part of residency program assessment and accreditation. Although the focus of these reforms has been on postgraduate training, the competencies are applicable to medical students who aspire to become excellent clinicians and can help guide the development of assessment tools to measure clinical and professional performance of students in the emergency department. This article reviews the definitions of clinical competence and briefly addresses the issues of evaluation tool validity, reliability, and feasibility as they relate to the assessment of medical students. Several assessment tools are outlined, with an eye to the ACGME competencies and the ACGME Toolbox of Assessment Methods. The pros and cons of global rating scales, direct observation, simulations, and oral examinations are reviewed. Multiple assessment tools are often necessary to provide a true evaluation of a student's clinical and professional skills. Their application in the setting of an emergency medicine rotation is described and discussed.     

Outcome Assessment in Emergency Medicine—A Beginning: Results of the Council of Emergency Medicine Residency Directors (CORD) Emergency Medicine Consensus Workgroup on Outcome Assessment

This article is designed to serve as a guide for emergency medicine (EM) educators seeking to comply with the measurement and reporting requirements for Phase 3 of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project. A consensus workshop held during the 2006 Council of Emergency Medicine Residency Directors (CORD) "Best Practices" conference identified specific measures for five of the six EM competencies—interpersonal communication skills, patient care, practice-based learning, professionalism, and systems-based practice (medical knowledge was excluded). The suggested measures described herein should allow for ease in data collection and applicability to multiple core competencies as program directors incorporate core competency outcome measurement into their EM residency training programs.     

Emergency Medicine Resident Moonlighting: A Survey of Program Directors

OBJECTIVES: 1) To systematically describe emergency medicine (EM) program directors' perceptions of the benefits and risks of resident moonlighting. 2) To assess moonlighting policies of EM residencies, the degree of compliance with these policies, and the methods of dealing with residents who are out of compliance. METHODS: A written survey was mailed or hand-delivered to all allopathic and osteopathic EM residency directors in the United States in 1992-93. Incomplete and ambiguous surveys were completed by phone. RESULTS: There was a 96% response rate (113/118). The average EM resident clinical workweek ranged from 38 to 50 hours while the resident was assigned to ED rotations. Most (90%) of the program directors believe moonlighting interferes with residency duties to some degree. Few (10%) programs prohibit moonlighting altogether, although 44% limit moonlighting to an average of 41.5 hours per month. Program directors believe residents moonlight primarily for financial reasons. Most (60%) of the program directors believe moonlighting offers experience not available in the residency, primarily related to autonomous practice. Fifteen programs reported residents who had been sued for malpractice while moonlighting, with one program director named along with the resident. One third of program directors have penalized residents for abuse of moonlighting privileges. CONCLUSIONS: EM residency directors are concerned about the effect of moonlighting on resident education. The directors' concerns regarding litigation, excessive work hours, and interference with residency duties are balanced by a general acceptance of the financial need to supplement residency income.     

A new view: tele-intensive care unit competencies

BACKGROUND Many hospitals have well-planned nursing competency assessment programs, but these are meant to measure competency in traditional bedside roles, not in tele-intensive care unit (tele-ICU) nurses practicing remotely. OBJECTIVE To determine whether current tele-ICU programs have a formal competency assessment program and to determine when and how competency of tele-ICU nurses is assessed. Method A 20-question survey was provided to a convenience sample of the 44 known tele-ICU programs nationally. RESULTS Of the surveys distributed, 75% were completed and returned. A formal competency assessment policy for assessing nurses' competency at the time of hire, during orientation, and ongoing was in place at the workplaces of 85% of respondents. The most common methods for competency validation were performance appraisal and observation, although peer review and self-assessment also were used. Respondents identified the following competencies as the highest priorities for defining tele-ICU nurse practice: effective listening, prioritization, collaboration, and effective use of tele-ICU application tools. CONCLUSION Although awaiting development of professional practice standards, many tele-ICU programs currently measure the competence of tele-ICU nurses through competency programs.     

A Novel Concept in Residency Education: Case-based Remediation

Background

Resident remediation is required for all residents who do not meet minimum standards in one or more of the Accreditation Council for Graduate Medical Education core competencies. The Council of Residency Directors in Emergency Medicine Remediation Taskforce identified the need for case-based examples of remediation efforts.

Objectives

1) To describe a complicated resident remediation case and employ consensus panel evaluation of the process. 2) To discuss the available assessment tools (including neuropsychologic/medical testing), due process, documentation, reassessment, and relevant barriers to implementation for this and other resident remediations.

Discussion

Details of a remediation case were altered to protect resident confidentiality, and then presented to a multidisciplinary group of program directors. The case details, action plan, and course were submitted and the remediation process, action plan, and course are assessed based on a standardized remediation approach. The resident entered remediation for poor organizational skills and an inability to make or follow through with patient care plans. Opportunities for improvement in the applied remediation process are identified and discussed. Legal concerns and utility of neuropsychological assessment of residents are reviewed.

Conclusions

Remediation requires a complicated and detailed effort. This case demonstrates issues that program directors may face when working with residents and provides suggestions for use of specific remediation tools.     

Using Standardized Oral Examinations to Evaluate General Competencies

Emergency medicine residency programs are required by the Accreditation Council for Graduate Medical Education (ACGME) to formally evaluate each resident with oral and written examinations. The Michigan State University Emergency Medicine Residency Program in Lansing conducts monthly standardized oral examinations (SOEs) as part of each resident's evaluation. Recently, the ACGME has advanced six areas, termed "general competencies," that should be acquired during graduate medical education. According to the ACGME, these competencies should be included in the educational process of all residency programs. In promulgating these competencies, the ACGME did not provide examples of core content, strategies for implementation, or methods of evaluation; rather, individual residency programs are required to develop their own methods. The authors describe a modification of an existing SOE strategy that assesses residents' knowledge, skills, experiences, and attitudes as reflected in the general competencies.     

Evaluation of Resident Performance and Intensive Bedside Teaching during Direct Observation

Objective: To describe the experience of a residency program in emergency medicine with an intensive observational evaluation of resident performance in the ED.
Methods: Each resident was directly observed and evaluated during a clinical shift four times each academic year: once by each residency codirector and twice by the resident's faculty advisor. The faculty members performed this evaluation outside of "clinical staffing time," shadowing the resident for several hours in the ED during the resident's assigned shift. The resident and assigned faculty member discussed the patients' histories and physical examination findings and developed treatment plans together. Prior to initiation of the observation, the faculty were provided with guidelines for the evaluation of specific skills. Immediate feedback of strengths and deficiencies was provided to the resident.
Results: Subjective evaluations by faculty suggest that new insights into resident clinical strengths and weaknesses were determined using this approach. Objective scoring of resident performance demonstrated heterogeneity of skills between residents as well as inconsistency of skills for specific residents.
Conclusions: The program provided the faculty with protected teaching time, an opportunity to share clinical pearls, and unique insights into resident performance that are not obvious during standard clinical interactions.     

Reliability of a 360-degree evaluation to assess resident competence

OBJECTIVE: To determine the feasibility and psychometric qualities of a 360-degree evaluation of physical medicine and rehabilitation (PM&R) residents' competence. DESIGN: Nurses, allied health staff, and medical students completed a 12-item questionnaire after each PM&R resident rotation from January 2002 to December 2004. The items were derived from five of the six competencies defined by the Accreditation Council for Graduate Medical Education (ACGME). RESULTS: Nine hundred thirty evaluations of 56 residents were completed. The alpha reliability coefficient for the instrument was 0.89. Ratings did not vary significantly by resident gender. Senior residents had higher ratings than junior residents. A reliability of >0.8 could be achieved by ratings from just five nurses or allied health staff, compared with 23 ratings from medical students. Factor analysis revealed all items clustered on one factor, accounting for 84% of the variance. In a subgroup of residents with low scores, raters were able to differentiate among skills. CONCLUSION: Resident assessment tools should be valid, reliable, and feasible. This Web-based 360-degree evaluation tool is a feasible way to obtain reliable ratings from rehabilitation staff about resident behaviors. The assignment of higher ratings for senior residents than junior residents is evidence for the general validity of this 360-degree evaluation tool in the assessment of resident performance. Different rater groups may need distinct instruments based on the exposure of rater groups to various resident activities and behaviors.     

Electrodiagnostic medicine skills competency in physical medicine and rehabilitation residents: a method for development and assessment

This project sought to create an educational module including evaluation methodology to instruct physical medicine and rehabilitation (PM&R) residents in electrodiagnostic evaluation of patients with neuromuscular problems, and to verify acquired competencies in those electrodiagnostic skills through objective evaluation methodology. Sixteen residents were trained by board-certified neuromuscular and electrodiagnostic medicine physicians through technical training, lectures, and review of self-assessment examination (SAE) concepts from the American Academy of Physical Medicine & Rehabilitation syllabus provided in the Archives of Physical Medicine and Rehabilitation. After delivery of the educational module, knowledge acquisition and skill attainment were measured in (1) clinical skill in diagnostic procedures via a procedure checklist, (2) diagnosis and ability to design a patient-care management plan via chart simulated recall (CSR) exams, (3) physician/patient interaction via patient surveys, (4) physician/staff interaction via 360-degree global ratings, and (5) ability to write a comprehensive patient-care report and to document a patient-care management plan in accordance with Medicare guidelines via written patient reports. Assessment tools developed for this program address the basic competencies outlined by the Accreditation Council for Graduate Medical Education (ACGME). To test the success of the standardized educational module, data were collected on an ongoing basis. Objective measures compared resident SAE scores in electrodiagnostics (EDX) before and after institution of the comprehensive EDX competency module in a PM&R residency program. Fifteen of 16 residents (94%) successfully demonstrated proficiency in every segment of the evaluation element of the educational module by the end of their PGY-4 electrodiagnostic rotation. The resident who did not initially pass underwent remedial coursework and passed on the second attempt. Furthermore, the residents' proficiency as demonstrated by the evaluation after implementation of the standardized educational module positively correlated to an increase in resident SAE scores in EDX compared with resident scores before implementation of the educational module. Resident proficiency in EDX medicine skills and knowledge was objectively verified after completion of the standardized educational module. Validation of the assessment tools is evidenced by collected data correlating with significantly improved SAE scores and American Association of Neuromuscular and Electrodiagnostic Medicine (AANEM) exam scores, as outlined in the result section. In addition, the clinical development tool (procedure checklist) was validated by residents being individually observed performing skills and deemed competent by an AANEM-certified physician. The standardized educational module and evaluation methodology provide a potential framework for the definition of baseline competency in the clinical skill area of EDX.     

Direct Observation of Residents in the Emergency Department: A Structured Educational Program

Objectives:  The objective was to describe the implementation of a program of structured direct observation of emergency medicine (EM) residents during clinical shifts in the emergency department (ED).
Methods:  The authors developed a program in which an observer spent 4 to 5 hours with each resident, without intervening in the clinical encounters. A structured data form was developed to document the resident's performance in a number of defined clinical areas relevant to patient care and mastery of the core competencies. Individual strengths and weaknesses were noted, and the observer provided directed feedback at the end of the session.
Results:  Over an 18-month period, 32 EM residents were observed during their ED shifts. The sessions not only provided specific information on individual residents' performances, but also identified areas where the residency program curriculum could be enhanced and provided a means of assessing mastery of the core competencies. In addition, the program provided an opportunity to give detailed and timely directed feedback to residents. Both residents and attending staff found the sessions acceptable and useful.
Conclusions:  Implementation of a structured direct observation program was feasible and well received and provided insight into the strengths and weaknesses of residents both individually and as a group.     

Introduction of a Direct Observation Program Into a Pediatric Resident Continuity Clinic: Feasibility,Acceptability, and Effect on Resident Feedback

Background: Direct observations of trainee performance are essential to the evaluation of clinical skills, and are now required by the Accreditation Council on Graduate Medical Education (ACGME). Purpose: We sought to describe the feasibility and acceptability of a direct observation program in a pediatric resident clinic, and evaluate its effect on feedback. Methods: We modified the Structured Clinical Observation (SCO) for use in a pediatric resident clinic. Preceptors and residents were asked about the frequency and content of feedback given before and after the introduction of the observations. Results: One-hundred-sixty-six structured clinical observations were performed. Their use increased the frequency of feedback provided, and resulted in more feedback on listening skills and less feedback on medical knowledge and decision making. Conclusions: Structured clinical observations can be successfully introduced into a pediatric resident continuity clinic. This is a valuable supplement to the traditional precepting, and fulfills the mandate to include direct observations in the assessment of residents’ clinical competence.     

Systematic Training in Internal Medicine-Pediatrics End of Residency Handoffs: Residency Director Attitudes and Perceived Barriers

Background: It is unclear why systematic training in end-of-residency clinic handoffs is not universal. Purposes: We assessed Internal Medicine-Pediatrics (Med-Peds) residency program directors’ attitudes regarding end-of-residency clinic handoff systems and perceived barriers to their implementation. Methods: We surveyed all Med-Peds program directors in the United States about end-of-residency outpatient handoff systems. Results: Program directors rated systems as important (81.5%), but only 31 programs (46.3%) utilized them. Nearly all programs with (29/31 [93.5%]), and most programs without systems (24/33 [72.7%]) rated them as important. Programs were more likely to have a system if the program director rated it important (p = .049), and less likely if they cited a lack of faculty interest (p = .023) or difficulty identifying residents as primary providers (p = .04). Conclusions: Most program directors believe it important to formally hand off outpatients. Barriers to establishing handoff systems can be overcome with modest curricular and cultural changes.     

The Seventh Competency

Background: Disclosure of error is gaining acceptance as an ethical imperative in health care. Despite this, residency training programs do not commonly address this in their curricula, and competence in the identification and disclosure of adverse events and medical error is typically not assessed. Summary: Although aspects of the identification, disclosure, and apology for medical error can be subsumed under existing competencies, the skills required in this area are in many ways fundamentally different from anything else physicians are taught. Conclusions: We propose that the identification of medical error recognition and disclosure be recognized as a seventh core competency and we suggest that residency program directors be invited to develop innovative approaches to teaching and assessing competence in this area. This will benefit training programs, residents, and ultimately society and the patients that we serve.