Direct Observation for Assessing Emergency Medicine Core Competencies: Interpersonal Skills

The American Board of Medical Specialties described six core competencies considered essential elements of medical practice: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. In response, the Accreditation Council for Graduate Medical Education (ACGME) mandated that all residency programs assess trainees for the newly defined core competencies. Despite the mandate for including these six competencies in residency training, neither a specific curriculum nor a method to assess the outlined objectives has been developed by the ACGME. Instead, it is up to individual residency programs to document how they plan to incorporate and assess the core competencies in their programs. This article describes the potential use of direct observation to assess resident performance in the interpersonal skills core competency.     

360-degree Feedback: Possibilities for Assessment of the ACGME Core Competencies for Emergency Medicine Residents

The Accreditation Council for Graduate Medical Education (ACGME) has challenged residency programs to provide documentation via outcomes assessment that all residents have successfully mastered the six core competencies. A variety of assessment "tools" has been identified by the ACGME for outcomes assessment determination. Although rarely cited in the medical literature, 360-degree feedback is currently in widespread use in the business sector. This tool provides timely, consolidated feedback from sources in the resident's sphere of influence (emergency medicine faculty, emergency medicine residents, off-service residents and faculty, nurses, ancillary personnel, patients, out-of-hospital care providers, and a self-assessment). This is a significant deviation from both the peer review process and the resident review process that almost exclusively use physicians as raters. Because of its relative lack of development, utilization, and validation as a method of resident assessment in graduate medical education, a great opportunity exists to develop the 360-degree feedback tool for resident assessment.     

Application of the Medical Knowledge General Competency to Emergency Medicine

In February 1999, the Accreditation Council for Graduate Medical Education (ACGME) identified six general competencies as the basic educational goals required by all training programs for their residents. This places emphasis on educational outcome assessment in residency programs and in the accreditation process. A concomitant goal is to have increasingly valid, reliable assessments of the ability of a resident physician to provide safe, evidenced-based, humanistic medical care to their patients. To better define these competencies for the specialty of emergency medicine (EM), the Council of Emergency Medicine Residency Directors (CORD-EM) held a consensus conference in March 2002. This article reports the results of the Consensus Group for the core competency area of medical knowledge. This competency is already being well addressed in residency programs, but there has been inadequate documentation of a resident's knowledge base. The Consensus Group focused on many assessment methods to determine those having the best potential for use in EM programs. Assessment methods felt to be most appropriate for assessment of the medical knowledge base of a resident are presented, as are practical suggestions for incorporating these into EM programs.     

Developing a Valid Evaluation for Interpersonal and Communication Skills

Objectives The Accreditation Council for Graduate Medical Education requires that residency programs evaluate the acquisition of six general competencies, including Interpersonal and Communication Skills (ICS). To develop a 360-degree method for accomplishing this, a semantic-differentiation matrix for various communication traits for nurses to evaluate physician ICS was developed. The authors sought to determine whether this evaluation method could discriminate between more experienced medical communicators (faculty) and less experienced medical communicators (residents).
Methods A 98-item questionnaire measured several communication dimensions by using an eight-element semantic-differentiation scale. In addition, global assessment ratings assessed nursing perceptions of physician ICS skills. This process was repeated for various clinical scenarios.
Results There were 26 nurse evaluators, 19 emergency medicine (EM) residents (EM2 and EM3), and 30 EM faculty. Each physician received five independent evaluations (total, 245 evaluations). All EM residents (EM2 and EM3) were compared with the EM faculty. All eight items on the semantic-differentiation scale were compared. Likewise, the global assessment scores were compared. In every category, the faculty scored higher (Mann-Whitney U: p < 0.001).
Conclusions An evaluation process integrating a semantic-differentiation matrix was applied to various clinical scenarios (as well as global assessment items) and demonstrated discriminatory results. Faculty physicians scored significantly higher than resident physicians. The ability to provide discriminatory results is a requisite in the development of a valid evaluation process. The described semantic-differentiation matrix and global assessment questions may be valid measurements of ICS.     

CORD‐AEUS: Consensus Document for the Emergency Ultrasound Milestone Project

In 2012, the Accreditation Council for Graduate Medical Education (ACGME) designated ultrasound (US) as one of 23 milestone competencies for emergency medicine (EM) residency graduates. With increasing scrutiny of medical educational programs and their effect on patient safety and health care delivery, it is imperative to ensure that US training and competency assessment is standardized. In 2011, a multiorganizational committee composed of representatives from the Council of Emergency Medicine Residency Directors (CORD), the Academy of Emergency Ultrasound of the Society for Academic Emergency Medicine (SAEM), the Ultrasound Section of the American College of Emergency Physicians (ACEM), and the Emergency Medicine Residents' Association was formed to suggest standards for resident emergency ultrasound (EUS) competency assessment and to write a document that addresses the ACGME milestones. This article contains a historical perspective on resident training in EUS and a table of core skills deemed to be a minimum standard for the graduating EM resident. A survey summary of focused EUS education in EM residencies is described, as well as a suggestion for structuring education in residency. Finally, adjuncts to a quantitative measurement of resident competency for EUS are offered.     

The Use of Standardized Patients within a Procedural Competency Model to Teach Death Disclosure

OBJECTIVE: To design, implement, and evaluate a multi-dimensional, interdisciplinary, educational training module that enables residents to deliver an effective and empathic death disclosure in the emergency setting. The Accreditation Council for Graduate Medical Education (ACGME) "Toolbox of Assessment Methods" to assess competency was adopted as the foundation of this project. METHODS: Sixteen emergency medicine residents, eight postgraduate year 1 (PGY-1) and eight PGY-2, underwent a one-day training and evaluation exercise. The exercise consisted of: 1) a large-group didactic session, 2) a small-group didactic session, and 3) two standardized patient (SP) examinations. Changes in comfort levels, training helpfulness, and competency were measured. Inter-rater agreement between evaluators was examined. RESULTS: Trainees reported improvement in comfort levels and high levels of satisfaction regarding the helpfulness of the training. Good interrater agreement was obtained regarding resident competency to perform a death disclosure between the faculty and SP evaluators [kappa 0.61; 95% confidence interval (95% CI) = 0.33 to 0.88]. However, overall agreement among raters was poor (kappa 0.16; standard error = 0.26). This poor agreement reflected a lack of agreement between resident and SP evaluators (kappa 0.08; 95% CI = 0.16 to 0.33) and resident and faculty evaluators (kappa -0.02; 95% CI = 0.30 to 0.26). CONCLUSIONS: This project used the ACGME "Toolbox of Assessment Methods" to evaluate the competency of emergency medicine trainees to perform an effective and empathic death disclosure. The finding of inconsistent competency assessments by resident self-evaluators compared with those assessments made by faculty and standardized patients have important implications in future curricular design.     

Outcome Assessment in Emergency Medicine—A Beginning: Results of the Council of Emergency Medicine Residency Directors (CORD) Emergency Medicine Consensus Workgroup on Outcome Assessment

This article is designed to serve as a guide for emergency medicine (EM) educators seeking to comply with the measurement and reporting requirements for Phase 3 of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project. A consensus workshop held during the 2006 Council of Emergency Medicine Residency Directors (CORD) "Best Practices" conference identified specific measures for five of the six EM competencies—interpersonal communication skills, patient care, practice-based learning, professionalism, and systems-based practice (medical knowledge was excluded). The suggested measures described herein should allow for ease in data collection and applicability to multiple core competencies as program directors incorporate core competency outcome measurement into their EM residency training programs.     

An Evaluation of Emergency Medicine Resident Interaction Time with Faculty in Different Teaching Venues

OBJECTIVES: To measure actual emergency medicine (EM) resident interaction time with faculty and to investigate the potential to use direct observation as an assessment tool for the core competencies. By 2006 all EM residencies must implement resident assessment techniques of the six Accreditation Council for Graduate Medical Education core competencies. Emergency medicine educators recommend direct observation as the optimal evaluation tool for patient care, systems-based practice, interpersonal and communication skills, and professionalism. Continuous faculty presence in the emergency department (ED) is widely believed to facilitate direct observation as an assessment technique. METHODS: Observational study of EM resident-faculty interaction time during two-hour periods. Study venues included two EDs, two trauma services, inpatient medicine, adult and pediatric intensive care units (ICUs), and a pediatric outpatient clinic. Using a priori definitions, the authors categorized faculty-EM resident interaction time as direct observation of patient care, indirect patient care, or non-patient care activities, and calculated total faculty interaction time. Subjects were blinded to the nature of the study, and data gathering was encrypted. RESULTS: Two hundred seventy observation periods of two hours each were conducted, sampling 32 EMR1, 33 EMR2-3, 41 EM, and 38 non-EM faculty. The mean total faculty interaction time ranged from a high of 30% (95% CI = 20% to 41%) in the pediatric ICU to a low of 10% (95% CI = 3% to 16%) on internal medicine wards. Overall, EM faculty interaction time was 20% (95% CI = 18% to 22%). Direct observation by faculty ranged from a high of 6% for EMR2-3s in the critical care areas of the ED (95% CI = 3% to 9%) to a low of 1% (95% CI = 0% to 2%) on internal medicine wards. Overall ED direct observation time was 3.6% (95% CI = 2.6% to 4.7%). Emergency department direct observation did not vary within EM resident training level or by ED site. Direct observation varied by treatment area within the EDs, with the critical care areas being substantially higher (6%) than the noncritical care areas (1%). CONCLUSIONS: Faculty direct observation time of EM residents was low in all training venues studied. Direct observation was the highest in ED critical care areas and lowest on medicine ward rotations. Emergency medicine faculty involved simultaneously in routine ED teaching, supervision, and patient care rarely performed direct observation, despite their continuous physical presence. This finding suggests that alternative strategies may be required to assess core competencies through direct observation in the ED.     

Direct Observation of Residents in the Emergency Department: A Structured Educational Program

Objectives:  The objective was to describe the implementation of a program of structured direct observation of emergency medicine (EM) residents during clinical shifts in the emergency department (ED).
Methods:  The authors developed a program in which an observer spent 4 to 5 hours with each resident, without intervening in the clinical encounters. A structured data form was developed to document the resident's performance in a number of defined clinical areas relevant to patient care and mastery of the core competencies. Individual strengths and weaknesses were noted, and the observer provided directed feedback at the end of the session.
Results:  Over an 18-month period, 32 EM residents were observed during their ED shifts. The sessions not only provided specific information on individual residents' performances, but also identified areas where the residency program curriculum could be enhanced and provided a means of assessing mastery of the core competencies. In addition, the program provided an opportunity to give detailed and timely directed feedback to residents. Both residents and attending staff found the sessions acceptable and useful.
Conclusions:  Implementation of a structured direct observation program was feasible and well received and provided insight into the strengths and weaknesses of residents both individually and as a group.     

Pediatric Pain Control and Conscious Sedation: A Survey of Emergency Medicine Residencies

Objective : To determine the availability and relative use of pediatric analgesia and sedation at sites of U.S. emergency medicine residency training programs.
Methods : A mail/telephone survey of residency directors at 80 U.S. emergency medicine residencies regarding resident experience with pediatric analgesia and sedation for painful procedures conducted during November 1991.
Results : Sixty of 80 surveys (75%) were completed and available for analysis. Emergency medicine faculty supervised conscious sedation and analgesia in 87% of responding programs, while pediatrics faculty and pediatrics-emergency medicine fellows supervised in the remainder. Ninety-three percent of the programs had sedating agents available in the emergency department; only four programs needed to have drugs brought from the pharmacy. Thirty-four programs (57%) had formal protocols for the administration of these drugs. Seventy-seven percent of the programs had airway resuscitation equipment at the bedside, while only 63% brought resuscitation drugs. However, 60% of the programs reported complications of sedation, including respiratory depression, prolonged sedation, agitation, and vomiting. The most commonly used agents were midazolam (82%), meperidine alone (68%) and with promethazine and chlorpromazine (67%), and chloral hydrate (67%). Only 25% of the programs used nitrous oxide, and 30% used ketamine.
Conclusions : Emergency medicine residencies generally have available agents for pain control and conscious sedation in children, although the agents used vary widely. Appropriate instruction by trained faculty should enhance resident experience with pediatric pain control and sedation.     

Interobserver Variability among Faculty in Evaluations of Residents Clinical Skills

Objective: To describe interobserver variability among emergency medicine (EM) faculty when using global assessment (GA) rating scales and performance-based criterion (PBC) checklists to evaluate EM residents clinical skills during standardized patient (SP) encounters. Methods: Six EM residents were videotaped during encounters with SPs and subsequently evaluated by 38 EM faculty at four EM residency sites. There were two encounters in which a single SP presented with headache, two in which a second SP presented with chest pain, and two in which a third SP presented with abdominal pain, resulting in two parallel sets of three. Faculty used GA rating scales to evaluate history taking, physical examination, and interpersonal skills for the initial set of three cases. Each encounter in the second set was evaluated with complaint-specific PBC checklists developed by SAEM's National Consensus Group on Clinical Skills Task Force. Results: Standard deviations, computed for each score distribution, were generally similar across evaluation methods. None of the distributions deviated significantly from that of a Gaussian distribution, as indicated by the Kolmogorov-Smirnov goodness-of-fit test. On PBC checklists, 80% agreement among faculty observers was found for 74% of chest pain, 45% of headache, and 30% of abdominal pain items. Conclusions: When EM faculty evaluate clinical performance of EM residents during videotaped SP encounters, interobserver variabilities are similar, whether a PBC checklist or a GA rating scale is used.     

Assessing clinical competence in physical medicine & rehabilitation residency programs

OBJECTIVE: Evaluation of resident clinical competence is a complex task. A multimodal approach is necessary to capture all of the dimensions of competence. Recent guidelines from the Accreditation Council for Graduate Medical Education delineate six general competencies that physicians should posses. Application of these guidelines presents challenges to residency program directors in defining educational experiences and evaluation methods. DESIGN: We surveyed 81 physical medicine and rehabilitation program directors regarding assessment tools used in their programs. Seventy-five percent responded. The most frequently used assessment tools included: In-training self-assessment examinations, faculty evaluations, direct observation, and conference participation. Program directors assigned the highest values to direct observation, faculty evaluations, self-assessment examinations, and oral examinations. RESULTS: Of the general competencies, more than 90% of program directors believed they did an adequate job rating dimensions of patient care, medical knowledge, professionalism, and communication skills. Approximately one-third, however, thought they did a less than fair job rating practice-based learning and improvement and systems-based practice. The majority of programs reported that they were able to identify a resident with difficulties during the first year of training, 44% within the first 6 months. Program directors reported that their residents spend a significant amount of their time with nurses and therapists during their inpatient rotations; however, this was not reflected in their evaluation practices, in which only one-fourth of programs reported the use of nurses and therapists in evaluating residents. CONCLUSIONS: Survey results indicate that physical medicine and rehabilitation program directors apply a variety of assessment tools in evaluating resident clinical competence. Although perceptions about the relative value of these tools vary, most programs report a high value to direct observation of residents by faculty. Of the six general competencies, program directors struggle the most with their evaluation of practice-based learning and improvement and systems-based practice.     

Establishing competency-based measures for Department of Veterans Affairs post-graduate nurse practitioner residencies

BackgroundIn the past decade, numerous nurse residency models have been created and implemented nationwide; however, validated specialty-specific competency standards have not been established to evaluate Nurse Practitioner (NP) resident core competencies.PurposeTo report the specialty-specific competency assessment tool devised to assess Department of Veterans Affairs (VA) NP residents' competencies and discuss the VA NP residency program's effectiveness in expanding new graduate NP knowledge and skills in the veteran-centric care setting.MethodsThe VA Nursing Academic Partnership NP residency faculty established and piloted a web-based Nurse Practitioner Resident Competency Assessment (NPRCA) instrument for the comprehensive, specialty-specific assessment of individual NP resident's skill competencies across 24 areas.ResultsThe VA specialty-specific competency assessment instrument demonstrates strong internal consistency. The robust VA NP residency program enhances new graduate NP competencies.ConclusionsThe VA NP residency model can further the goal of standardizing clinical competencies in NP residency programs.     

Assessment of Communication and Interpersonal Skills Competencies

Excellent communication and interpersonal (C-IP) skills are a universal requirement for a well-rounded emergency physician. This requirement for C-IP skill excellence is a direct outgrowth of the expectations of our patients and a prerequisite to working in the increasingly complex emergency department environment. Directed education and assessment of C-IP skills are critical components of all emergency medicine (EM) training programs and now are a requirement of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project. In keeping with its mission to improve the quality of EM education and in response to the ACGME Outcome Project, the Council of Emergency Medicine Residency Directors (CORD-EM) hosted a consensus conference focusing on the application of the six core competencies to EM. The objective of this article is to report the results of this consensus conference as it relates to the C-IP competency. There were four primary goals: 1) define the C-IP skills competency for EM, 2) define the assessment methods currently used in other specialties, 3) identify the methods suggested by the ACGME for use in C-IP skills, and 4) analyze the applicability of these assessment techniques to EM. Ten specific communication competencies are defined for EM. Assessment techniques for evaluation of these C-IP competencies and a timeline for implementation are also defined. Standardized patients and direct observation were identified as the criterion standard assessment methods of C-IP skills; however, other methods for assessment are also discussed.     

An Approach to Fulfilling the Systems-based Practice Competency Requirement

The Accreditation Council for Graduate Medical Education (ACGME)-identified core competency of systems-based practice requires the demonstration of an awareness of the larger context and system of health care, and the ability to call on system resources to provide optimum care. This article describes an approach to teaching and fulfilling the requirement of this core competency in an emergency medicine residency. Beginning residents are oriented to community resources that are important to the larger context of care outside the emergency department. Each resident completes a community project during his or her residency. Readings and discussions concerning community-oriented medical care and the literature of research and injury prevention in emergency medicine precede the project development. Several projects are described in detail. Such projects help to teach not only awareness of the community resources of the greater context of medical practice outside the emergency department, but also how to use those resources. Projects could be a main component of a resident portfolio. This approach to teaching the core competency of systems-based practice is proposed as an innovative and substantial contribution toward satisfying the requirement of the core competency.     

Addressing the Systems-based Practice Core Competency: A Simulation-based Curriculum

Systems-based practice is one of the six core competencies implemented by the Accreditation Council for Graduate Medical Education to direct residency educational outcome assessment and accreditation. Emergency medicine–specific systems-based practice criteria have been described to define the expected knowledge and skill sets pertinent to emergency medicine practitioners. High-fidelity patient simulation is increasingly used in graduate medical education to augment case-based learning. The authors describe a simulation-based curriculum to address the emergency medicine–specific systems-based practice core competency.     

Clinical Assessment in Emergency Medicine

The Accreditation Council for Graduate Medical Education (ACGME) has challenged all residencies with a new paradigm-to teach and evaluate residents based on six core competencies. One of these core competencies is clinical assessment. Standardized patients and direct observation are the most promising for emergency medicine educators to use to assess this competency. There is much room for research and national standardization of methods.     

Can Unannounced Standardized Patients Assess Professionalism and Communication Skills in the Emergency Department?

Objectives:  The authors piloted unannounced standardized patients (USPs) in an emergency medicine (EM) residency to test feasibility, acceptability, and performance assessment of professionalism and communication skills.
Methods:  Fifteen postgraduate year (PGY)-2 EM residents were scheduled to be visited by two USPs while working in the emergency department (ED). Multidisciplinary support was utilized to ensure successful USP introduction. Scores (% well done) were calculated for communication and professionalism skills using a 26-item, behaviorally anchored checklist. Residents' attitudes toward USPs and USP detection were also surveyed.
Results:  Of 27 USP encounters attempted, 17 (62%) were successfully completed. The detection rate was 44%. Eighty-three percent of residents who encountered a USP felt that the encounter did not hinder daily practice and did not make them uncomfortable (86%) or suspicious of patients (71%). Overall, residents received a mean score of 60% for communication items rated "well done" (SD ± 28%, range = 23%–100%) and 53% of professionalism items "well done" (SD ± 20%, range = 23%-85%). Residents' communication skills were weakest for patient education and counseling (mean = 43%, SD ± 31%), compared with information gathering (68%, SD ± 36% and relationship development (62%, SD ± 32%). Scores of residents who detected USPs did not differ from those who had not.
Conclusions:  Implementing USPs in the ED is feasible and acceptable to staff. The unpredictability of the ED, specifically resident schedules, accounted for most incomplete encounters. USPs may represent a new way to assess real-time resident physician performance without the need for faculty resources or the bias introduced by direct observation.