Direct Observation of Residents in the Emergency Department: A Structured Educational Program

Objectives:  The objective was to describe the implementation of a program of structured direct observation of emergency medicine (EM) residents during clinical shifts in the emergency department (ED).
Methods:  The authors developed a program in which an observer spent 4 to 5 hours with each resident, without intervening in the clinical encounters. A structured data form was developed to document the resident's performance in a number of defined clinical areas relevant to patient care and mastery of the core competencies. Individual strengths and weaknesses were noted, and the observer provided directed feedback at the end of the session.
Results:  Over an 18-month period, 32 EM residents were observed during their ED shifts. The sessions not only provided specific information on individual residents' performances, but also identified areas where the residency program curriculum could be enhanced and provided a means of assessing mastery of the core competencies. In addition, the program provided an opportunity to give detailed and timely directed feedback to residents. Both residents and attending staff found the sessions acceptable and useful.
Conclusions:  Implementation of a structured direct observation program was feasible and well received and provided insight into the strengths and weaknesses of residents both individually and as a group.     

An Evaluation of Emergency Medicine Resident Interaction Time with Faculty in Different Teaching Venues

OBJECTIVES: To measure actual emergency medicine (EM) resident interaction time with faculty and to investigate the potential to use direct observation as an assessment tool for the core competencies. By 2006 all EM residencies must implement resident assessment techniques of the six Accreditation Council for Graduate Medical Education core competencies. Emergency medicine educators recommend direct observation as the optimal evaluation tool for patient care, systems-based practice, interpersonal and communication skills, and professionalism. Continuous faculty presence in the emergency department (ED) is widely believed to facilitate direct observation as an assessment technique. METHODS: Observational study of EM resident-faculty interaction time during two-hour periods. Study venues included two EDs, two trauma services, inpatient medicine, adult and pediatric intensive care units (ICUs), and a pediatric outpatient clinic. Using a priori definitions, the authors categorized faculty-EM resident interaction time as direct observation of patient care, indirect patient care, or non-patient care activities, and calculated total faculty interaction time. Subjects were blinded to the nature of the study, and data gathering was encrypted. RESULTS: Two hundred seventy observation periods of two hours each were conducted, sampling 32 EMR1, 33 EMR2-3, 41 EM, and 38 non-EM faculty. The mean total faculty interaction time ranged from a high of 30% (95% CI = 20% to 41%) in the pediatric ICU to a low of 10% (95% CI = 3% to 16%) on internal medicine wards. Overall, EM faculty interaction time was 20% (95% CI = 18% to 22%). Direct observation by faculty ranged from a high of 6% for EMR2-3s in the critical care areas of the ED (95% CI = 3% to 9%) to a low of 1% (95% CI = 0% to 2%) on internal medicine wards. Overall ED direct observation time was 3.6% (95% CI = 2.6% to 4.7%). Emergency department direct observation did not vary within EM resident training level or by ED site. Direct observation varied by treatment area within the EDs, with the critical care areas being substantially higher (6%) than the noncritical care areas (1%). CONCLUSIONS: Faculty direct observation time of EM residents was low in all training venues studied. Direct observation was the highest in ED critical care areas and lowest on medicine ward rotations. Emergency medicine faculty involved simultaneously in routine ED teaching, supervision, and patient care rarely performed direct observation, despite their continuous physical presence. This finding suggests that alternative strategies may be required to assess core competencies through direct observation in the ED.     

Direct Observation for Assessing Emergency Medicine Core Competencies: Interpersonal Skills

The American Board of Medical Specialties described six core competencies considered essential elements of medical practice: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. In response, the Accreditation Council for Graduate Medical Education (ACGME) mandated that all residency programs assess trainees for the newly defined core competencies. Despite the mandate for including these six competencies in residency training, neither a specific curriculum nor a method to assess the outlined objectives has been developed by the ACGME. Instead, it is up to individual residency programs to document how they plan to incorporate and assess the core competencies in their programs. This article describes the potential use of direct observation to assess resident performance in the interpersonal skills core competency.     

Systems-Based Practice: The Sixth Core Competency

Systems-Based Practice (SBP) is the sixth competency defined by the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project. Specifically, SBP requires "Residents [to] demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value." This competency can be divided into four subcompetencies, all of which are integral to training emergency medicine (EM) physicians: resources, providers, and systems; cost-appropriate care; delivery systems; and patient advocacy. In March 2002, the Council of Emergency Medicine Residency Directors (CORD-EM) convened a consensus conference to assist residency directors in modifying the SBP competency specific for EM. The Consensus Group modified the broad ACGME definition for SBP into EM-specific goals and objectives for residency training in SBP. The primary assessment methods from the Toolbox of Assessment Methods were also identified for SBP. They are direct observation, global ratings, 360-degree evaluations, portfolio assessment, and testing by both oral and written exams. The physician tasks from the Model of the Clinical Practice of Emergency Medicine that are most relevant to SBP are out-of-hospital care, modifying factors, legal/professional issues, diagnostic studies, consultation and disposition, prevention and education, multitasking, and team management. Suggested EM residency curriculum components for SBP are already in place in most residency programs, so no additional resources would be required for their implementation. These include: emergency medical services and administrative rotations, directed reading, various interdisciplinary and hospital committee participation, continuous quality improvement project participation, evidence-based medicine instruction, and various didactic experiences, including follow-up, interdisciplinary, and case conferences. With appropriate integration and evaluation of this competency into training programs, it is likely that future generations of physicians and patients will reap the benefits of an educational system that is based on well-defined outcomes and a more systemic view of health care.     

Application of the Medical Knowledge General Competency to Emergency Medicine

In February 1999, the Accreditation Council for Graduate Medical Education (ACGME) identified six general competencies as the basic educational goals required by all training programs for their residents. This places emphasis on educational outcome assessment in residency programs and in the accreditation process. A concomitant goal is to have increasingly valid, reliable assessments of the ability of a resident physician to provide safe, evidenced-based, humanistic medical care to their patients. To better define these competencies for the specialty of emergency medicine (EM), the Council of Emergency Medicine Residency Directors (CORD-EM) held a consensus conference in March 2002. This article reports the results of the Consensus Group for the core competency area of medical knowledge. This competency is already being well addressed in residency programs, but there has been inadequate documentation of a resident's knowledge base. The Consensus Group focused on many assessment methods to determine those having the best potential for use in EM programs. Assessment methods felt to be most appropriate for assessment of the medical knowledge base of a resident are presented, as are practical suggestions for incorporating these into EM programs.     

Defining and Evaluating Professionalism: A Core Competency for Graduate Emergency Medicine Education

Professionalism, long a consideration for physicians and their patients, is coming to the forefront as an essential element of graduate medical education as one of the six new core competency requirements of the Accreditation Council for Graduate Medical Education (ACGME). Professionalism is also integral to the widely endorsed Model of the Clinical Practice of Emergency Medicine (Model). Program directors have now been charged with implementing the new core competencies in training programs and to assess the acquisition of these competencies in their trainees. To assist emergency medicine (EM) program directors in this endeavor, the Council of Emergency Medicine Residency Directors (CORD-EM) held a consensus conference in March 2002. A focused Consensus Group addressed the specific core competency of professionalism during the course of this conference, and the results are highlighted in this article. The definition and curricular requirements relating to professionalism are highlighted, specific techniques for evaluating this core competency in EM are reviewed, and recommendations are provided regarding the most appropriate assessment method for EM programs.     

The Use of Simulation for Emergency Medicine Resident Assessment

Simulations are exercises designed to mimic real-life situations in which learners are given the opportunity to reason through a clinical problem and make critical decisions without the potential of harming actual patients. Simulation, using a variety of formats, is useful for assessing the core competencies-particularly patient care (decision making, prioritizing, procedural skills), interpersonal skills (team leadership, communication), and systems-based practice (team structure and utilization, resource use). High-fidelity computerized human simulators are a relatively new tool for use in medical simulation. These realistic mannequins mimic physical findings including respiratory rate, breath sounds, central and peripheral pulses, murmurs, and pupil reactivity. They generate an electrocardiographic (ECG) waveform, cardiac indices, and oxygen saturation that can be viewed on standard cardiac monitoring equipment and can be programmed to respond physiologically to medications and invasive procedures. The use of human simulators to reproduce life-threatening situations will be especially useful in assessing the clinical competence of emergency medicine physicians. Operational definitions of competence and tools with which to evaluate performance must first be developed. Standardization of scenarios and evaluation tools will permit assessment of the reproducibility of scenarios and the reliability and validity of the tools used to measure competence.     

助产士核心胜任力量表信度和效度研究

目的:对助产士核心胜任力量表进行信度和效度的检测。方法:采用文献回顾的方法,重点参考国际助产联盟制定的助产士胜任力标准,通过助产专业的专家,形成助产士核心胜任力量表,并对北京市19家医院的300名助产士进行测评,对量表进行信度和效度分析,最终形成量表。结果:有效量表295份。助产士核心胜任力量表共由6个维度,54项条目组成,其内部一致性Cronbach’sα系数为0.978,各分维度的Cronbach’sα系数为0.921～0.938之间,均在0.9以上,总量表的内容效度比为0.95,结构效度6个因子的累计解释变量为70.927%,均在测量学可接受的范围。结论:该助产士核心胜任力量表具有良好的信度和效度,条目设置适用于我国助产士核心胜任力的评价。     

Patient Care Competency in Emergency Medicine Graduate Medical Education: Results of a Consensus Group on Patient Care

"Patient Care" is the first listed core competency of the six new core competencies recently formulated by the Accreditation Council for Graduate Medical Education (ACGME) and, arguably, the most important. To assist emergency medicine (EM) program directors in incorporating and assessing this competency, the Council of Emergency Medicine Residency Directors (CORD-EM) held a consensus conference in March 2002. Definitions of this competency were generated that are specific for the training of practitioners in EM. These built upon the ACGME base definition, but include elements unique to or critically important in EM. In addition, all of the ACGME assessment tools were examined and prioritized for use in assessing the competency of EM residents in the area of patient care. Suggestions for an implementation process are also described.     

A Case Study in Medical Error: The Use of the Portfolio Entry

The Accreditation Council for Graduate Medical Education (ACGME) Practice-Based Learning and Improvement competency incorporates lifelong learning techniques and self-reflection. Resident portfolios have received attention as a useful method for addressing this competency. A recent patient encounter provided an experienced clinical educator with the opportunity to develop a portfolio entry that was distributed to all of the residents and faculty in an emergency medicine residency. This report may assist educators in explaining how one could approach the development of the portfolio as a tool for self-assessment. A candid discussion by a senior faculty member about issues that contributed to medical error has been underreported in the medical literature.     

Reliability of Performance-based Clinical Skill Assessment of Emergency Medicine Residents

Objective: To test the overall reliability of a performance-based clinical skill assessment for entering emergency medicine (EM) residents. Also, to investigate the reliability of separate reporting of diagnostic and management scores for a standardized patient case, subjective scoring of patient notes, and interstation exercise scores.
Methods: Thirty-four first-year EM residents were tested using a 10-station standardized patient (SP) examination. Following each 10-minute encounter, the residents completed a patient note that included differential diagnosis and management. The residents also were asked to read an ECG or chest x-ray (CXR) associated with each case. History, physical examination, and interpersonal skills were scored by the SPs. The patient note, CXR, and ECG readings were scored by faculty emergency physicians. Intercase reliability was determined for the residents.
Results: Global score reliability, Cronbach's α = 0.85. Reliabilities for the other components were: history, 0.77; physical examination, 0.83; and interpersonal skills, 0.80. Differential diagnosis and management reliabilities were 0.61 and 0.66, respectively. Subjective scoring of the patient note resulted in acceptable reliability for legibility (0.80), history completeness (0.80), and history organization (0.81). Physical examination completeness and organization reliabilities were 0.74 and 0.73. For ECG and CXR readings, α = 0.74 and 0.34, respectively.
Conclusions: SPs can be used to reliably assess bedside clinical skills of EM residents. While component reliability levels are slightly lower than the global clinical skill reliability coefficient, they are still high enough to use for identification of individual strengths and weaknesses.     

Faculty Prediction of In-training Examination Scores of Emergency Medicine Residents

Background

The Emergency Medicine In-Training Examination (EMITE) is one of the only valid tools for medical knowledge assessment in current use by emergency medicine (EM) residencies. However, EMITE results return late in the academic year, providing little time to institute potential remediation.

Objective

The goal of this study was to determine the ability of EM faculty to accurately predict resident EMITE scores prior to results return.

Methods

We asked EM faculty at the study site to predict the 2012 EMITE scores of the 50 EM residents 2 weeks prior to results being available. The primary outcome was prediction accuracy, defined as the proportion of predictions within 6% of the actual score. The secondary outcome was prediction precision, defined as the mean deviation of predictions from the actual scores. We assessed several faculty background variables, including years of experience, educational leadership status, and clinical hours worked, for correlation with the two outcomes.

Results

Thirty-two of the 38 faculty (84.2%, 95% confidence interval [CI] 69.6–92.6) participated in the study, rendering a total of 1600 predictions for 50 residents. Mean resident EMITE score was 81.1% (95% CI 79.5–82.8%). Mean prediction accuracy for all faculty participants was 69% (95% CI 65.9–72.1%). Mean prediction precision was 5.2% (95% CI 4.9–5.5%). Education leadership status was the only background variable correlated with the primary and secondary outcomes (Spearman's ρ = 0.51 and −0.53, respectively).

Conclusion

Faculty possess only moderate accuracy at predicting resident EMITE scores. We recommend a multicenter study to evaluate the generalizability of the present results.     

Reliability of Faculty Clinical Evaluations of Non-Emergency Medicine Residents during Emergency Department Rotations

Objectives: To assess the reliability of faculty evaluations of non-emergency medicine (non-EM) residents during clinical ED rotations and to determine the effect that the “leniency” of grading by these evaluators had on the residents' final evaluations. Methods: A prospective, observational study of the evaluation patterns of EM faculty was performed in an academic ED (50, 000 visits yearly census). Each resident was evaluated on a daily basis by a board-certified or board-prepared emergency physician. The evaluation form rated 7 characteristics, but only the rating for overall clinical competence was used for data analysis. If an attending evaluated the same resident more than once, only the first evaluation was used to avoid bias from prior exposure. The scoring patterns of the evaluators, both individually and in groups, were analyzed using 1-way analysis of variance. Evaluator leniency was estimated using the mean evaluator score across all residents. Since each resident was evaluated by a different combination of evaluators, evaluator leniency for each resident was estimated from the mean leniency of the evaluators who specifically assessed that resident. Results: During the period of the study, 66 residents rotated through the ED, yielding a total of 401 evaluations. When the scoring patterns of individual evaluators were analyzed, a high degree of variability was found in the mean scores (range 5.23–8.09) and SDs (range 0.45–1.55) across evaluators; p = 0.0001. There was a moderate correlation between the mean overall competence score received by each resident and that resident's evaluators' leniency, r = 0.52; p = 0.0001. Conclusions: There is significant variability in the scoring patterns of individual evaluators. The evaluators in this study showed large variations in both leniency (as measured by their mean score) and range restriction (as measured by their SD). The differences in evaluator scoring leniency have a moderate correlation with the overall score received by the resident.     

Assessing Competencies in a Master of Science in Clinical Research Program: The Comprehensive Competency Review

Competencies in Master of Science Clinical Research programs are becoming increasingly common. However, students and programs can only benefit fully from competency‐based education if students’ competence is formally assessed. Prior to a summative assessment, students must have at least one formative, formal assessment to be sure they are developing competence appropriate for their stage of training. This paper describes the comprehensive competency review (CCR), a milestone for MS students in Clinical Research at the University of Pittsburgh''s Institute for Clinical Research Education. The CCR involves metacognitive reflection of the student''s learning as a whole, written evidence of each competency, a narrative explaining the choice of evidence for demonstrating competencies, and a meeting in which two faculty members review the evidence and solicit further oral evidence of competence. CCRs allow for individualized feedback at the midpoint in degree programs, providing students with confidence that they will have the means and strategies to develop competence in all areas by the summative assessment of competence at their thesis defense. CCRs have also provided programmatic insight on the need for curricular revisions and additions. These benefits outweigh the time cost on the part of students and faculty in the CCR process.     

我国高校护理学科核心竞争力的实证研究

目的验证自建的高校护理学科核心竞争力评价指标体系,分析我国高校护理学科核心竞争力的特点。方法采用分层随机抽样的方法选择国内3所高校的护理学科为实证研究对象,采用自建评价模型进行实证研究。结果3所高校护理学科核心竞争力差异明显,由强到弱依次为某市属医科大学护理学科、某部属综合大学护理学科、某省属医科大学护理学科。结论高校护理学科核心竞争力评价模型鉴别力较强,提示我国高校护理学科核心竞争力总体薄弱,未形成明显优势和特色,但有待进一步研究证明。     

Situation,Background, Assessment,Recommendation: Competency Assessment for Agency Nurses in Radiology

Agency (daily registry) nurses play a crucial role in health care. Agencies provide supplemental staffing for health-care organizations on an as-needed basis. Health-care organizations are required to apply the same standards for competency assessment for agency nurses as they would for their own staff. Radiology is a highly specialized area that demands a high level of competency for all staff nurses, permanent or temporary. This article outlines the process one organization used to assess and develop competency assessment of agency nurses in radiology.     

护士心理授权与工作满意度对核心能力的影响

目的探讨护士心理授权、工作满意度对核心能力的影响,以期为提高护士的核心能力提供理论依据。方法2015年7-8月,便利抽样法选取济南市某三级甲等综合医院的护士172名为研究对象。采用一般情况调查表、心理授权量表(psychological empowerment scale,PES)、McCloskey/Mueller工作满意度量表(McCloskey/Mueller job satisfaction scale,MMSS)和注册护士核心能力量表(competency inventory for registered nurses,CIRN)对其进行测评。结果护士的核心能力总均分为(2.627±0.603)分,心理授权总均分为(3.623±0.506)分,工作满意度总均分为(3.078±0.618)分。核心能力中以法律/伦理实践维度得分最高,教育咨询和批判性思维能力维度得分最低。心理授权和工作满意度与核心能力存在明显的正相关关系(均P0.01)。多元回归分析结果显示,心理授权解释核心能力总变异的40.4%,其中自我效能与核心能力密切相关。结论护士的核心能力处于中等水平,管理者应该重视护士核心能力的培养,通过改善护士心理授权水平,提高护士的工作积极性和工作满意度,从而提高护士的核心能力。