Assessment of Communication and Interpersonal Skills Competencies

Excellent communication and interpersonal (C-IP) skills are a universal requirement for a well-rounded emergency physician. This requirement for C-IP skill excellence is a direct outgrowth of the expectations of our patients and a prerequisite to working in the increasingly complex emergency department environment. Directed education and assessment of C-IP skills are critical components of all emergency medicine (EM) training programs and now are a requirement of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project. In keeping with its mission to improve the quality of EM education and in response to the ACGME Outcome Project, the Council of Emergency Medicine Residency Directors (CORD-EM) hosted a consensus conference focusing on the application of the six core competencies to EM. The objective of this article is to report the results of this consensus conference as it relates to the C-IP competency. There were four primary goals: 1) define the C-IP skills competency for EM, 2) define the assessment methods currently used in other specialties, 3) identify the methods suggested by the ACGME for use in C-IP skills, and 4) analyze the applicability of these assessment techniques to EM. Ten specific communication competencies are defined for EM. Assessment techniques for evaluation of these C-IP competencies and a timeline for implementation are also defined. Standardized patients and direct observation were identified as the criterion standard assessment methods of C-IP skills; however, other methods for assessment are also discussed.     

Entrustable Professional Activities in Nurse Practitioner Education

How do you measure trust in nurse practitioner education? A growing body of literature is aimed at assessing health professional learners’ clinical competencies from a readiness for clinical practice perspective via an evaluation of entrustable professional activities (EPAs). This review describes the clinical assessment process and tools as part of a major curriculum reform that faculty at the University of California San Francisco, School of Nursing developed and implemented for evaluating Adult Gerontology Primary Care Nurse Practitioner students’ readiness for practice.     

360-degree Feedback: Possibilities for Assessment of the ACGME Core Competencies for Emergency Medicine Residents

The Accreditation Council for Graduate Medical Education (ACGME) has challenged residency programs to provide documentation via outcomes assessment that all residents have successfully mastered the six core competencies. A variety of assessment "tools" has been identified by the ACGME for outcomes assessment determination. Although rarely cited in the medical literature, 360-degree feedback is currently in widespread use in the business sector. This tool provides timely, consolidated feedback from sources in the resident's sphere of influence (emergency medicine faculty, emergency medicine residents, off-service residents and faculty, nurses, ancillary personnel, patients, out-of-hospital care providers, and a self-assessment). This is a significant deviation from both the peer review process and the resident review process that almost exclusively use physicians as raters. Because of its relative lack of development, utilization, and validation as a method of resident assessment in graduate medical education, a great opportunity exists to develop the 360-degree feedback tool for resident assessment.     

Using Standardized Oral Examinations to Evaluate General Competencies

Emergency medicine residency programs are required by the Accreditation Council for Graduate Medical Education (ACGME) to formally evaluate each resident with oral and written examinations. The Michigan State University Emergency Medicine Residency Program in Lansing conducts monthly standardized oral examinations (SOEs) as part of each resident's evaluation. Recently, the ACGME has advanced six areas, termed "general competencies," that should be acquired during graduate medical education. According to the ACGME, these competencies should be included in the educational process of all residency programs. In promulgating these competencies, the ACGME did not provide examples of core content, strategies for implementation, or methods of evaluation; rather, individual residency programs are required to develop their own methods. The authors describe a modification of an existing SOE strategy that assesses residents' knowledge, skills, experiences, and attitudes as reflected in the general competencies.     

Patient Care Competency in Emergency Medicine Graduate Medical Education: Results of a Consensus Group on Patient Care

"Patient Care" is the first listed core competency of the six new core competencies recently formulated by the Accreditation Council for Graduate Medical Education (ACGME) and, arguably, the most important. To assist emergency medicine (EM) program directors in incorporating and assessing this competency, the Council of Emergency Medicine Residency Directors (CORD-EM) held a consensus conference in March 2002. Definitions of this competency were generated that are specific for the training of practitioners in EM. These built upon the ACGME base definition, but include elements unique to or critically important in EM. In addition, all of the ACGME assessment tools were examined and prioritized for use in assessing the competency of EM residents in the area of patient care. Suggestions for an implementation process are also described.     

Clinical Assessment in Emergency Medicine

The Accreditation Council for Graduate Medical Education (ACGME) has challenged all residencies with a new paradigm-to teach and evaluate residents based on six core competencies. One of these core competencies is clinical assessment. Standardized patients and direct observation are the most promising for emergency medicine educators to use to assess this competency. There is much room for research and national standardization of methods.     

Direct Observation for Assessing Emergency Medicine Core Competencies: Interpersonal Skills

The American Board of Medical Specialties described six core competencies considered essential elements of medical practice: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. In response, the Accreditation Council for Graduate Medical Education (ACGME) mandated that all residency programs assess trainees for the newly defined core competencies. Despite the mandate for including these six competencies in residency training, neither a specific curriculum nor a method to assess the outlined objectives has been developed by the ACGME. Instead, it is up to individual residency programs to document how they plan to incorporate and assess the core competencies in their programs. This article describes the potential use of direct observation to assess resident performance in the interpersonal skills core competency.     

Application of the Medical Knowledge General Competency to Emergency Medicine

In February 1999, the Accreditation Council for Graduate Medical Education (ACGME) identified six general competencies as the basic educational goals required by all training programs for their residents. This places emphasis on educational outcome assessment in residency programs and in the accreditation process. A concomitant goal is to have increasingly valid, reliable assessments of the ability of a resident physician to provide safe, evidenced-based, humanistic medical care to their patients. To better define these competencies for the specialty of emergency medicine (EM), the Council of Emergency Medicine Residency Directors (CORD-EM) held a consensus conference in March 2002. This article reports the results of the Consensus Group for the core competency area of medical knowledge. This competency is already being well addressed in residency programs, but there has been inadequate documentation of a resident's knowledge base. The Consensus Group focused on many assessment methods to determine those having the best potential for use in EM programs. Assessment methods felt to be most appropriate for assessment of the medical knowledge base of a resident are presented, as are practical suggestions for incorporating these into EM programs.     

Nursing and competencies - a natural fit: the politics of skill /competency formation in nursing

WINDSOR C, DOUGLAS C and HARVEY T. Nursing Inquiry 2012; 19: 213-222 Nursing and competencies - a natural fit: the politics of skill/competency formation in nursing The last two decades have seen a significant restructuring of work across Australia and other industrialised economies, a critical part of which has been the appearance of competency based education and assessment. The competency movement is about creating a more flexible and mobile labour force to increase productivity and it does so by redefining work as a set of transferable or 'soft' generic skills that is transportable and is the possession of the individual. This article sought to develop an analysis of competency based clinical assessment of nursing students across a bachelor of nursing degree course. This involved an examination of a total of 406 clinical assessment tools that covered the years 1992-2009 and the three years of a bachelor degree. Data analysis generated three analytical findings: the existence of a hierarchy of competencies that prioritises soft skills over intellectual and technical skills; the appearance of skills as personal qualities or individual attributes; and the absence of context in assessment. The article argues that the convergence in nursing of soft skills and the professionalisation project reform has seen the former give legitimacy to the enduring invisibility and devaluation of nursing work.     

Challenges in measuring ACGME competencies: considerations for milestones

Background

Measuring milestones, competencies, and sub-competencies as residents progress through a training program is an essential strategy in Accreditation Council for Graduate Medical Education (ACGME)’s attempts to ensure graduates meet expected professional standards. Previous studies have found, however, that physicians make global ratings often by using a single criterion.

Methods

We use advanced statistical analysis to extend these studies by examining the validity of ACGME International competency measures for an international setting, across emergency medicine (EM) and neurology, and across evaluators. Confirmatory factor analysis (CFA) models were fitted to both EM and neurology data. A single-factor CFA was hypothesized to fit each dataset. This model was modified based on model fit indices. Differences in how different EM physicians perceived the core competencies were tested using a series of measurement invariance tests.

Results

Extremely high alpha reliability coefficients, factor coefficients (>?.93), and item correlations indicated multicollinearity, that is, most items being evaluated could essentially replace the underlying construct itself. This was true for both EM and neurology data, as well as all six EM faculty.

Conclusions

Evaluation forms measuring the six core ACGME competencies did not possess adequate validity. Severe multicollinearity exists for the six competencies in this study. ACGME is introducing milestones with 24 sub-competencies. Attempting to measure these as discrete elements, without recognizing the inherent weaknesses in the tools used will likely serve to exacerbate an already flawed strategy. Physicians likely use their “gut feelings” to judge a resident’s overall performance. A better process could be conceived in which this subjectivity is acknowledged, contributing to more meaningful evaluation and feedback.

     

Reliability of a 360-degree evaluation to assess resident competence

OBJECTIVE: To determine the feasibility and psychometric qualities of a 360-degree evaluation of physical medicine and rehabilitation (PM&R) residents' competence. DESIGN: Nurses, allied health staff, and medical students completed a 12-item questionnaire after each PM&R resident rotation from January 2002 to December 2004. The items were derived from five of the six competencies defined by the Accreditation Council for Graduate Medical Education (ACGME). RESULTS: Nine hundred thirty evaluations of 56 residents were completed. The alpha reliability coefficient for the instrument was 0.89. Ratings did not vary significantly by resident gender. Senior residents had higher ratings than junior residents. A reliability of >0.8 could be achieved by ratings from just five nurses or allied health staff, compared with 23 ratings from medical students. Factor analysis revealed all items clustered on one factor, accounting for 84% of the variance. In a subgroup of residents with low scores, raters were able to differentiate among skills. CONCLUSION: Resident assessment tools should be valid, reliable, and feasible. This Web-based 360-degree evaluation tool is a feasible way to obtain reliable ratings from rehabilitation staff about resident behaviors. The assignment of higher ratings for senior residents than junior residents is evidence for the general validity of this 360-degree evaluation tool in the assessment of resident performance. Different rater groups may need distinct instruments based on the exposure of rater groups to various resident activities and behaviors.     

CORD‐AEUS: Consensus Document for the Emergency Ultrasound Milestone Project

In 2012, the Accreditation Council for Graduate Medical Education (ACGME) designated ultrasound (US) as one of 23 milestone competencies for emergency medicine (EM) residency graduates. With increasing scrutiny of medical educational programs and their effect on patient safety and health care delivery, it is imperative to ensure that US training and competency assessment is standardized. In 2011, a multiorganizational committee composed of representatives from the Council of Emergency Medicine Residency Directors (CORD), the Academy of Emergency Ultrasound of the Society for Academic Emergency Medicine (SAEM), the Ultrasound Section of the American College of Emergency Physicians (ACEM), and the Emergency Medicine Residents' Association was formed to suggest standards for resident emergency ultrasound (EUS) competency assessment and to write a document that addresses the ACGME milestones. This article contains a historical perspective on resident training in EUS and a table of core skills deemed to be a minimum standard for the graduating EM resident. A survey summary of focused EUS education in EM residencies is described, as well as a suggestion for structuring education in residency. Finally, adjuncts to a quantitative measurement of resident competency for EUS are offered.     

Quantitative Validation of a General Competency Composite Assessment Evaluation

OBJECTIVES: The authors sought to modify and validate a composite assessment evaluation process that assesses resident acquisition of the Accreditation Council for Graduate Medical Education (ACGME) general competencies (GCs). METHODS:This study critically analyzed the evaluation process used in a multicenter study (150 emergency medicine resident evaluations) to determine whether the procedure was psychometrically valid. For each GC, principal component analysis (PCA) was used to determine whether certain evaluation items could be eliminated, as well as to determine the magnitude of variability explained by up to three linear combinations or "principal components." The factor proportions (factor loadings) of various eigenvectors were measured to determine the degree of variability (determined by the square of the factor proportion) within a data or item set. The factor proportions essentially measure the length of the eigenvector as determined from a correlation matrix. RESULTS: The first three principal components are reported as factor proportion sum (% of total variability) as follows: patient care 0.91 (83%), medical knowledge 0.87 (76%), practice-based learning and improvement 0.90 (81%), interpersonal and communication skills 0.84 (71%), professionalism 0.74 (55%), and systems-based practice 0.80 (64%). PCA showed that evaluating certain traditional categories such as medical knowledge seemed to capture a single element, whereas professionalism appeared to measure a more complex, multidimensional phenomenon. CONCLUSIONS: By using a structured development process, the authors were able to create valid evaluation items for determining resident acquisition of the ACGME GCs.     

标准化患者结合临床模拟情境在《护理学基础》技能考核中的应用

目的探讨标准化患者结合临床模拟情境在《护理学基础》技能考核中的效果。方法在我校护理本科2009级《护理学基础》课程中采用标准化患者结合临床模拟情境对学生护理学基础技能进行考核,并考核后运用自制的问卷对参加考核的学生进行问卷调查。结果调查问卷结果显示,学生对于此种考核形式持肯定意见,认为有助于提高自身的综合技能。结论标准化患者结合临床模拟情境进行《护理学基础》技能考核可以为学生创设较为真实的临床情境,使考核教师能全面观察学生综合运用的知识和技能、职业情感、人文关怀意识和能力等各方面水平,但对教师的组织实施和学生的临场发挥等方面都有一定的挑战。     

Embarking on a Journey of Discovery: Developing Transitional Skill Sets through a Scholarly Concentration Program

Problem: Medical student participation in research enhances appreciation of the scientific literature and the conduct of investigation, and may lead to an interest in academic medicine. Independent medical student research offers frequently overlooked opportunities to develop and assess professional practice abilities, including project design and implementation, interprofessional team communication, and time management. These skills, useful to physicians, are often challenging for medical students to master as they transition into clinical careers. To address this challenge, we designed and embedded interventional modalities into a highly mentored and longitudinal scholarly concentration component of the curriculum. Intervention: The Embark scholarly concentration program incorporates traditional research training with the development of professional practice skills essential for transitioning to clinical practice. The program includes individualized and just-in-time components enabling student access to information and feedback specific to their projects and development of professional practice skills. Context: The Embark program is a required longitudinal component of the Oakland University William Beaumont School of Medicine undergraduate medical curriculum. The Embark program consists of courses that inform and facilitate a required longitudinal independent research project. Outcome: A retrospective evaluation of the Embark program’s success with development of professional practice skills through the lens of both faculty and student perceptions included analysis of project records and course evaluation feedback. Evaluation of individual student development of transitional skill ability is possible through both quantitative and qualitative analysis of data collected from student project records. More than 80% of course evaluation commentary on strengths of the program addressed activities related to professional practice skills. To systematize the evaluation of these data sources, we have piloted a framework, iSAIL, designed to assess student development in these skills during the planning and conduct of a research project. Lessons learned: By developing professional practice skills in the context of a scholarly concentration program, medical students can build a foundation for future engagement in research while they develop skills to overcome challenges that they are likely to encounter in their clinical careers. Modalities designed to evaluate individualized student development of professional practice skills through research participation define program successes and may lead to the identification of additional resources needed by students. By offering medical students opportunities to develop professional practice skills within the protected environment of an independent research project, this scholarly concentration program provides a valuable opportunity to influence the early development of skills necessary throughout their clinical careers.