A Case Study in Medical Error: The Use of the Portfolio Entry

The Accreditation Council for Graduate Medical Education (ACGME) Practice-Based Learning and Improvement competency incorporates lifelong learning techniques and self-reflection. Resident portfolios have received attention as a useful method for addressing this competency. A recent patient encounter provided an experienced clinical educator with the opportunity to develop a portfolio entry that was distributed to all of the residents and faculty in an emergency medicine residency. This report may assist educators in explaining how one could approach the development of the portfolio as a tool for self-assessment. A candid discussion by a senior faculty member about issues that contributed to medical error has been underreported in the medical literature.     

Assessing clinical competence in physical medicine & rehabilitation residency programs

OBJECTIVE: Evaluation of resident clinical competence is a complex task. A multimodal approach is necessary to capture all of the dimensions of competence. Recent guidelines from the Accreditation Council for Graduate Medical Education delineate six general competencies that physicians should posses. Application of these guidelines presents challenges to residency program directors in defining educational experiences and evaluation methods. DESIGN: We surveyed 81 physical medicine and rehabilitation program directors regarding assessment tools used in their programs. Seventy-five percent responded. The most frequently used assessment tools included: In-training self-assessment examinations, faculty evaluations, direct observation, and conference participation. Program directors assigned the highest values to direct observation, faculty evaluations, self-assessment examinations, and oral examinations. RESULTS: Of the general competencies, more than 90% of program directors believed they did an adequate job rating dimensions of patient care, medical knowledge, professionalism, and communication skills. Approximately one-third, however, thought they did a less than fair job rating practice-based learning and improvement and systems-based practice. The majority of programs reported that they were able to identify a resident with difficulties during the first year of training, 44% within the first 6 months. Program directors reported that their residents spend a significant amount of their time with nurses and therapists during their inpatient rotations; however, this was not reflected in their evaluation practices, in which only one-fourth of programs reported the use of nurses and therapists in evaluating residents. CONCLUSIONS: Survey results indicate that physical medicine and rehabilitation program directors apply a variety of assessment tools in evaluating resident clinical competence. Although perceptions about the relative value of these tools vary, most programs report a high value to direct observation of residents by faculty. Of the six general competencies, program directors struggle the most with their evaluation of practice-based learning and improvement and systems-based practice.     

Evaluation of the Educational Utility of Patient Follow-up

OBJECTIVES: To date, no studies in emergency medicine (EM) have addressed the educational value of the Residency Review Committee for Emergency Medicine's (RRC-EM) requirement for patient follow-up (FU). The authors examined whether performance of patient FU improved EM resident education. METHODS: All EM resident FU encounters from September 25, 2001, through September 24, 2002, were documented and analyzed. All EM residents at a regional tertiary referral emergency department (ED) initiated patient FU encounters by entering information regarding patients' initial ED presentations into a Web-based follow-up system (WBFUS), subsequently entered FU information, and indicated whether they thought that the specific FU encounters improved their education (yes/no). Supervising faculty members then reviewed the residents' completed FU entries. Blinded to residents' responses regarding educational utility, faculty members evaluated whether they thought the specific FU encounters were educational for the residents (yes/no). Data entered into the WBFUS were then summarized as percentages. RESULTS: Eight hundred forty-seven FU encounters were completed by 18 EM residents and 29 EM faculty. Ninety-three percent of the FU entries were deemed by at least one evaluator (resident or faculty) to have educational value. Residents found the act of performing FU educational in 81.3% of cases, whereas faculty thought 80.4% were educational for the resident. Although the residents and faculty agreed on the educational value in 75.4% of cases, the overall strength of the agreement was slight to fair (kappa statistic = 0.21). CONCLUSIONS: This study indicates that EM residents and faculty believe that the act of performing patient FU has educational value for EM residents; however, the interobserver agreement between residents and faculty was low.     

Direct Observation for Assessing Emergency Medicine Core Competencies: Interpersonal Skills

The American Board of Medical Specialties described six core competencies considered essential elements of medical practice: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. In response, the Accreditation Council for Graduate Medical Education (ACGME) mandated that all residency programs assess trainees for the newly defined core competencies. Despite the mandate for including these six competencies in residency training, neither a specific curriculum nor a method to assess the outlined objectives has been developed by the ACGME. Instead, it is up to individual residency programs to document how they plan to incorporate and assess the core competencies in their programs. This article describes the potential use of direct observation to assess resident performance in the interpersonal skills core competency.     

A Review of Portfolio Use in Residency Programs and Considerations before Implementation

Background: Portfolios, often described as collections of evidence, are discussed as a means of teaching or assessing the Accreditation Council for Graduate Medical Education competencies. Yet, it is unclear how many residency programs utilize portfolios. The purpose of this article is to (a) review the literature on portfolio use in graduate medical education; (b) examine efficacy of portfolio use, based upon studies in the field; and (c) offer a discussion of considerations for implementing portfolios. Summary: Two searches of PubMed, OVID, JSTOR, SCOPUS, and FirstSearch Wilson Select were conducted between October 2006 and April 2007 to identify studies and articles related to portfolio usage. Thirty-nine articles met criteria and were reviewed. Conclusions: There is wide variation in how portfolios are utilized within U.S. residency programs. The challenge for graduate medical education is to create consensus on the definition and purpose of portfolios, such that best practices in portfolio implementation and assessment can be achieved.     

The Use of Standardized Patients within a Procedural Competency Model to Teach Death Disclosure

OBJECTIVE: To design, implement, and evaluate a multi-dimensional, interdisciplinary, educational training module that enables residents to deliver an effective and empathic death disclosure in the emergency setting. The Accreditation Council for Graduate Medical Education (ACGME) "Toolbox of Assessment Methods" to assess competency was adopted as the foundation of this project. METHODS: Sixteen emergency medicine residents, eight postgraduate year 1 (PGY-1) and eight PGY-2, underwent a one-day training and evaluation exercise. The exercise consisted of: 1) a large-group didactic session, 2) a small-group didactic session, and 3) two standardized patient (SP) examinations. Changes in comfort levels, training helpfulness, and competency were measured. Inter-rater agreement between evaluators was examined. RESULTS: Trainees reported improvement in comfort levels and high levels of satisfaction regarding the helpfulness of the training. Good interrater agreement was obtained regarding resident competency to perform a death disclosure between the faculty and SP evaluators [kappa 0.61; 95% confidence interval (95% CI) = 0.33 to 0.88]. However, overall agreement among raters was poor (kappa 0.16; standard error = 0.26). This poor agreement reflected a lack of agreement between resident and SP evaluators (kappa 0.08; 95% CI = 0.16 to 0.33) and resident and faculty evaluators (kappa -0.02; 95% CI = 0.30 to 0.26). CONCLUSIONS: This project used the ACGME "Toolbox of Assessment Methods" to evaluate the competency of emergency medicine trainees to perform an effective and empathic death disclosure. The finding of inconsistent competency assessments by resident self-evaluators compared with those assessments made by faculty and standardized patients have important implications in future curricular design.     

360-degree Feedback: Possibilities for Assessment of the ACGME Core Competencies for Emergency Medicine Residents

The Accreditation Council for Graduate Medical Education (ACGME) has challenged residency programs to provide documentation via outcomes assessment that all residents have successfully mastered the six core competencies. A variety of assessment "tools" has been identified by the ACGME for outcomes assessment determination. Although rarely cited in the medical literature, 360-degree feedback is currently in widespread use in the business sector. This tool provides timely, consolidated feedback from sources in the resident's sphere of influence (emergency medicine faculty, emergency medicine residents, off-service residents and faculty, nurses, ancillary personnel, patients, out-of-hospital care providers, and a self-assessment). This is a significant deviation from both the peer review process and the resident review process that almost exclusively use physicians as raters. Because of its relative lack of development, utilization, and validation as a method of resident assessment in graduate medical education, a great opportunity exists to develop the 360-degree feedback tool for resident assessment.     

Application of the Medical Knowledge General Competency to Emergency Medicine

In February 1999, the Accreditation Council for Graduate Medical Education (ACGME) identified six general competencies as the basic educational goals required by all training programs for their residents. This places emphasis on educational outcome assessment in residency programs and in the accreditation process. A concomitant goal is to have increasingly valid, reliable assessments of the ability of a resident physician to provide safe, evidenced-based, humanistic medical care to their patients. To better define these competencies for the specialty of emergency medicine (EM), the Council of Emergency Medicine Residency Directors (CORD-EM) held a consensus conference in March 2002. This article reports the results of the Consensus Group for the core competency area of medical knowledge. This competency is already being well addressed in residency programs, but there has been inadequate documentation of a resident's knowledge base. The Consensus Group focused on many assessment methods to determine those having the best potential for use in EM programs. Assessment methods felt to be most appropriate for assessment of the medical knowledge base of a resident are presented, as are practical suggestions for incorporating these into EM programs.     

Definitions and Competencies for Practice-based Learning and Improvement

The Outcome Project is a long-term initiative by which the Accreditation Council for Graduate Medical Education (ACGME) is increasing emphasis on educational outcomes in the evaluation of residency programs. The ACGME initiated the Outcome Project to "ensure and improve the quality of graduate medical education." In order to assist program directors in emergency medicine (EM) to begin complying with components of the ACGME Outcome Project, the Council of Residency Directors in Emergency Medicine (CORD-EM) convened a consensus conference in March 2002 in conjunction with several other EM organizations. The working group for the competency of Practice-based Learning and Improvement (PBL) defined the components of PBL as: 1) analyze and assess practice experience and perform practice-based improvement; 2) locate, appraise, and utilize scientific evidence related to the patient's health problems and the larger population from which they are drawn; 3) apply knowledge of study design and statistical methods to critically appraise the medical literature; 4) utilize information technology to enhance personal education and improve patient care; and 5) facilitate the learning of students, colleagues, and other health care professionals in EM principles and practice. Establishing resident portfolios is a preferred method to chronicle resident competence in PBL. Traditional global evaluation of resident performance is de-emphasized. Checklist evaluation is appropriate for assessing any competency that can be broken down into specific behaviors or actions. 360-degree evaluation may be used to assess teamwork, communication skills, management skills, and clinical decision making. Chart-stimulated recall and record review are additional evaluation methods that can be used to assess resident competency in PBL. Simulations and models, such as computer-based scenarios, may be ideal for low-frequency but critical procedures.     

Developing a teaching portfolio.

Teaching portfolios--a compilation of carefully selected materials that describe the faculty member's teaching activities in the classroom, clinical practice, and other settings--are being used more widely for documenting teaching effectiveness in nursing education programs. The purpose of the teaching portfolio is to document the full range of teaching competencies of the faculty member over a period of time. This article describes teaching portfolios and their uses in nursing education, items to include in a portfolio, and how to develop a portfolio.     

An Approach to Fulfilling the Systems-based Practice Competency Requirement

The Accreditation Council for Graduate Medical Education (ACGME)-identified core competency of systems-based practice requires the demonstration of an awareness of the larger context and system of health care, and the ability to call on system resources to provide optimum care. This article describes an approach to teaching and fulfilling the requirement of this core competency in an emergency medicine residency. Beginning residents are oriented to community resources that are important to the larger context of care outside the emergency department. Each resident completes a community project during his or her residency. Readings and discussions concerning community-oriented medical care and the literature of research and injury prevention in emergency medicine precede the project development. Several projects are described in detail. Such projects help to teach not only awareness of the community resources of the greater context of medical practice outside the emergency department, but also how to use those resources. Projects could be a main component of a resident portfolio. This approach to teaching the core competency of systems-based practice is proposed as an innovative and substantial contribution toward satisfying the requirement of the core competency.     

Educating fellows in practice-based learning and improvement and systems-based practice: The value of quality improvement in clinical practice

Introduction

In 1999, the Accreditation Council for Graduate Medical Education identified 6 general competencies in which all residents must receive training. In the decade since these requirements went into effect, practice-based learning and improvement (PBLI) and systems-based practice (SBP) have proven to be the most challenging competencies to teach and assess. Because PBLI and SBP both are related to quality improvement (QI) principles and processes, we developed a QI-based curriculum to teach these competencies to our fellows.

Methods

This experiential curriculum engaged our fellows in our neonatal intensive care unit's (NICU's) structured QI process. After identifying specific patient outcomes in need of improvement, our fellows applied validated QI methods to develop evidence-based treatment protocols for our neonatal intensive care unit.

Results

These projects led to immediate and meaningful improvements in patient care and also afforded our fellows various means by which to demonstrate their competence in PBLI and SBP. Our use of portfolios enabled us to document our fellows' performance in these competencies quite easily and comprehensively.

Discussion

Given the clinical and educational structures common to most intensive care unit–based training programs, we believe that a QI-based curriculum such as ours could be adapted by others to teach and assess PBLI and SBP.     

Using portfolios for clinical practice learning and assessment: the pre-registration nursing student's perspective

Portfolios have been introduced to help to integrate theory and practice and thereby address the issue of the theory-practice divide. Although there has been much theoretical discussion about portfolio use in clinical placements, few studies have focused on the students' perceptions regarding their use. To obtain adult branch pre-registration nursing students' perspectives on using portfolios for their clinical practice learning and assessment, postal questionnaires were sent to 253 diploma of nursing students with a reminder to all students three weeks later. The response rate was 69% (174/253). This paper reports on the qualitative findings of the study, which employed both quantitative and qualitative methods. Although students stated that portfolios helped them in their development of self-awareness and independent learning, they indicated that portfolios do not sufficiently address the assessment of their clinical skills and the integration of theory and practice. They considered that portfolios could be greatly improved in three areas, namely in the conflict between using portfolios for both assessment and learning, the amount of support and guidance students feel they receive with their portfolio use and the portfolio design.     

Developing a Valid Evaluation for Interpersonal and Communication Skills

Objectives The Accreditation Council for Graduate Medical Education requires that residency programs evaluate the acquisition of six general competencies, including Interpersonal and Communication Skills (ICS). To develop a 360-degree method for accomplishing this, a semantic-differentiation matrix for various communication traits for nurses to evaluate physician ICS was developed. The authors sought to determine whether this evaluation method could discriminate between more experienced medical communicators (faculty) and less experienced medical communicators (residents).
Methods A 98-item questionnaire measured several communication dimensions by using an eight-element semantic-differentiation scale. In addition, global assessment ratings assessed nursing perceptions of physician ICS skills. This process was repeated for various clinical scenarios.
Results There were 26 nurse evaluators, 19 emergency medicine (EM) residents (EM2 and EM3), and 30 EM faculty. Each physician received five independent evaluations (total, 245 evaluations). All EM residents (EM2 and EM3) were compared with the EM faculty. All eight items on the semantic-differentiation scale were compared. Likewise, the global assessment scores were compared. In every category, the faculty scored higher (Mann-Whitney U: p < 0.001).
Conclusions An evaluation process integrating a semantic-differentiation matrix was applied to various clinical scenarios (as well as global assessment items) and demonstrated discriminatory results. Faculty physicians scored significantly higher than resident physicians. The ability to provide discriminatory results is a requisite in the development of a valid evaluation process. The described semantic-differentiation matrix and global assessment questions may be valid measurements of ICS.     

Students' perceptions on the use of portfolios in pre-registration nursing education: a questionnaire survey

BACKGROUND: Portfolios encourage both personal and professional development through the process of reflective practice and critical analysis. In clinical placements portfolios are used both as a learning tool and as an assessment tool. OBJECTIVES: To determine nursing students' perceptions on the use and effectiveness of portfolios in their education as well as their perceptions on the use of the portfolio as both an assessment and learning tool in clinical placement. DESIGN: Postal questionnaire survey. SETTINGS: Higher Education Institute in the UK. PARTICIPANTS: 90 first and 84 third year pre-registration diploma of nursing students. METHODS: Postal questionnaires were sent to 131 first year and 122 third year students, with a reminder 3 weeks later. The total response rate was 69% (174/253), with a response rate of 69% (90/131) for first year students and a response rate of 69% (84/122) for third year students. RESULTS: Students felt that portfolios were very time-consuming, causing them a great deal of anxiety, and were not very effective in developing and assessing their learning and competence. Using portfolios for both assessment and learning creates a conflict. Any assessment reduces the honesty and learning value of reflective writing and of the portfolio. Additionally, students become increasingly demoralized with portfolio use over time with experience. The study suggested three main linked reasons for this, namely the conflict between using portfolios for both assessment and learning, the design of the portfolio and the amount of support and guidance students feel they receive with their portfolio use. CONCLUSIONS: Portfolios can be very effective as an assessment and learning tool, but it is essential that both students and mentors receive clear guidelines on and comprehensive support with their use. They should be designed in such as way that they are relevant, clear and user-friendly for both students and mentors.     

Evaluation of Resident Performance and Intensive Bedside Teaching during Direct Observation

Objective: To describe the experience of a residency program in emergency medicine with an intensive observational evaluation of resident performance in the ED.
Methods: Each resident was directly observed and evaluated during a clinical shift four times each academic year: once by each residency codirector and twice by the resident's faculty advisor. The faculty members performed this evaluation outside of "clinical staffing time," shadowing the resident for several hours in the ED during the resident's assigned shift. The resident and assigned faculty member discussed the patients' histories and physical examination findings and developed treatment plans together. Prior to initiation of the observation, the faculty were provided with guidelines for the evaluation of specific skills. Immediate feedback of strengths and deficiencies was provided to the resident.
Results: Subjective evaluations by faculty suggest that new insights into resident clinical strengths and weaknesses were determined using this approach. Objective scoring of resident performance demonstrated heterogeneity of skills between residents as well as inconsistency of skills for specific residents.
Conclusions: The program provided the faculty with protected teaching time, an opportunity to share clinical pearls, and unique insights into resident performance that are not obvious during standard clinical interactions.     

Competency in nursing: a concept analysis

Competency is a topic of great interest to educators and administrators in practice disciplines, particularly health care disciplines such as nursing. This article focuses on the role of competency in nursing. Through a concept analysis process, various elements of competency were assessed. The defining attributes of competency are the application of skills in all domains for the practice role, instruction that focuses on specific outcomes or competencies, allowance for increasing levels of competency, accountability of the learner, practice-based learning, self-assessment, and individualized learning experiences. The learning environment for competency assurance involves the learner in assessment and accountability, provides practice-based learning opportunities, and individualizes learning experiences.     

A model of four hierarchical levels to train Chinese residents’ teaching skills for “practice-based learning and improvement” competency

Objectives: The current study focused on validating a protocol for training and auditing the resident’s practice-based learning and improvement (PBLI) and quality improvement (QI) competencies for primary care. Methods: Twelve second-year (R₂), 12 first-year (R₁) and 12 postgraduate year-1 residents were enrolled into group A, B and C, respectively, as trainees. After three training protocols had been completed, a writing test, self-assessed questionnaire and mini-OSTE and end-of-rotation assessment were used in auditing the PBLI competency, performance and teaching ability of trainees. Results: Baseline expert-assessed PBLI and QI knowledge application tool writing scores were low for the R₁ and R₂ residents. After three training protocols, PBLI and QI proficiencies, performance and teaching abilities were improved to similar levels cross the three training levels of residents based on the expert-assessed writing test-audited assessments and on the faculty and standardized clerk-assessed end-of-rotation-/mini-OSTE-audited assessments. Conclusion: The different four-level hierarchical protocols used to teach group A, B and C were equally beneficial and fitted their needs; namely the different levels of the trainees. Specifically, each level was able to augment their PBLI and QI proficiency. This educational intervention helps medical institutions to train residents as PBLI instructors.     

Helping busy educators become gero-savvy: an approach to developing faculty geriatric expertise

Integrating geriatric competencies into an entry-level nursing curriculum is more than identifying content to be added. The knowledge and ability of faculty to teach these competencies are another important consideration as well as gaining Board of Registered Nursing faculty approval. The authors present an innovative remediation plan that addresses competency in both theory and practice to assist faculty to become gero-savvy.