Demonstration of Portfolios to Assess Competency of Residents

Residency educators are identifying approaches to measure resident competence. Portfolios are well suited since they require work already completed as part of patient care where competency must be demonstrated. This paper describes assessment of the reliability and validity of portfolios in a psychiatry residency program. This was a cross-sectional study across 4years of residency education. Using guidelines, 18 residents assembled portfolios containing five entries chosen from 13 skills. Trained raters scored the portfolios. Residents and faculty were interviewed about their perceptions. Generalizability results indicated five entries and two raters were sufficient for relative decisions. Six entries or a third rater would be sufficient for absolute decisions. Portfolio scores tended to improve with years of training and correlated with psychiatric knowledge but not clinical performance. Residents and faculty identified benefits to assembling a portfolio. Portfolios incorporate tasks embedded in the residency to provide evidence of resident competency. The results support that the score is reliable and valid.     

A comprehensive collaborative patient safety residency curriculum to address the ACGME core competencies

CONTEXT: Patient safety currently receives only scant attention in most residency curricula. Safety is a subject that transcends the US Accreditation Council for Graduate Medical Education's 6 core competencies. OBJECTIVE: To design and implement a new patient safety curriculum in collaboration with the Schools of Nursing and Pharmacy, in such a way as to address all 6 competencies. SETTING AND PARTICIPANTS: The curriculum applies to a university-based family medicine residency programme with 45 residents at 5 sites, including urban, suburban and rural sites. CURRICULUM DESIGN: The curriculum includes introductory workshops for faculty and residents, a series of didactic courses, individual portfolios and a series of small group exercises including chart reviews, case presentations and a longitudinal quality improvement project. The activities are run by a multidisciplinary team. OUTCOME MEASURES: Main outcome measures include assessment of resident performance in curriculum activities and in an annual objective structured clinical examination (OSCE) that includes standardised patient interviews, simulations and a written examination. Programme evaluation will include comparison of OSCE performance with that at a neighbouring residency. RESULTS: Residents identified safety problems and system-based solutions using a safety journal. Cases of polypharmacy were identified using journals and chart reviews, and medication changes proposed and discussed. At resident practice sites, residents identified safety priorities based on a staff survey and proposed system-based solutions. Results of the OSCE will be presented elsewhere. CONCLUSIONS: A new patient safety curriculum was successfully introduced into a family medicine residency. The curriculum integrates patient safety into residents' daily activities and incorporates input from the disciplines of nursing and pharmacy so as to help build more effective clinical teams and inculcate a culture of safety.     

Tying it all together? A competency-based linkage model for family medicine

Residency faculty in all specialties will be required by the Accreditation Council for Graduate Medical Education (ACGME) to fully implement competencies into residency programs by 2006. Understanding the new requirements is complicated by having several sets of guiding documents from different sources, including the general competencies of the ACGME, the Residency Review Committee for Family Practice requirements, the competencies developed by the Society of Teachers of Family Medicine, and the Recommended Curriculum Guidelines for Family Practice Residents by the American Academy of Family Physicians. A competency linkage model brings together the various guidelines and shows specifically how they are related. This model helps family practice residency faculty better understand the guiding expectations for their programs and develop more appropriate learning objectives and assessment methods.     

The 'Collaborative Care' curriculum: an educational model addressing key ACGME core competencies in primary care residency training

AIM: The 'Collaborative Care' curriculum is a 12-month senior resident class project in which one evidence-based clinical guideline is designed, implemented and evaluated in our residency practice. This curriculum specifically addresses three of the six Accreditation Council for Graduate Medical Education (ACGME) core competencies: Practice-Based Learning and Improvement, Interpersonal and Communication Skills and System-Based Practices. Additionally, the project enhances the quality of patient care within the model family practice centre in a family practice residency. METHODS: During the project, the third-year residency class selects the disease, develops the clinical guideline, leads its implementation and guides the evaluation process. Select faculty members serve as mentors and coach the resident class through each phase of the project. Specific educational objectives are developed for each content area: evidence-based medicine, clinical guideline development, continuous quality improvement and team leadership. A series of seminars are presented during the project year to provide 'just-in-time' learning for the key content and skills required for each step in the project. By working together to develop the practice guideline, then working with nurses and allied health staff to implement the guideline and review its effectiveness, the resident team gains competence in the areas of practice-based learning and improvement, interpersonal and communication skills and system-based practices. RESULTS: The self-reported level of resident confidence in skill acquisition for each content area was measured for each resident at the time of graduation from the residency programme. Results from the first 2 years of this curriculum are reported (resident n = 12), and demonstrate a high level of physician confidence in the skills addressed and their utility for future practice. CONCLUSIONS: The senior resident seminar and team project model reported here creates learning experiences that appear to address at least three of the ACGME general competency expectations: practice-based learning and improvement, interpersonal communication skills, and systems-based practice. From the initial resident feedback, this educational model seems to establish a high level of physician confidence in the skills addressed and their utility for future practice.     

Conditions for successful reflective use of portfolios in undergraduate medical education

Aim Portfolios are often used as an instrument with which to stimulate students to reflect on their experiences. Research has shown that working with portfolios does not automatically stimulate reflection. In this study we addressed the question: What are the conditions for successful reflective use of portfolios in undergraduate medical education? Methodology/research design We designed a portfolio that was aimed at stimulating reflection in early undergraduate medical education, using experiences described in the medical education literature and elsewhere. Conditions for reflective portfolio use were identified through interviews with 13 teachers (mentors), who were experienced in mentoring students in the process of developing their portfolios. The interviews were analysed according to the principles of grounded theory. Results The conditions for successful reflective use of portfolios that emerged from the interviews fell into 4 categories: coaching; portfolio structure and guidelines; relevant experiences and materials, and summative assessment. According to the mentors, working with a portfolio designed to meet these conditions will stimulate students' reflective abilities. Conclusion This study shows that portfolios are a potentially valuable method of assessing and developing students' reflective skills in undergraduate medical training, provided certain conditions for effective portfolios are recognised and met. Portfolios have a strong potential for enhancing learning and assessment but they are very vulnerable and may easily lead to disappointment. Before implementing portfolios in education, one should first consider whether the necessary conditions can be fulfilled, including an appropriate portfolio structure, an appropriate assessment procedure, the provision of enough new experiences and materials, and sufficient teacher capacity for adequate coaching and assessment.     

Continuity of care in a family practice residency program. Impact on physician satisfaction

One of the primary benefits of continuity of care is its influence upon patient and physician satisfaction. This prospective pilot study involved a cohort of 14 second- and third-year family practice residents and 4 full-time faculty at a community hospital-based family practice residency in Cleveland, Ohio. Rates of continuity that physicians experience were calculated using the usual provider continuity (UPC) measure of continuity, and were correlated with physician satisfaction with outpatient care using a practice satisfaction scale (PSS) developed specifically for this purpose. Residents and faculty were also asked to rank order the importance of several aspects of outpatient care, including continuity of care. The results indicate that both residents and faculty value continuity of care highly compared with other aspects of outpatient care. The average continuity rates were 59% for second-year residents, 54% for third-year residents, and 82% for faculty. The UPC continuity measure correlated highly with the PSS scores (corrected r2 = .55; P less than .001). The data support the hypothesis that continuity of care with patients is an important determinant of resident and faculty physician satisfaction with their outpatient experience.     

Documentation of inpatient clinical experience of family practice residents: a manual system.

A simple, efficient, and inexpensive system for quantitatively documenting the range of clinical experience of individual residents on inpatient rotations is described. Information provided by this system can be used by program faculty to make informed educational decisions concerning both the design of rotations and individual resident program planning. The data are also useful to residency graduates to document their clinical experience when applying for hospital privileges.     

House officers in general medicine: their perception of educational value of rotation components

Summary: As curriculum planners in general medicine residency training programmes we were concerned about house officers' anecdotal reports that hospital work requirements often overshadow individual learning goals. After each of five rotations, we asked residents to identify the educational 'usefulness' of certain rotation components which can be included in three categories; team members, work-related activities and educational events. Of 165 surveys distributed, 127 (77%) were returned. Data were analysed by residency year and by all years combined. The mean overall perception of learning was 3.9 out of a possible 5 points suggesting that residents do find some learning value. Results suggest that different residency years vary as to the significance of specific educational components. The importance of faculty/resident relationships to residents' perceptions of learning value was highlighted in particular.     

Portfolios in medical education: why do they meet with mixed success? A systematic review

CONTEXT: The move towards competence-based medical education has created a need for instruments that support and assess competence development. Portfolios seem suitable but mixed reports of their success are emerging. METHODS: To examine the effectiveness of portfolios, we searched PubMed and EMBASE using the keyword 'portfol*', PsychInfo and ERIC using the keywords 'portfol*' and 'medical education' and references of retrieved papers for empirical studies on portfolios in all phases of medical education. Thirty of 1939 retrieved papers met the inclusion criteria and were analysed. Data were collated against the research question, number of subjects, design, setting, findings and limitations, purpose and content, mentoring and assessment. We analysed impact using a modified version of Kirkpatrick's hierarchy. RESULTS: Because differences across studies precluded statistical meta-analysis, the data were analysed by context, goals and procedure. Positive effects were strongest in undergraduate education. Important factors for success were: clearly communicated goals and procedures; integration with curriculum and assessment; flexible structure; support through mentoring, and measures to heighten feasibility and reduce required time. Moderately good inter-rater reliability was reported and global criteria and discussions among raters were beneficial. Formative and summative assessment could be combined. Without assessment, portfolios were vulnerable to competition from other summative assessment instruments. CONCLUSIONS: For portfolios to be effective in supporting and assessing competence development, robust integration into the curriculum and tutor support are essential. Further studies should focus on the effectiveness and user-friendliness of portfolios, the merits of holistic assessment procedures, and the competences of an effective portfolio mentor.     

The reliability of assessment criteria for undergraduate medical students' communication skills portfolios: the Nottingham experience

INTRODUCTION: Some educators have argued that portfolios should not be assessed summatively because there is little evidence supporting the reliability of their assessment. This study aims to determine the reliability of assessment criteria used for a portfolio at the University of Nottingham. METHODS: Two independent analysts assessed a random sample of portfolios (n = 100, 49.5%) using criterion-referenced assessment. Students' performances were examined against subjective items in five areas: 1) portfolio structure, 2) level of critical reflection, 3) level of skills development, 4) use of documentary evidence, and 5) use of relevant literature. These subjective judgements were later converted into quantitative scales ranging from 0 to 3 so that interrater reliability could be established. The level of agreement between the two analysts for the total percentage score was established using an intraclass correlation coefficient and for the individual items using weighted kappa coefficients. RESULTS: The level of agreement between the two raters for the total percentage score was 0.771 (95% CI = 0.678, 0.840), as measured by an intraclass correlation coefficient. The levels of agreement between the two raters for the individual items of the assessment criteria ranged from kappa=0.359 (item 3) to kappa=0.693 (item 4). DISCUSSION: This study provides some support for the summative assessment of portfolios. The findings suggest that discussion and negotiation between independent assessors can enhance the reliability of assessment criteria. Therefore, medical educators are encouraged to use such procedures in the summative assessment of portfolios.     

How Prospective School Health Educators Can Build a Portfolio to Communicate Professional Expertise

ABSTRACT: Portfolios — purposeful and selective collections of authentic evidence regarding teaching and learning — occur increasingly in K-12 classroom, preservice teacher education, and university initiatives to enhance teaching effectiveness of faculty. In addition to being used to improve instruction, portfolios can help teacher candidates with initial employment. This article describes an organized set of materials prospective school health educators should develop to accurately represent their newly acquired skills and competencies. (J Sch Health. 1997;67(6):228–231)     

The Montana Model: Integrated Primary Care and Behavioral Health in a Family Practice Residency Program

To address the local health care needs of both patients and primary care providers in Montana, an integrated primary care and behavioral health family practice clinic was developed. In this paper we describe our experience with integrating mental health and substance abuse services into a primary care setting (a community health center) while simultaneously teaching family practice physicians to take the lead in providing these services. The Deering Community Health Center in Billings, Montana, is a Federally Qualified Health Center serving a largely low-income patient population. The medical care at the clinic is provided primarily by the faculty and residents of the Montana Family Medicine Residency. The teaching model was founded on the belief that improved care will result when physicians have increased comfort with, and are able to enjoy the challenges of, patients with mental illnesses. The enhanced longitudinal curriculum incorporates mental health across the 3 years of the family practice residency. Unique characteristics of this model include staffing and the concurrent delivery of a high volume mental health service while teaching family practice resident physicians and the faculty to integrate this competency into their primary care practices.     

A systematic review of resident‐as‐teacher programmes

Context Residents in all disciplines serve as clinical teachers for medical students. Since the 1970s, there has been increasing evidence to demonstrate that residents wish to teach and that they respond positively to formal teacher training. Effective resident‐as‐teacher (RaT) programmes have resulted in improved resident teaching skills. Current evidence, however, is not clear about the specific features of an effective RaT programme. Objectives This study was performed in order to investigate the effectiveness of RaT programmes on resident teaching abilities and to identify the features that ensure success. Methods of assessment used to ascertain the effectiveness of RaT programmes are also explored. Methods The literature search covered the period between 1971 and 2008. Articles focusing on improving resident teaching skills were included. Each study was reviewed by two reviewers and data were collected using a standard abstraction summary sheet. Study outcomes were graded according to a modified Kirkpatrick's model of educational outcomes. Results Twenty‐nine studies met review inclusion criteria. Interventions included workshops, seminars, lectures and teaching retreats. Twenty‐six studies used a pre‐ and post‐intervention outcome comparison method. Subjective outcome measures included resident self‐evaluation of teaching skills or evaluation by medical students, peers and faculty members. Objective outcome measures included written tests, evaluation of teaching performance by independent raters and utilisation of objective structured teaching examinations. One study objectively measured learning outcomes at the level of medical students, utilising the results of an objective structured clinical examination. Overall resident satisfaction with RaT programmes was high. Participants reported positive changes in attitudes towards teaching. Participant knowledge of educational principles improved. Study methodologies allowed for significant risks of bias. Conclusions More rigorous study designs and the use of objective outcome measures are needed to ascertain the true effectiveness of RaT programmes. Future research should focus on determining the impact of RaT programmes on learning achievement at the level of medical students.     

Learning management systems: technology to measure the medical knowledge competency of the ACGME

AIMS: We report how the learning management system (LMS) Web Course Tools (WebCT) was used to design, implement and evaluate the web-based course "Principles of Ambulatory Paediatrics", taken by paediatric residents during an ambulatory block rotation. This report also illustrates how WebCT can be used to measure the medical knowledge competency required by the Accreditation Council for Graduate Medical Education (ACGME). METHODS: Eighty paediatric residents completed a 1-month outpatient rotation between July 1, 2001 and June 30, 2002. During this rotation residents were required to complete 4 modules in asthma, otitis media, gastroenteritis and fever, respectively. Each module was evaluated using a standard questionnaire. RESULTS: Completion rates for the required modules ranged from 64-72%. Residents in all 3 years of training showed improvement between the pre- and post-test scores for each module, except for postgraduate Year 2 residents in the asthma module. Most residents somewhat agreed, agreed or strongly agreed that the module components were useful and that the experience of completing the modules would improve their ability to take care of patients. CONCLUSIONS: The LMS WebCT is an innovative and adaptable approach for designing a web-based course for primary care education in paediatrics. The LMS addresses the educational needs of both a clinical division and a residency programme. The LMS also provides an information technology infrastructure to measure the medical knowledge competency required by the ACGME.     

Opening the black box of clinical skills assessment via observation: a conceptual model

Medical Education 2011: 45 : 1048–1060 Objectives This study was intended to develop a conceptual framework of the factors impacting on faculty members’ judgements and ratings of resident doctors (residents) after direct observation with patients. Methods In 2009, 44 general internal medicine faculty members responsible for out‐patient resident teaching in 16 internal medicine residency programmes in a large urban area in the eastern USA watched four videotaped scenarios and two live scenarios of standardised residents engaged in clinical encounters with standardised patients. After each, faculty members rated the resident using a mini‐clinical evaluation exercise and were individually interviewed using a semi‐structured interview. Interviews were videotaped, transcribed and analysed using grounded theory methods. Results Four primary themes that provide insights into the variability of faculty assessments of residents’ performance were identified: (i) the frames of reference used by faculty members when translating observations into judgements and ratings are variable; (ii) high levels of inference are used during the direct observation process; (iii) the methods by which judgements are synthesised into numerical ratings are variable, and (iv) factors external to resident performance influence ratings. From these themes, a conceptual model was developed to describe the process of observation, interpretation, synthesis and rating. Conclusions It is likely that multiple factors account for the variability in faculty ratings of residents. Understanding these factors informs potential new approaches to faculty development to improve the accuracy, reliability and utility of clinical skills assessment.     

The fate of medical students with different levels of knowledge: Are the basic medical sciences relevant to physician competence?

Purpose — This study was designed to test the hypothesis that an early gap in knowledge of sciences basic to medicine could have a sustained negative effect throughout medical school and beyond.Method — A longitudinal prospective study of 4,437 students who entered Jefferson Medical College between 1972 and 1991 was conducted in which the students were divided into three groups. Group I consisted of 392 who failed at least one of the basic sciences courses in the first year of medical school, Group II was comprised of 398 who did not fail but had low first-year grade-point averages; and 3,647 of the remaining sample were included in Group III. The groups were compared on retention and dismissal rates, medical school assessment measures, scores on medical licensing examinations, ratings of clinical competency in residency, board certification rates, and faculty appointments.Results — Significant differences were observed among the three groups confirming the hypothesis that students' level of knowledge in sciences basic to medicine early in medical school could predict later performance during medical school and beyond. Implications for early diagnosis of academic deficiencies, in better preparation of medical students, and in the assessment of clinical competency are discussed.     

Are family practice residents able to interpret electrocardiograms?

Background: Teaching electrocardiogram (ECG) interpretation is a recommended component of the family practice residency curriculum. Published information concerning the ECG interpretation ability of residents is sparse. This study sought to ascertain the base line knowledge of family practice residents' ECG interpretation skills and extent of improvement after one year of training. Methods: A 15 ECG examination was administered to 38 PG-1 and 14 upper level residents at 5 residency programs at the beginning of the academic year and to residents at the authors' program at the end of the academic year. Pre-test scores among the five programs were compared using an analysis of variance (ANOVA). Pre-test and post-test scores were compared using a paired randomization test. Results: No difference was found between average scores from each site, or between the beginning and end of the academic year. Residents were more likely to misinterpret items such as myocardial infarction, myocardial ischemia, and a trial fibrillation. Conclusions: Residents in family practice have considerable deficiencies in ECG interpretation skills. Further studies are needed to determine effective ECG teaching curricula. This revised version was published online in September 2006 with corrections to the Cover Date.