Positive effects of a clinical performance assessment program

Since 1986, there has been a clinical performance assessment program for fourth-year students at the University of Massachusetts Medical School. Students interact with several standardized patients (SPs) and complete other tasks such as interpretation of electrocardiograms and interpretation of X-rays. Scores are generated both by checklists and rating forms completed by the SPs and by paperwork completed by the students at the end of each encounter. Since 1986, students have been asked how frequently they have been observed by faculty and residents as they interacted with actual patients; the students report that such observations have markedly increased. Since 1989, there has been increased feedback to students by the attending faculty during and following clinical rotations. Although it is difficult to claim cause and effect, it is clear that since the inception of this exercise, the faculty have made a conscious effort to improve students' clinical skills by providing increased observation and feedback.     

A systematic approach to faculty development in Women's health: lessons from education, feminism, and conflict theory.

The authors present a strategy for residency faculty development in women's health, the reasons that such a strategy is necessary (e.g., women's health encompasses much more than reproductive and disease issues, and is cross-disciplinary and intrinsic to all of family practice), and their residency program's experience with its development and implementation from 1994 to the present. In creating the program, the residency program's faculty used as a context some lessons learned from family medicine, since the rapid growth of family medicine provides a historical example of dealing with a critical shortage of faculty for new residency programs and the need for a new way to train educators. Also, the faculty reviewed the literature about faculty development in medicine, models of teaching and learning from women's studies, and group theory, specifically the skills concerning conflict and diversity. They used the salient elements from each (which the authors outline) in fashioning their new faculty development program. The resulting program also grew out of focus groups with patients, input from staff, residents, and faculty, and meetings and workshops, including some intense and highly charged discussions in which the faculty participants, both men and women, confronted their own views and biases and worked hard to successfully forge a common and relevant vision of women's health. The program has fostered faculty who are knowledgeable about the diverse educational skills required to teach women's health. They use these skills in all curricular content, thus demonstrating a new way to educate residents as well as faculty.     

A clinical performance exercise for medicine--pediatrics residents emphasizing complex psychosocial skills.

PURPOSE: To assess the skills of internal medicine-pediatrics (med-peds) residents in evaluating and counseling patients with complex psychosocial problems using a clinical performance exercise (CPE). METHOD: The authors designed a 13-station CPE [nine standardized-patient (SP) stations and four non-SP stations]. Eight of the SP stations focused on counseling or assessing complex psychosocial needs, and three were videotaped and analyzed for specific verbal and nonverbal communication skills. Residents completed a written task for each station and SPs completed a checklist on interviewing and communication skills and a 52-item patient's-satisfaction survey. All first- and third-year residents (n = 25) from two academic years participated. RESULTS: The range of the average scores on the nine SP stations was 43-75%. The residents performed better with common problems (newborn hospital discharge instructions and cardiac risk-factor counseling) than with more complex problems that are less often encountered in the institution (HIV counseling), or problems less often recognized (adult survivor of childhood sexual abuse). As expected, third-year residents scored better than did first-year residents on the written "plan" part of the SP stations and on the non-SP stations. Third-year and first-year residents had similar scores, however, on measures of verbal and nonverbal communication and patient's satisfaction, and for gathering data and providing information. CONCLUSION: This is the first performance-based evaluation of residents in a combined med-peds residency program. The stations addressed more complex clinical skills than those reported for objective structured clinical evaluations of residents.     

The Group Objective Structured Clinical Experience: Building communication skills in the clinical reasoning context

Objective

Students are rarely taught communication skills in the context of clinical reasoning training. The purpose of this project was to combine the teaching of communication skills using SPs with clinical reasoning exercises in a Group Objective Structured Clinical Experience (GOSCE) to study feasibility of the approach, the effect on learners’ self-efficacy and attitude toward learning communication skills, and the effect of providing multiple sources of immediate, collaborative feedback.

Methods

GOSCE sessions were piloted in Pediatrics and Medicine clerkships with students assessing their own performance and receiving formative feedback on communication skills from peers, standardized patients (SPs), and faculty. The sessions were evaluated using a retrospective pre/post-training questionnaire rating changes in self-efficacy and attitudes, and the value of the feedback.

Results

Results indicate a positive impact on attitudes toward learning communication skills and self-efficacy regarding communication in the clinical setting. Also, learners considered feedback by peers, SPs, and faculty valuable in each GOSCE.

Conclusion

The GOSCE is an efficient and learner-centered method to attend to multiple goals of teaching communication skills, clinical reasoning, self-assessment, and giving feedback in a formative setting.

Practice implications

The GOSCE is a low-resource, feasible strategy for experiential learning in communication skills and clinical reasoning.     

Implementation of the mini-CEX to evaluate medical students' clinical skills.

OBJECTIVE: The majority of medicine clerkships use faculty and resident summative ratings to assess medical students' clinical skills. Still, many medical students complete training without ever being observed performing a clinical skill. The mini-CEX is method of clinical skills assessment developed by the American Board of Internal Medicine for graduate medical education. The brief, focused encounters are feasible and produce scores with adequate reproducibility if enough observations are made.(1) The mini-CEX has been used in the medicine core clerkship, being performed once to augment feedback by faculty evaluators in the inpatient setting.(2) However, additional study is needed to address at least two feasibility issues if the mini-CEX is to be used as a measurement tool: (1) multiple settings (inpatient and outpatient) and (2) resident-completed evaluations. Our objective was to determine the feasibility of having students receive multiple mini-CEX's in both the inpatient and outpatient settings from resident and faculty evaluators. DESCRIPTION: We introduced the mini-CEX into our nine-week medicine clerkship (six weeks inpatient and three weeks outpatient) in July 2001. The clerkship uses four inpatient clinical sites and 16 outpatient practices. Inpatient faculty rotate on two-week blocks and residents on four-week blocks. At our clerkship orientation, each student (n = 39) received a booklet of ten adapted mini-CEX forms. In the mini-CEX, students are observed conducting a focused history and physical examination and then receive immediate feedback. Students are rated in seven competencies (interviewing, physical examination, professionalism, clinical judgment, counseling, organization, and overall clinical competence) using a nine-point rating scale (1 = unsatisfactory and 9 = superior). Our students were instructed to collect nine evaluations: three from faculty (one every two weeks), three from residents (one every two weeks), and three from their out-patient attendings (one per week). Students and evaluators were asked to rate their satisfaction with the exercise using a nine-point scale (1 = low and 9 = high). Students were asked to turn in their booklets the day of the exam. Prior to implementation, we reviewed the mini-CEX forms and rationale for use with residents and inpatient faculty. Similar information was mailed to outpatient faculty preceptors. DISCUSSION: Booklets were received from 32 students. The mean number of evaluations completed per student was 7.3 (range 2-9), for a total of 232 evaluations. Faculty completed 58% of the evaluations; 68% were from the inpatient setting. The observation and feedback took an average of 21 minutes and 8 minutes, respectively. Satisfaction with the exercise was rated by faculty/residents as 7.2 and by students as 6.8. We believe these findings support the feasibility of collecting multiple mini-CEX assessments from both inpatient and outpatient sites using faculty and resident evaluators. The feasibility of collecting multiple assessments is important if the mini-CEX is to be a reproducible assessment of clinical skills. Having established feasibility, we plan to look at the reproducibility and validity of mini-CEX scores in order to determine if it can be used as a formal means of clinical skills assessment. We also plan to evaluate the impact on the quality and specificity of end-of-clerkship summative ratings.     

A theory-based faculty development program for clinician-educators.

This essay describes the development, implementation, and evaluation of a theory-based faculty development program for physician-educators in medicine and pediatrics at The Cleveland Clinic. The program comprises a 12-hour course (focused on skills in precepting, bedside teaching, leading small-group discussions, giving lectures, designing curricula, and giving effective feedback); onsite coaching of teaching (on wards, in outpatient clinics, or in formal lectures); and innovative projects in clinical medical education. The program advocates and demonstrates a tailored-teaching approach combined with learner-centered, interactive, experiential, and reflective teaching strategies. The evaluation of the program includes satisfaction ratings, self-assessment of teaching competencies, and independent ratings of teaching effectiveness by the participants' trainees (medical students, residents, and fellows). The program is rated highly, shows significant improvements in teaching skills as measured by both participants' self assessments and independent ratings by participants' trainees, and is ongoing.     

Assessment of a 360-degree instrument to evaluate residents' competency in interpersonal and communication skills.

OBJECTIVE: To test the reliability of the 360-degree evaluation instrument for assessing residents' competency in interpersonal and communication skills. METHOD: Ten-item questionnaires were distributed to residents and evaluators at Monmouth Medical Center in Long Branch, New Jersey, in March/April, 2002. The scoring scale was 1-5; the highest score was 50. Data were maintained strictly confidential; each resident was assigned a code. Completed data sheets were collated by category and entered into a spreadsheet. The total and mean scores by each category of evaluator were calculated for each resident and a rank order list created. Shrout-Fleiss (model 2) intraclass correlation coefficients measured reliability of ratings within each group of evaluators. Reliability/reproducibility among evaluators' scores were tested by the Pearson correlation coefficient (p <.05). RESULTS: Intraclass correlation coefficients showed a narrow range, from.85-.54. The highest ranked resident overall ranked high and the lowest was low with most evaluators. The rank order among fellow residents was markedly different from other evaluator categories. Pearson correlation coefficients showed significant correlation between faculty and ancillary staff, (p =.002). Patients as evaluators did show intraclass correlation, but did not correlate significantly with other categories. Scores from colleagues correlated negatively with all other categories of evaluators. CONCLUSIONS: The 360-degree instrument appears to be reliable to evaluate residents' competency in interpersonal and communication skills. Information from the assessment may provide feedback to residents. Areas of improvement identified by the scores would suggest areas for improvement and further ongoing assessment.     

What residents are not learning: observations in an NICU.

In light of the November 1999 report of the Institute of Medicine on medical errors as a leading cause of death and injury, and the July 2000 report of the Accreditation Council for Graduate Medical Education citing violations of work-hour standards for residents and interns, there is a clear need for substantial changes in residency training. The author, a clinical bioethicist, uses his extended observations at a neonatal intensive care unit (NICU) of a major U.S. teaching hospital to outline specific concerns about residents' and interns' training, medical and otherwise, that create unnecessary hazards and other difficulties in the medical care of children. These concerns-which arise from constructive criticisms he makes of specific NICU procedures, methods, approaches, and policies-apply directly to training residents in several areas of medicine and more generally to all residents' training, and echo many of the issues stated in the reports mentioned above. The author maintains that a well-rounded medical education, fostering not only clinical skills but others (e.g., skills in teaching; in communication; in collaborating with nurses, social workers, and others; in working with families; in showing compassion; in dealing with confidentiality issues; in using common sense; in being the patient's advocate), is crucial for producing well-rounded physicians. He emphasizes that in order for such a well-rounded education to occur, the residency program-which in many cases means the attending physicians-must teach and model these varied skills and attitudes to their trainees.     

Part-time residency training in internal medicine: analysis of a ten-year experience.

PURPOSE: To evaluate a ten-year experience (1983-1993) with a part-time residency curriculum. METHOD: In 1994, the authors analyzed the curriculum through interviews with graduates of a part-time residency track, surveys of faculty and graduates of a full-time residency program, and a quantitative comparison of faculty evaluations of those part-time and full-time residents. RESULTS: Both participants and full-time residents supported the part-time track and reported no adverse effect on the residency program as a whole. Analysis of faculty evaluations found that part-time residents scored significantly higher with respect to clinical skills (p = .0005) and humanistic skills (p = .0001), while there was no difference between the groups in leadership or teaching skills. CONCLUSIONS: This part-time residency curriculum provided a highly useful program track for a group of internal medicine residents with concomitant obligations, allowing them to complete their training in an uninterrupted fashion. The part-time structure did not adversely affect clinical competence and may have fostered humanistic attributes. The authors believe that this form of curriculum deserves wider consideration in residency training.     

A pilot survey study to define quality in residency education.

PURPOSE: To begin to define indicators of quality in internal medicine residency training. METHOD: In 1995, through a modified Delphi process, the Association of Program Directors in Internal Medicine's Research Committee developed a questionnaire containing 44 items (34 process and ten outcome indicators). The survey was mailed to all 418 internal medicine program directors and a convenience sample of medical residents. RESULTS: Responding at a rate of 78% (326), program directors rated several indicators as important. These included such faculty characteristics as stability, completeness, supervision, clinical skills, and teaching commitment; institutional support; amount of resident evaluation and feedback; encouragement of lifelong learning; and ability to meet its program goals. There was strong agreement between faculty and residents (r = 0.91). Items rated less important included graduates' selecting academic or generalist careers, residents' caring for elective cardiac catheterization patients, resident community service, training minorities and women, and faculty research. CONCLUSION: These results demonstrate the diversity of opinion of what defines quality in residency education and the emphasis placed on process rather than outcome indicators. To be valid, future endeavors must include all those with a stake in graduate medical education, including accrediting bodies, future employers, and patients.     

Description and evaluation of an EBM curriculum using a block rotation

Background  

While previous authors have emphasized the importance of integrating and reinforcing evidence-based medicine (EBM) skills in residency, there are few published examples of such curricula. We designed an EBM curriculum to train family practice interns in essential EBM skills for information mastery using clinical questions generated by the family practice inpatient service. We sought to evaluate the impact of this curriculum on interns, residents, and faculty.     

Can we rely on simulated patients’ satisfaction with their consultation for assessing medical students’ communication skills? A cross-sectional study

Background

In medical education, teaching methods offering intensive practice without high utilization of faculty resources are needed. We investigated whether simulated patients’ (SPs’) satisfaction with a consultation could predict professional observers’ assessment of young doctors’ communication skills.

Methods

This was a comparative cross-sectional study of 62 videotaped consultations in a general practice setting with young doctors who were finishing their internship. The SPs played a female patient who had observed blood when using the toilet, which had prompted a fear of cancer. Immediately afterwards, the SP rated her level of satisfaction with the consultation, and the scores were dichotomized into satisfaction or dissatisfaction. Professional observers viewed the videotapes and assessed the doctors’ communication skills using the Arizona Communication Interview Rating Scale (ACIR). Their ratings of communication skills were dichotomized into acceptable versus unacceptable levels of competence.

Results

The SPs’ satisfaction showed a predictive power of 0.74 for the observers’ assessment of the young doctors and whether they reached an acceptable level of communication skills. The SPs’ dissatisfaction had a predictive power of 0.71 for the observers’ assessment of an unacceptable communication level. The two assessment methods differed in 26 % of the consultations. When SPs felt relief about their cancer concern after the consultation, they assessed the doctors’ skills as satisfactory independent of the observers’ assessment.

Conclusions

Accordance between the dichotomized SPs’ satisfaction score and communication skills assessed by observers (using the ACIR) was in the acceptable range.These findings suggest that SPs’ satisfaction scores may provide a reliable source for assessing communication skills in educational programs for medical trainees (students and young doctors).Awareness of the patient’s concerns seems to be of vital importance to patient satisfaction.

     

An advanced communication skills course for fourth-year, post-clerkship students.

OBJECTIVE: A novel five-module advanced communication skills course entitled "Doctor-Patient Relationships" was planned and implemented in 2000-01 at the University of British Columbia (UBC). The course was part of the final four-month component of the new MD undergraduate program: Effective Skills for Medical Practice. The goals of the communication skills course were to (1) address problems experienced by the students so far; (2) address deficiencies in achieving the UBC exit competencies; (3) help the students pass the Medical Council of Canada examinations, in particular objectives related to the Considerations of the Legal, Ethical, and Organizational aspects of the practice of medicine (CLEO); and (4) help students prepare for their roles beyond undergraduate medicine (residency, independent practice). DESCRIPTION: The course was developed by an interdisciplinary team (family practice, pathology, pediatrics, psychiatry, surgery) with input from students. The broad strengths and weaknesses of their communication skills training were identified by seven third-year medical students who kept logs over the course of their clinical clerkships to document their learning of communication skills. Analysis of these logs plus feedback meetings with the students revealed attitudinal and skills issues that needed to be addressed in the new course. The goals and principles of the course were in part agreed upon by focus groups with students, attended by faculty observers, to ensure their relevance to students. The first module "Beyond the Mask: Surviving and Thriving in Residency Training" is designed to focus students' attention on the personal relevance of developing excellence in communication skills in preparation for residency training. It includes a video of residents talking about their experiences of communication problems to trigger reflection and discussion. In the remaining four modules the students are required to put communication skills together with their medical knowledge. Each module includes pre-readings, video demonstrations (in sessions 4 and 5), practice with standardized patients (total of 14 scenarios) and structured feedback from SPs, students, and tutor. The themes of the sessions are "Dealing with Emotionally Challenging Patient Situations (informing about bad news), "Compliance and Patient Information," "Informed Consent and Shared Decision Making," and "Difficult Physician-Patient Encounters." Each module lasts two hours. The course was implemented for 120 students, facilitated by 14 tutors (seven to eight students per group). DISCUSSION: Student involvement in many different ways provided an important reality check and made us think about how to present the new course so that it was relevant and interesting to students. Attention to student input was a major contributor to the good evaluations given the course. Students rated the course highly: the relevance of the weekly themes was rated 4.21 on a five-point scale; the effectiveness of the SP interviews, 4.10; the effectiveness of the group discussion and feedback, 4.18; and overall course effectiveness in enhancing communication skills, 3.91. The tutors also rated the course highly, and the students rated the tutors highly. Minor changes will be made to the course next year based on the specific suggestions for improvement, which were identified.     

Teaching the teachers: helping faculty in a family practice residency improve their informatics skills.

Faculty members in family practice residencies are increasingly being asked to help residents develop skills in the use of informatics and evidence-based medicine (EBM). In order to do this successfully the teachers themselves must be skilled in the use of these tools. Recognizing the need for such training, the Maine Medical Center Family Practice Residency Program designed a faculty development project to increase knowledge and skills in the use of information technology. This project, which was carried out in 1999-2001, utilized a multifaceted approach that included improving the residency's technology infrastructure, conducting two instructional workshops, and offering EBM mentoring for preceptors. Faculty members also designed and carried out independent informatics projects. Pre- and post-project assessments of faculty members demonstrated a significant improvement in computer and EBM skills, and informal feedback from residents indicates that these skills have been successfully applied to the faculty members' teaching of residents and their practice of family medicine. This project had a positive impact on the faculty members in the residency program, increasing both their ability to employ information technology in individual and group teaching sessions and their use of EBM in clinical practice. Also, the culture within the residency program has been changed to one of utilizing computers and the Internet as principal resources for up-to-date information.     

Using children as simulated patients in communication training for residents and medical students: a pilot program.

Medical schools are charged with the challenge of teaching effective communication skills, a core competency for residents and medical students. Especially challenging is the task of developing effective methods for training residents and students to communicate with children with mental health issues. The authors describe a pilot program at Mercer University School of Medicine that used pediatric standardized patients (SPs), ages 9-19, to aid in training residents and medical students in complex interviewing skills addressing mental health issues. New curriculum components for four pediatric problems, anorexia nervosa, depression, separation anxiety, and attention deficit hyperactivity disorder (ADHD), were designed and implemented by the authors in 2002-04. The training sessions were evaluated by the participating SPs as well as the residents and medical students in training. The components of the training were a lecture and subsequent practice using pediatric SPs and adults acting as their mothers. Evaluation included the qualitative analysis of SPs' reactions to participation in the training as described during a posttraining-session focus-group, as well as questionnaire responses by residents and medical students. The children role-playing the scenario of a difficult-to-manage situation and their adult "parent" actors voiced strongly positive reactions to participating in training residents and medical students. The reactions of physicians in training were also positive. The authors thus conclude that child and adolescent actors can be effectively used as SPs to train residents and students in complex interviewing skills, even in cases involving children with challenging mental health issues.     

Validation of a global measure of faculty's clinical teaching performance.

PURPOSE: To validate the University of Michigan Global Rating Scale (GRS), a single-item, five-point global measure of faculty members' clinical teaching performances previously shown to be reliable. METHOD: In June 1998, 98 senior medical residents (98% of seniors) from four academic institutions completed the GRS for all teaching faculty at their institutions. Each resident also completed the 26-item Stanford Faculty Development questionnaire (SFDP26) for ten faculty with whom he or she had had teaching contact during residency. The SFDP26 is a validated instrument that measures seven specific aspects of clinical teaching (e.g., communication of goals, feedback). RESULTS: The mean GRS score (SE) was 3.83 (.07). There was no difference in mean GRS or SFDP26 subscale scores across institutions. In a random-effects model that controlled for interrater reliability, correlation coefficients comparing the mean GRS scores and the mean scores for the individual SFDP26 subscales ranged from.86 to.98. CONCLUSIONS: The GRS correlates highly with measures of seven specific aspects of teaching effectiveness among senior medical residents. The GRS is a simple, readily administered measure of faculty's teaching performances that can be used by residency programs as part of an incentive or reward program, to identify teachers as potential candidates for faculty development, or for consideration in promotion decisions.     

Humanism in medical education: a study of educational needs perceived by trainees of three Canadian schools

The authors examined how medical trainees and recent graduates of three Quebec medical schools value 16 dimensions of medical competence classified in four broad categories: clinical, technological, humanistic, and social and preventive. To assess perceived educational needs, the trainees' perceptions of the importance that medical faculties attribute to these same dimensions in the education of physicians were also examined. The survey was conducted in 1986-87 via a questionnaire mailed to 2,030 individuals, including freshmen, juniors, interns, residents, and newly practicing generalists; 80.3% responded. Compared with the views attributed to the faculty, the medical trainees gave more importance to basic diagnostic and therapeutic skills such as the medical history, the physical examination, and the treatment of common diseases. They also valued to a greater extent non-biological dimensions of clinical competence, such as communication with patients, patient education, the social context of disease, and the multidisciplinary nature of patient care, while they ascribed less importance to medical technology and rare diseases. The study raises the question of the relevance of medical education to medical practice by suggesting that those who are preparing themselves to become doctors may not be receiving the training they wish to receive.     

Competency-based residency training in pathology: challenges and opportunities

The Accreditation Council for Graduate Medical Education (ACGME) is requiring that all medical specialties adopt a new paradigm for residency education: competency-based residency education. Competency-based education includes not only the acquisition of knowledge and the demonstration of safe medical practice, but also competency in practice-based learning, practice improvement, interpersonal skills and communication, professionalism, and an awareness of pathology's role in a larger health care system. Implementation of this new training program will require new educational resources and the implementation of new faculty and resident skills and incentives.     

Multicultural training in pediatric and clinical child psychology predoctoral internship programs

Surveyed directors of training of pediatric and clinical child psychology predoctoral internships regarding the extent of multicultural training within their programs. Respondents completed a brief survey assessing faculty and trainee ethnic composition, recruitment of minority trainees, content of training curricula, and perceptions of barriers to multicultural training. Ratings of the importance of multicultural training and the presence of minority interns were directly related to the presence of minority faculty. Recruitment of minority trainees appears to occur largely through program announcements. Didactic seminars on multiculturalism and informal resolution of intercultural conflicts predominate internship settings with less emphasis on formal structures for addressing multicultural conflicts. Implications for internship training are discussed.     

Effectiveness of a brief training program in relational/communication skills for medical residents

ObjectiveTo assess the effectiveness of a brief training program in relational/communication skills (RCS) for medical residents.MethodsThis longitudinal study enrolled 64 medical residents who participated in a RCS training program in small groups. Teaching was based on interviews with standardized patients and reflective practice. Video-recorded consultations were coded according to the Verona-Coding-Definitions-of-Emotional-Sequences (VR-CoDES) and a coding system developed to assess ten communication skills for breaking bad news. The outcome measures were: independent raters’ score in RCS for breaking bad news and the percentage of providing space and empathic responses, by comparing baseline (T1) skills with those after three-days (T2) and three-months (T3).ResultsAfter the training program residents provided more space for further disclosure of cues and concerns according to VR-CoDES definitions. There were significant improvements in seven of the ten communication skills for breaking bad news. All of these improvements were observed either at T2 or at T3.ConclusionThis study demonstrates the effectiveness of a brief RCS training program designed to improve medical residents’ ability to respond appropriately to patients’ cues and concerns and to conduct a breaking bad news encounter.Practice implicationsBrief RCS training programs adopting multiple approaches, should be offered as mandatory during residency programs.