Ongoing Faculty Evaluations: Developmental Gain or Just More Pain?

PURPOSE: We continue to increase the amount of evaluations to improve the outcomes of our residency programs. Although ongoing faculty evaluations clearly are an important part of faculty development, their value in terms of improving the program needs to be evaluated. The questions asked were as follows: (1) Do faculty evaluations continue to improve the faculty over the course of successive evaluation periods? (2) Are there groups of faculty who would benefit the most from faculty evaluation feedback? (3) Are there any specific objective categories within the evaluation that carry more value and may help to shorten this form? METHODS: Forty-two faculty members were evaluated by 40 surgical residents with an assessment form developed by surgical residents that assessed faculty members by 10 different criteria. The initial set of data was collected, and attending surgeons were given an intervention in the form of a letter detailing how they had been assessed in each of the 10 categories. The attending surgeons were evaluated again 6 months later and were given an intervention in the form of verbal feedback regarding their evaluations. The attending faculty members were then assessed 1 year after that. One way analyses of variance and Fisher Protected Least Significant Difference (PLSD) were used to analyze the resulting data to determine if there were significant differences in the faculty evaluations. A part-whole correlation was performed that correlated the 10 evaluation criteria against the mean score on each evaluation, and partial eta-squared analysis was used to determine which criterion had the largest effect on the overall means. RESULTS: The means for the 42 faculty members as a whole continued to improve from the first to the final evaluation period, with 30 faculty members increasing their mean score (18 significantly) and 12 decreasing their mean score (4 significantly). Seven of the 10 evaluation criteria's means improved sequentially by feedback session. These were (1) Didactic Teaching, (2) Teaching Rounds, (3) Attendance at Didactic Activities, (4) Allows [Resident] Autonomy to Make Independent Decisions, (5) Provides Feedback, (6) Stimulates Critical Thinking with Use of Literature, and (7) Encourages and Maintains an Atmosphere of Professional Mutual Respect for All Members of Health Care Team (Role Model). The faculty group with the lowest evaluations improved significantly more than those of both the middle and the role model group, with the middle and the role model groups improving, but not significantly differently from each other. Of the criteria that improved, only Provides Feedback improved significantly from the first to second and the second to third evaluation periods. The three criteria with the highest correlation coefficients were Role Model (0.76), Provides Feedback (0.75), and Stimulates Critical Thinking (0.74). The results from the partial eta-squared test showed that the criterion with the largest effect size was Provides Feedback (0.28). These analyses indicate that the criterion Provides Feedback was both highly correlated with the average score on a faculty member's evaluation and was more responsible than any other criteria for the overall improvement in the mean evaluation score of the faculty members. CONCLUSIONS: (1) Ongoing faculty evaluations indeed are a powerful tool to improve the faculty as a whole. (2) The faculty members with the lowest evaluations showed the largest amount of improvement. (3) Providing feedback to the residents seems to be the most valued factor by the residents for faculty evaluations and perhaps could become the basis of the evaluation for the most accomplished faculty.     

Comparing resident measurements to attending surgeon self-perceptions of surgical educators

OBJECTIVE: The purpose of this study was to evaluate the initiation and utility of evaluating attending surgeons as educators by resident trainees. Additionally, we were interested in comparing resident measurements to attending self-perceptions. METHODS: A written evaluation form, (utilizing five-point ordinal scale assignments) queried respondents regarding the performance of surgical attendings in the operating room, and other clinical settings. A similar form was distributed to the faculty members, which they used to evaluate themselves. Mean scores were determined, as were comparisons between self-perception and resident assessments. Differences in scores with p values less than 0.05 were considered statistically significant. RESULTS: Thirty-six residents evaluated 23 attendings. Mean assignments by residents of performance in the operating room, other clinical settings, and overall scores for all faculty members as a group were 4.22 +/- 0.04, 4.11 +/- 0.03, and 4.16 +/- 0.03, respectively, with a score of five, generally corresponding to a most favorable rating. When overall scores were analyzed, 10 attendings received scores that differed significantly from those received by their peers, with half of subjects above, and the other half being below the 95% confidence interval. Eighteen (78%) of attendings completed the self-evaluation forms, and of these, 11 (61%) had self-perceptions that differed significantly from overall scores as reported by the residents. CONCLUSIONS: Our evaluation process delineated significant differences among attending faculty members and identified individual strengths and weaknesses. Many educators' self-perceptions differed significantly from resident assessments, and attendings who did not evaluate themselves scored lower than their peers.     

Are you a surgical role model?

PURPOSE: Multiple papers have examined the recent decline in the number of surgical residency applicants. Many have concluded that a lack of role models in surgery is at least partially responsible for this decline. However, to date, the definition of a surgical role model does not exist. This paper defines a surgical role model based on criteria determined by surgery residents. METHODS: Fourth- and fifth-year surgical residents from 1 residency program were asked to collaboratively define 9 characteristics that make a surgical role model. The 9 criteria as defined by the residents were didactic teaching, teaching rounds, attendance at didactic activities, demonstrates skills and decision making in the operating room with confidence and virtuosity, allows [resident] to do procedures according to ability, allows autonomy to make independent decisions, provides feedback, stimulates critical thinking with use of literature, and assists [resident] to find and complete research for publication. Each resident in the program was then given a questionnaire and asked to evaluate each of the 49 teaching attendings on each of the 9 criteria, on a 3-point scale. Finally, residents evaluated each attending based on professionalism and mutual respect on a scale of I Don't Want To Emulate Him/Her, OK, or Role Model. These categories were also assigned a number (1, 2, and 3, respectively). Pearson correlation and stepwise multiple regression were used to determine the relationship between "Role Model" and the 9 criteria. The unit of analysis was the mean rating given each attending on each of the 10 scales. RESULTS: A total of 847 questionnaires were analyzed. Each of the 9 criteria correlated significantly with the Role Model rating (all p < 0.01). The average correlation was 0.73 (range, 0.64 to 0.78). Of the 9 criteria, 4 correlated best with the Role Model, as shown in. The stepwise regression indicates that 3 of the 9 criteria are uniquely associated with the Role Model variable. These 3 criteria are stimulates critical thinking with use of literature, allows autonomy to make independent decisions, and attendance at didactic activities. CONCLUSIONS: All of the 9 criteria are important factors in residents' perception of the Role Model characteristics of attending faculty. A parsimonious, operational definition of the surgical attending role model is one who stimulates the resident to think, gives the resident the opportunity to think and act independently, and is available to the resident.     

Is there gender bias in the evaluation of surgical residents?

Although it might be assumed that the performance of surgical residents is assessed by faculty without regard to gender, no study to date has been undertaken to evaluate this. for 19 years, the Department of Surgery in our institution has had female faculty members to evaluate residents and, with the exception of 1969, female residents to be evaluated. All residents are evaluated by supervisory faculty after each clinical rotation. The same evaluation process has been used throughout this time, assessing residents on a five-point scale (1 = best) in each of six attributes: ethics, judgment, technical skills, knowledge, interpersonal skills, and work habits. The faculty evaluations of general surgery residents from academic years 1967 to 1985 were reviewed. A total of 2356 evaluations of 144 residents (22% female) by 69 faculty members (22% female) was available. For valid comparison, a subset of 702 evaluations was selected. Included were only those evaluations in which residents at the same level of training during the same academic year were evaluated by the same faculty members and in which both the resident and faculty groups had both male and female members. The average scores for male and female residents were nearly identical as determined by male and female faculty members, and no differences were statistically significant. In our program, with more than 20% women residents and faculty members over a 19-year period, no bias related to gender was demonstrated.     

Do Men and Women Use the Same Criteria in Selecting a General Surgery Residency Program? Results of a 20-Year Study in a Non-University Program

BACKGROUND: To assess applicant preferences in general surgery program selection, we surveyed current and former residents of our non-university general surgery residency program over the last 20 years, with particular emphasis on male and female selection preferences. METHODS: Surveys were distributed to current and former categorical residents. Respondents were asked to rate 25 residency criteria using a Likert scale. Responses by males and females were compared using the Mann-Whitney U-test. Results are reported as mean scores, with p-values indicating statistical significance of trends toward higher scores. RESULTS: Of 50 former (76% male, 24% female) and 18 current residents (56% male, 44% female), 56 responded (38 male, 18 female), for an overall response rate of 82%. For both male and female respondents, the top 4 selection criteria by mean average score were identical: variety and number of cases, friendly training environment, camaraderie among residents, and quality of relationships with attendings. Selection criteria that received significantly higher scores among women were camaraderie among residents, the number of female residents, and the number of female attendings (p < 0.05). For men, a suburban location, compensation and benefits, and the reputation of the program director received significantly higher scores (p < 0.05). Gender-related selection preference was most marked for the number of female residents (mean, 2.4 for women vs 1.3 for men) and the number of female attendings (mean, 2.3 for women vs 1.4 for men). These 2 criteria, however, were ranked 20th and 21st (of the 25), respectively, by the female respondents. CONCLUSIONS: The most important selection criteria, regardless of gender, relate to operative experience, training environment, and quality relationships. Gender-based preferences seem to play only a minor role in general surgery program selection.     

Surgical teaching quality makes a difference

BACKGROUND: This study examined the influence of the quality of faculty members' teaching on student performance in a third-year surgery clerkship. METHODS: Eighty-nine third-year students on a surgery clerkship completed preceptor evaluation forms. The faculty member's overall score was the mean of ratings from all the third-year students for whom that faculty member served as preceptor during the year. We examined associations between these ratings and student performance on the National Board of Medical Examiners (NBME) surgery subject examination and clerkship Objective Structured Clinical Examination (OSCE) by using an analysis of covariance that controlled for prior academic achievement [United States Medical Licensure Examination (USMLE) Part I]. RESULTS: The average mean teaching evaluation score was associated with the scores on the NBME surgery subject examination (P = 0.0005). Students with attendings who received poor teaching evaluations performed more poorly on OSCE data-gathering stations than did students with attendings rated as average or good. CONCLUSIONS: The study results indicate that the teaching quality of surgery faculty appears to have an impact on student performance.     

Is USMLE Step 1 score a valid predictor of success in surgical residency?

Background

Many programs rely extensively on United States Medical Licensing Examination (USMLE) scores for interviews/selection of surgical residents. However, their predictive ability remains controversial. We examined the association between USMLE scores and success in surgical residency.

Methods

We compared USMLE scores for 123 general surgical residents who trained in the past 20 years and their performance evaluation. Scores were normalized to the mean for the testing year and expressed as a ratio (1 = mean). Performances were evaluated by (1) rotation evaluations; (2) “dropouts;” (3) overall American Board of Surgery pass rate; (4) first-time American Board of Surgery pass rate; and (5) a retrospective comprehensive faculty evaluation. For the latter, 16 surgeons (average faculty tenure 22 years) rated residents on a 1 to 4 score (1 = fair; 4 = excellent).

Results

Rotation evaluations by faculty and “drop out” rates were not associated with USMLE score differences (dropouts had average above the mean). One hundred percent of general surgery practitioners achieved board certification regardless of USMLE score but trainees with an average above the mean had a higher first-time pass rate (P = .04). Data from the comprehensive faculty evaluations were conflicting: there was a moderate degree of correlation between board scores and faculty evaluations (r = .287, P = .001). However, a score above the mean was associated with a faculty ranking of 3 to 4 in only 51.7% of trainees.

Conclusion

Higher USMLE scores were associated with higher faculty evaluations and first-time board pass rates. However, their positive predictive value was only 50% for higher faculty evaluations and a high overall board pass rate can be achieved regardless of USMLE scores. USMLE Step 1 score is a valid tool for selecting residents but caution might be indicated in using it as a single selection factor.     

Effectiveness of laparoscopic cadaveric dissection in enhancing resident comprehension of pelvic anatomy

BACKGROUND: Anatomic instruction during preclinical years of medical school has been in decline recently. There is evidence that residents already lose a considerable portion of basic anatomic knowledge in the transition from student to clinician, and this deficit is even more dramatic in residents who start their training with a decreased understanding of anatomy. We questioned whether anatomy could be adequately retaught to new residents as surgical anatomy. In an effort to address this deficiency, we developed a program to teach pelvic anatomy in fresh cadavers using a laparoscopic approach. The purpose of this investigation is to determine if such a program is effective in enhancing residents' pelvic anatomy comprehension. STUDY DESIGN: An obstetrics and gynecology residency was divided into intervention (n = 15) and control (n = 13) groups. The intervention was a 4-hour laparoscopic dissection in a fresh cadaver. Outcomes measures included a multiple-choice test, practical exam, faculty evaluation, and satisfaction assessment. The faculty evaluation and satisfaction assessment used a visual analog scale. Univarate and nonparametric analysis were used when appropriate. RESULTS: Initial test scores (p = 0.32), faculty evaluations (p = 0.25), and satisfaction scores (p = 0.17) were similar. Both groups improved their anatomic knowledge based on test scores (p = 0.004) and faculty evaluations (p < 0.001), and final test scores were not significantly different (p = 0.19). Data measured on a 10-cm visual analog scale suggested higher faculty evaluations in the intervention group (14mm versus 10.3mm, (p = 0.23). Similarly there were higher scores on the cadaver test in the intervention group (65% versus 50%), (p = 0.13). The intervention group was significantly more satisfied with their anatomic training (16.1 mm versus-10.1 mm, p = 0.001). CONCLUSIONS: This study did not have sufficient power to demonstrate that a single laparoscopic cadaveric dissection improves cognitive measures of anatomic perception, but suggested that it improves spatial perception of anatomy and is perceived by residents to be a valuable educational approach.     

Resident and faculty differences in student evaluations: implications for changes in a clerkship grading system

Data were collected for 163 students completing a surgical clerkship including scores on patient write-ups, ward performance evaluations, oral examinations, and National Board of Medical Examiners (NBME) Surgery subscores. Oral examination scores and patient write-ups were rated from 74 (failing) to 100 (honors) by faculty members. The ward performance evaluation included ratings on nine components of ward performance from 1 (unsatisfactory) to 4 (superior) and an overall ward score of 74 (failing) to 100 (honors). Similarities and discrepancies in the way that evaluators viewed clerks were found. Weights given to ward component ratings in relationship to the overall ward score were equivalent for faculty members and residents. Residents rated clerks higher than faculty members on five ward components. Faculty ratings were poorly correlated with resident ratings. Faculty members and residents gave different ratings to the same students in contrast to the similarities in which faculty members and residents gave weight to the ward component ratings. Resident ratings were better than faculty ratings in predicting the NBME Surgery subscore. Although these results suggest that residents are better evaluators of a clerk's performance than are faculty members, other studies indicated the opposite. The elimination of participation of either faculty or residents in the grading of students is unwise. Periodic monitoring of evaluation practices is necessary to ensure fairness in grading procedures.     

How learning preferences and teaching styles influence effectiveness of surgical educators

BackgroundEffective surgical educators have specific attributes and learner-relationships. Our aim was to determine how intrinsic learning preferences and teaching styles affect surgical educator effectiveness.MethodsWe determined i) learning preferences ii) teaching styles and iii) self-assessment of teaching skills for all general surgery attendings. All general surgical residents in our program completed teaching evaluations of attendings.ResultsMultimodal was the most common learning preference (20/28). Although the multimodal learning preference appears to be associated with more effective educators than kinesthetic learning preferences, the difference was not statistically significant (80.0% versus 66.7%, p = 0.43). Attendings with Teaching Style 5 were more likely to have a lower “professional attitude towards residents” score on SETQ assessment by residents (OR 0.33 (0.11, 0.96), p = 0.04). Attendings rated their own “communication of goals” (p < 0.001), “evaluation of residents” (p = 0.04) and “overall teaching performance” (p = 0.01) per STEQ domains as significantly lower than the resident’s assessment of these cofactors.ConclusionIdentification of factors intrinsic to surgical educators with high effectiveness is important for faculty development. Completion of a teaching style self-assessment by attendings could improve effectiveness.     

Intraoperative communication of residents with faculty: perception versus reality

INTRODUCTION: During a surgical procedure, an attending may propose a surgical treatment that a resident believes is not the standard of care based on previous readings, discussions with other faculty, or experience. We hypothesized that resident disagreements over intraoperative management are common and are handled differently by residents than faculty members perceive. METHODS: Surveys were sent to 68 residents (82% response) and 45 faculty (91% response) in general surgery and seven subspecialties. Questions were asked to determine the frequency of disagreements and the differences in perceptions of how residents should act when they have disagreements over the intraoperative decisions of attendings. Significance was defined using Fisher's exact test at P < 0.05. RESULTS: Residents reported having major disagreements with faculty over intraoperative decisions as often as monthly (41%) or two to three times per year (37%). Estimates of the frequency of major resident disagreements did not differ between attendings and residents. Fewer residents felt comfortable asking the faculty to discuss their intraoperative decisions than was perceived by faculty members (41% vs 78%, P < 0.001). Residents at all levels are less comfortable than perceived by their mentors in discussing intraoperative decisions even when residents believe that patient injury may result. Residents felt that most of their disagreements occurred because of attending knowledge deficit, while most attendings attributed these differences to resident knowledge deficits. CONCLUSION: Residents frequently have disagreements with faculty over intraoperative management. Education to improve preoperative and intraoperative communication between surgical residents and faculty is needed.     

Impact of a productivity-based compensation system on faculty clinical teaching scores,as evaluated by anesthesiology residents

Study ObjectiveTo determine whether financial incentives given to faculty members for favorable teaching scores improve the quality of clinical education.DesignRetrospective analysis.SettingLarge U.S. academic anesthesiology department.Study Subjects61 academic and 72 clinical faculty members.MeasurementsSince, academic year (AY) 2004, as part of a comprehensive clinical and academic productivity-based compensation system, academic faculty members receiving higher operating room (OR) teaching evaluation scores from the residents have been rewarded financially. Clinical Faculty members also have been rated, but have not received incentives based on scores. Annual averaged OR teaching scores of each faculty member on a 0–9 scale, where 9 = best, were gathered anonymously with faculty classification (academic or clinical). Average overall scores and percentage of faculty with each score category (8.51-9.00, 8.01-8.50, 7.00-8.00, or <7.00) were compared between the pre-implementation (AY2002-AY2003) and post-implementation (AY2004-AY2005) periods. Scores between the academic and clinical faculty also were compared.Main ResultsNo significant difference was noted in the average scores between the pre-implementation and post-implementation periods in a paired comparison (academic: 7.83 ± 0.48 vs 7.85 ± 0.50, P = 0.61; clinical: 7.54 ± 0.75 vs 7.66 ± 0.60, P = 0.21). No statistically significant change was noted in the composition of score categories in the academic (P = 0.63) or clinical faculty (P = 0.20) members. Overall, the academic faculty received significantly higher scores than the clinical faculty (7.84 ± 0.49 vs 7.60 ± 0.67, P = 0.0003).ConclusionsA productivity-based faculty compensation system did not appear to influence faculty OR teaching scores.     

An Internet-based evaluation system for a surgical residency program

BACKGROUND: Efficient and meaningful evaluation of performance is critical to the professional development of trainees in surgical residency programs. Current paper-based evaluation instruments have numerous limitations. We developed an Internet-based evaluation system to more rapidly and efficiently assess the experience of residents, faculty, and rotations. STUDY DESIGN: An on-line evaluation system was designed and implemented in October 1999. Custom evaluations were created for residents, faculty, and rotations. Evaluations were completed via the Internet site from remote locations with standard computers and standard Web browser software. Completed evaluations were automatically available on-line for review and data analysis. Data were analyzed by chi-square analysis with a probability value of less than 0.05 considered statistically significant. RESULTS: Compliance in completion of evaluations improved from 50% to 80% in the initial 6 months of implementation (p < 0.01) with the Web-based system. There was no significant difference between faculty and resident compliance. In evaluation of "ease of use," a total of 612 responses were received over this period with a total average score of 3.5 (5 point scale, 5 = strongly agree). Residents' opinions (average score, 3.69) were slightly more positive than those of faculty (average score, 3.31). Confidentiality was improved over paper-based systems by a detailed security network. CONCLUSIONS: This Internet-based evaluation system is a potentially powerful instrument for evaluating our surgical residency program and making changes to improve the educational experience in a timely and efficient manner.     

Are the Communication and Professionalism Competencies the New Critical Values in a Resident’s Global Evaluation Process?

BACKGROUND: The ACGME requires the assessment of resident competency in 6 domains. Global evaluations covering all 6 competencies are routinely used. Evaluators may be overly influenced by resident affability and availability, thereby resulting in a halo effect. We hypothesized that the Interpersonal Skills and Communications (ICS) and Professionalism (PR) competencies would unduly influence other competency scores. METHODS: General surgery resident evaluations are performed by staff and peers on a rotational basis using competency-based questions. Each question is scored using a 5-point Likert scale. Mean individual composite scores for each competency were calculated and then correlated with other mean composite competency scores. Data from patient evaluations were similarly analyzed. A final correlation of competency scores to ABSITE scores, as an objective, standardized measure of a specific competency, Medical knowledge (MK) was also performed. RESULTS: Results were available for 37 residents (PGY 1-5). There was a significant association between ICS scores and higher scores in MK (r = 0.52, p = 0.004), PR (r = 0.826, p < 0.0001) and patient care (PC) (r = 0.619, p < 0.0001). No correlation, however, was found between patient evaluations of residents and their faculty/peer-based ICS scores. We found no association between ICS scores and improved patient evaluations. Lastly, we found no association between ICS or MK scores and ABSITE scores. CONCLUSIONS: It was difficult to ascertain whether residents with better ICS scores had higher PR, PC, and MK scores because of the halo effect, improper completion of evaluations, or whether those residents were truly performing better clinically. External measures of resident performance did not correlate with faculty/peer evaluations of ICS and PR. Residency programs should consider adopting a more standardized way to objectively evaluate residents.     

A comparison of anesthesiology resident and faculty electronic evaluations before and after implementation of automated electronic reminders

STUDY OBJECTIVE: To develop an automated e-mail reminder system to contact residents and faculty regarding incomplete evaluations. DESIGN, SETTING, INTERVENTION, AND MEASUREMENTS: In the retrospective study, two 9-month periods were evaluated representing pre- and post-introduction of the automated e-mail reminder system. Data collected contained the number of evaluations completed and the rating of residents and faculty at the University of Michigan Health System in 5 different categories on a 5-point scale. MAIN RESULTS: The use of electronic reminders resulted in a nearly 4-fold increase in the number of resident evaluations by faculty from 1050 to 3761. Faculty completing evaluations increased from 40 (61%) to 66 (100%). The mean evaluation scores showed statistically significant but clinically negligible change for resident judgement, interpersonal skills, and intraoperative management. Resident preoperative evaluation and knowledge did not show statistically significant changes. Residents completing evaluations of faculty increased from 244 to 1798, and the number of residents completing evaluations rose from 40 (56%) to 87 (100%). There were no statistically significant changes in the evaluation of faculty except in the category of feedback. CONCLUSION: An automated e-mail reminder system implemented to contact residents and faculty regarding incomplete evaluations for residents and faculty dramatically enhanced participation in the evaluation process.     

Attending, house officer, and medical student perceptions about teaching in the third-year medical school general surgery clerkship

BACKGROUND: There has been declining interest in surgery among medical students and one reason might be the third-year clerkship experience. The aim of this study was to clarify the perceptions and expectations of attendings, residents, and medical students on the clerkship experience. STUDY DESIGN: A survey was distributed to all general surgery attendings, the entire general surgery house staff, and an entire third year medical school class inclusive of the 2001-2002 academic year at a single institution. Statistic analysis consisted of chi-square and Kruskal-Wallis-ANOVA on ranks with Dunn's test for multiple comparisons. A p < 0.05 was significant. RESULTS: Responses were obtained from 59 attending surgeons (50%), 38 surgical residents (32%), and 107 medical students (66%). Of this student cohort, 35% were planning to choose a surgical specialty as a career. Agreement was high among faculty, students, and residents about factors considered important in evaluation, expectations of skills, and level of skills needed before the clerkship. Medical students desired more hours of instruction, believed they performed fewer procedures per week, and thought that feedback was poor compared with the opinions of faculty and residents (p < 0.002). Nearly 50% of medical students believed they were an inconvenience to the service; 30% of house officers and 27% of faculty (p < 0.001) believed this also. Almost all faculty and residents, however, wanted medical students on the service (> 95%). Faculty believed residents did a better job teaching than either the students or residents themselves did (p < 0.001), and students thought that residents were the primary source of education in patient care. CONCLUSIONS: Considerable differences exist between faculty, surgical resident, and medical student perceptions and expectations of medical student education. Structured direct faculty contact, definition of medical student roles on the surgical team, and more consistent feedback can be rapidly improved.     

Medical knowledge: the importance of faculty involvement and curriculum in graduate medical education

PURPOSE: The Accreditation Council for Graduate Medical Education requires that each residency program must demonstrate and document actual accomplishments through objective measures. At University of Washington we identified deficiencies in resident medical knowledge and designed interventions that would assist in improving resident AUA IS scores, using this as a metric to document the outcomes. MATERIALS AND METHODS: In 2001 the University of Washington syllabus and faculty precepted chapter review sessions were formally established. The national AUA IS was used as an objective standardized examination to determine an annual group percentile score. RESULTS: We noted a gradual and consistent increase in the average percentile group score on the national annual AUA IS examinations of our residents. A generalized estimating equation model demonstrated a significant difference between pre-intervention and post-intervention average percentile resident AUA IS scores (p <0.001). Average resident percentile rankings in 2001 to 2004 were 25.6 points higher than the average rankings of residents tested in 1997 to 2000. CONCLUSIONS: We developed and evaluated a cohesive core curriculum designed to improve resident knowledge in urology, as measured by the AUA IS metric. With the active participation of faculty the curriculum enhanced resident education.     

The value of resident teaching to improve student perceptions of surgery clerkships and surgical career choices

BACKGROUND: A fundamental function of attending faculty is to teach and mentor medical students, but the benefit of the resident's role is recognized increasingly. METHODS: Our Standardized Institutional Clinical Clerkship Assessment allows students to rate 27 factors relative to a clinical clerkship. Scores from 1998 to 2005 were used to evaluate our surgical clerkship program and to compare resident and attending teachers. Student surgery career choices also were monitored. RESULTS: Medical students routinely scored residents more highly than attending faculty. Attendings' scores did not improve; however, residents' teaching and overall clerkship scores improved during the study period and paralleled students' increased selection of a surgical career. CONCLUSIONS: Students perceived residents as teachers more than attendings. Residents may have significant influence over students' career choice by their teaching and mentoring activities, which benefit attending efforts.     

Is there a gender bias in milestones evaluations in general surgery residency training?

BackgroundStudies of gender disparity in surgical training have yielded conflicting results. We hypothesize that there is no influence of gender on resident self-evaluation Milestone (SEM) scores and those assigned by the Clinical Competency Committee (CCC).Methods42 residents (25 male & 17 female) and faculty completed 300 Accreditation Council for Graduate Medical Education (ACGME) Milestone evaluations over a 4-year period. Two-way ANOVA, intraclass correlations coefficients, and general linear mixed models were used for analysis.ResultsCCC Milestone scores from 150 evaluations, 51 (34%) for female residents and 99 (66%) for male residents, were compared to corresponding SEM scores. There is a high interrater reliability (self vs. CCC). There was a significant increase in scores with advancing PGY levels (p < 0.001). No effect of gender on Milestones scores (p > 0.05) was noted.ConclusionsWe found no significant differences in Milestones scores between male and female residents as determined by the CCC. Both scores improved significantly as residents progressed in training.     

Improving professionalism: making the implicit more explicit

PURPOSE: Professionalism is one of the most challenging of the Accreditation Council for Graduate Medical Education Competencies to define, teach and evaluate. At University of Washington we assessed whether defining professionalism, and training faculty and residents in the evaluation process would improve professional behavior. MATERIALS AND METHODS: In 2003 the Accreditation Council for Graduate Medical Education Global Resident Competency Rating Form was distributed to faculty after each rotation to evaluate residents. The project included all 16 residents and 18 clinical faculty during a total of 15 rotations for 3 years at 4 Seattle area hospitals. After 21 months a training lecture on professionalism was presented to faculty and residents. Following this intervention all trained faculty completed a Graduate Medical Education Global Resident Competency Rating Form on every resident after each rotation. Three specific professionalism questions from the Graduate Medical Education Global Resident Competency Rating Form were chosen as representations of standard professional behaviors. These questions were used to assess the change in resident professional behavior. RESULTS: Pre-intervention and post-intervention scores were compared. Mean post-intervention scores were higher by 0.4 to 0.5 points for each of the 3 questions (each p <0.01). There was less variation in scores after the intervention, suggesting that 1) residents were more consistent in their professional behavior and/or 2) the faculty observation of resident professional behaviors was more focused. CONCLUSIONS: These results suggest that training faculty and residents in professionalism may have a significant positive influence on improving resident professional behavior as well as the faculty ability to more objectively evaluate resident professional behaviors based on defined standards.