Faculty and resident evaluations of medical students on a surgery clerkship correlate poorly with standardized exam scores

Background

The clinical knowledge of medical students on a surgery clerkship is routinely assessed via subjective evaluations from faculty members and residents. Interpretation of these ratings should ideally be valid and reliable. However, prior literature has questioned the correlation between subjective and objective components when assessing students' clinical knowledge.

Methods

Retrospective cross-sectional data were collected from medical student records at The Johns Hopkins University School of Medicine from July 2009 through June 2011. Surgical faculty members and residents rated students' clinical knowledge on a 5-point, Likert-type scale. Interrater reliability was assessed using intraclass correlation coefficients for students with ≥4 attending surgeon evaluations (n = 216) and ≥4 resident evaluations (n = 207). Convergent validity was assessed by correlating average evaluation ratings with scores on the National Board of Medical Examiners (NBME) clinical subject examination for surgery. Average resident and attending surgeon ratings were also compared by NBME quartile using analysis of variance.

Results

There were high degrees of reliability for resident ratings (intraclass correlation coefficient, .81) and attending surgeon ratings (intraclass correlation coefficient, .76). Resident and attending surgeon ratings shared a moderate degree of variance (19%). However, average resident ratings and average attending surgeon ratings shared a small degree of variance with NBME surgery examination scores (ρ² ≤ .09). When ratings were compared among NBME quartile groups, the only significant difference was for residents' ratings of students with the lower 25th percentile of scores compared with the top 25th percentile of scores (P = .007).

Conclusions

Although high interrater reliability suggests that attending surgeons and residents rate students with consistency, the lack of convergent validity suggests that these ratings may not be reflective of actual clinical knowledge. Both faculty members and residents may benefit from training in knowledge assessment, which will likely increase opportunities to recognize deficiencies and make student evaluation a more valuable tool.     

Disparities between resident and attending surgeon perceptions of intraoperative teaching

Background

This study aimed to assess attending surgeon and resident recall of good and poor intraoperative teaching experiences and how often these experiences occur at present.

Methods

By web-based survey, we asked US surgeons and residents to describe their best and worst intraoperative teaching experiences during training and how often 26 common intraoperative teaching behaviors occur in their current environment.

Results

A total of 346 residents and 196 surgeons responded (51 programs; 26 states). Surgeons and residents consistently identified trainee autonomy, teacher confidence, and communication as positive, while recalling negatively contemptuous, arrogant, accusatory, or uncommunicative teachers. Residents described intraoperative teaching behaviors by faculty as substantially less frequent than faculty self-reports. Neither sex nor seniority explained these results, although women reported communicative behaviors more frequently than men.

Conclusions

Although veteran surgeons and current trainees agree on what constitutes effective and ineffective teaching in the operating room, they disagree on how often these behaviors occur, leaving substantial room for improvement.     

Variations in the perception of trauma-related complications between attending surgeons, surgery residents, critical care nurses, and medical students

Background

The morbidity and mortality conference (M&M) is a key component of the performance improvement process. The audience response system (ARS) has been shown to improve audience participation and promote more truthful responses in various settings. We implemented the ARS in our trauma M&M and evaluated the responses we received from different categories of participants.

Methods

This was a prospective observational study undertaken between November 2006 and July 2007. Cases were graded based on the American College of Surgeons scoring system. We evaluated the responses of attending surgeons, residents, critical care nurses, and medical students using the ARS.

Results

We had 695 responses for complications and 936 responses for deaths. Residents consistently scored complications as more severe than other groups (P = .03). There was no difference in the scoring of deaths.

Conclusions

Surgical residents assign higher severity to trauma-related complications than other groups when using an anonymous automated scoring system.     

Identification of teaching excellence in operating room and clinic settings

BACKGROUND: A system for obtaining learner feedback on surgical faculty teaching is a program-specific resource for recognizing faculty accomplishments as well as being a requirement of the Accreditation Council for Graduate Medical Education (ACGME). This investigation uses 5 years of feedback from residents to identify surgical teaching behaviors that define teaching excellence. METHODS: Between 1995 and 1999 full-time surgeons in a division of general surgery were evaluated biannually by every resident on their services, using two 10-item Likert scales to assess frequency of performing selected teaching behaviors. Response categories ranged from 0 (does not demonstrate) to 4 (demonstrates the behavior to a very high degree). Mean scores > or =3.7 (1 SD above the mean) were categorized as evidence of superior teaching, whereas mean scores < or =2.4 (1 SD below the mean) were categorized as mediocre. Residents wrote statements identifying teaching strengths. RESULTS: There were 753 individual resident assessments of 16 faculty. The overall mean rating for operating room and clinic teaching was 3.1, with 24% of the ratings > or =3.7 and 14% of the ratings < or =2.4. For operating room, discriminant behaviors were: demonstrates sensitivity to resident learning needs (3.85 versus 1.62, P <0.01) and provides direct feedback (3.60 versus 1.27, P <0.01). Residents' statements yielded themes tied to superior teaching: demonstrates technical expertise, allows resident participation, and maintains a learning climate of respect. CONCLUSIONS: A resident-based teaching assessment system can offer a reasonable and valid form of feedback to academic surgeons. The use of mixed methods to identify teaching behaviors that characterize excellence informs faculty of how they are perceived as educators and provides examples of specific behaviors that merit commendation.     

Surgery residents recognized less frequently over time by medical students: A twelve year study of the Arnold P. Gold Humanism and Excellence in Teaching Award

Background

There is a decreasing institutional percentage of surgical resident recipients of The Arnold P. Gold Humanism and Excellence in Teaching Award over time. The hypothesis was that this trend was a national phenomenon.

Teaching medical students and residents how to inject local anesthesia almost painlessly

The objective of the present study was to determine whether it is possible to consistently and reliably teach medical students and resident learners how to administer local anesthetics in an almost painless manner. Using the published technique, 25 consecutive medical students and residents were taught how to inject local anesthetics for carpal tunnel release by watching the senior author perform the technique once. The learner then independently administered the anesthesia to the next patient who then scored the learner’s ability to inject the local anesthetic from a pain perspective. The teaching technique is demonstrated in an accompanying online video. The learners were consistently capable of administering local anesthetics with minimal pain. During the injection process, the patients only felt pain once (‘hole-in-one’) 76% of the time. This pain was attributed to the first 27-gauge needle poke. The other 24% of the time, patients felt pain twice (eagle) during the 5 min injection process. All 25 patients rated the entire pain experience to be less than 2/10. Eighty-four per cent of the patients indicated that the experience was better than local anesthetic given at the dentist’s office. Medical students and residents can quickly and reliably learn how to administer local anesthesia for carpal tunnel release with minimal pain to the patient.     

Spaced education improves the feedback that surgical residents give to medical students: a randomized trial

Background

Surgery residents teach medical students; feedback is one critical teaching skill. We investigated whether feedback given by surgery residents to students could be improved through an online spaced education program.

Methods

Surgery residents were randomized to receive either a weekly spaced education e-mail during a 9-month period containing teaching bullets on how to provide effective feedback, or no intervention. Medical students rated the frequency and quality of feedback they received from the residents.

Results

Students reported 45% (67 of 149) of the spaced education residents gave frequent feedback, compared with 31% (55 of 175) of control residents (relative risk [RR], 1.43; P = .016). Students reported resident feedback was “helpful in their learning” in 92% (132 of 143) of their evaluations of spaced education residents, compared with 82% (132 of 161) of their evaluations of control residents (RR, 1.13; P = .01).

Conclusions

Educational programs using feedback bullets e-mailed weekly can significantly improve the frequency and quality of feedback that surgical residents provide medical students.     

Lights, camera, surgery: a novel pilot project to engage medical students in the development of pediatric surgical learning resources

Background/Purpose

It is important to engage junior medical students in the pediatric surgical environment to showcase possible future career choices. Our aim was to assess how the students valued their experience in the realm of clinical learning, exposure to surgical careers, and development of skill sets necessary for creating learning resources.

Methods

A novel pilot project entitled Lights, Camera, Surgery engaged 13 medical students in the production of instructional videos of basic surgical procedures. An electronic survey questionnaire allowed the students to provide formal feedback on the project outcomes.

Results

Of the students who responded, 87.5% expressed appreciation of the enhanced clinical experience. All of the students either agreed or strongly agreed that the project afforded them valuable leadership experience, practical skills in creating educational learning resources, and opportunities to explore careers in surgery. All of the students either agreed or strongly agreed that the project allowed them to gain valuable skills in educational video production. The project videos are now available as educational tools.

Conclusions

Engaging medical students in the production of surgical videos potentially improves leadership skills and promotes the use of educational resources while affording them opportunities to explore pediatric surgery as a future career choice.     

Training general surgery residents in pediatric surgery: educational value vs time and cost

Background

This study evaluated the educational value of pediatric surgery rotations, the likelihood of performing pediatric operations upon completing general surgery (GS) residency, and time and cost of training GS residents in pediatric surgery.

Methods

A survey was administered to GS residents that evaluated the pediatric surgery rotation and anticipated practice intentions. A retrospective analysis (2005-2006) of operative times for unilateral inguinal hernia repair, bilateral inguinal hernia repair, and umbilical hernia repair was also performed. Procedure times were compared for operations performed by a pediatric surgeon with and without GS residents. Cost analysis was based on time differences.

Results

General surgery residents (n = 19) considered the pediatric surgery rotation to have high educational value (4.7 ± 0.6 of 5) with extensive teaching (4.6 ± 0.7) and operative experience (4.4 ± 0.8). Residents listed pediatric surgery exposure, operative technique, and observed work ethic as most valuable. Upon graduation, residents expect to perform pediatric operations 2 to 3 times annually. Thirty-seven percent of residents felt competent to perform appendectomy (patients >5 years), 32% appendectomy (3-5 years), 21% gastrostomy (>1 year), and 11% inguinal herniorrhaphy (>1 year). Operative times and costs were significantly higher in operative procedures performed with a GS resident.

Conclusion

General surgery residents considered pediatric surgery as a valuable educational experience. Residents anticipate performing pediatric operations a few times annually. Training GS residents in pediatric surgery increased operative time and cost. This information may be useful in determining the appropriate setting for resident education as well as budget planning for pediatric surgical practices.     

Students’ expectations on the surgery clerkship exceed those of residents and faculty

Purpose

The perceptions and expectations of students on the surgery clerkship were evaluated and compared with those of surgical residents and faculty.

Methods

A voluntary and anonymous survey was distributed to third year medical students, general surgery residents, and surgery faculty. Statistical analysis was performed using the χ² and Cochran-Mantel-Haenszel tests (P value <0.05 was significant).

Results

Medical students, surgery residents and faculty largely agreed on student responsibilities during morning rounds. However, more students (96.9%) than residents (86.2%) and faculty (77.3%) believed they should be writing progress notes, and fewer students (85%) than residents (100%) and faculty (95.5%) thought they should be taught during morning rounds (P < 0.01 and P = 0.04, respectively). The expected and actual educational experience on the surgical clerkship was similar. The amount of instruction provided each week by residents and faculty was similar to the amount expected by students; each group believed that surgical residents were good educators. Students, residents, and faculty agreed on many essential skills for a student to learn by the clerkship’s end; however, more students (85.8%) than residents (58.6%) and faculty (68.2%) viewed identifying surgical complications as an essential skill (P < 0.01).

Conclusions

Overall, the expectations of the medical students, surgical residents, and surgical faculty on the surgery clerkship were well matched. However, there were some instances where the students thought they should be more involved in patient care than did the residents and the faculty. It is important for surgical educators to be aware of these discrepancies to fully maximize the educational experience of medical students.     

Differences in operative self-assessment between male and female plastic surgery residents: A survey of 8,149 cases

BackgroundPrevious studies show female residents tend to underrate and male residents to overrate their own performance. We sought to determine if plastic surgery resident trainee self-evaluations differ by resident sex.MethodsWe extracted Operative Entrustability Assessment (OEA) data for plastic surgery programs from MileMarker?, a program capable of storing assessment data for CPT-coded procedures. Complete OEAs contain a trainee self-assessment and attending surgeon assessment. We used simple statistics and linear regression to assess differences, stratifying by trainee sex and post-graduate year (PGY).ResultsWe analyzed 8149 OEAs from 3 training programs representing 64 residents (25% female) and 51 attendings. Compared to attending assessments, both male and female residents significantly underrated their performance during PGY1. However, during PGY2-6 male residents’ self-evaluations were significantly higher and female residents’ self-evaluations significantly lower than their attending evaluations.ConclusionsResults demonstrated female plastic surgery residents underestimated and male residents overestimated their performance. Further studies are needed to determine reasons for these differences.     

Assessing the 16 hour intern shift limit: Results of a multi-center,mixed-methods study of residents and faculty in general surgery

Background

The study explores how residents and faculty assess the ACGME's 16-h limit on intern shifts.

本科内科护理学四位一体多元化教学模式研究

目的构建有利于培养护生综合能力的内科护理学教学模式。方法选择本科护生106人,分为实验班与普通班各53人。内科护理学教学过程中,普通班采用常规课堂讲授、实验、见习及实习的教学模式;实验班将4个阶段整体计划,采用情境设置、角色扮演、问题式、小组讨论式等多元化教学模式。结果实验班理论考试、综合技能操作成绩显著优于普通班(均P<0.01);对教学效果的评价9项中有8项优于普通班(均P<0.01)。结论内科护理学教学过程中采取四位一体多元化教学模式有利于提高教学效果和护生的综合能力。     

沈阳汉族医学生骨强度指数和体成分变化及相关性分析

目的调查分析沈阳市汉族医学生骨强度指数和体成分变化特点及两者之间的相关性,明确影响骨强度指数的主要因素。方法 2017年4月选取沈阳汉族医学生1 949例(男生721例,女生1 228例),年龄19~21岁,采用超声骨密度仪和生物电阻抗分析仪,分别测量学生右足跟骨骨强度及体成分指标,采用多元逐步回归法研究骨强度指数与体成分各指标之间的关系。结果沈阳汉族医学生男、女生骨强度指数均在19岁时最高,之后20岁和21岁时骨强度指数均降低,且只有女生差异显著(P0.01);男生各年龄组骨强度指数值均大于女生(P0.01)。汉族男生各年龄组脂肪含量、体脂率及水肿系数均显著低于同年龄组女生结果(P0.01);除了19岁年龄组中腰臀围比,男生各年龄段体重、总水分、体质量指数(BMI)、腰臀围比、骨质、蛋白质、肌肉量、瘦体重、骨骼肌含量及基础代谢等指标均显著高于同年龄组女生结果(P0.01或P0.05)。多元线性逐步回归分析显示,汉族医学生骨强度指数与BMI和骨骼肌含量正相关(P0.01),与脂肪含量负相关(P=0.000)。结论沈阳汉族医学生骨强度指数在20岁和21岁时显著下降,女生骨强度指数下降情况较为严重,是骨质疏松的重点防护对象;BMI和骨骼肌含量是预防骨质疏松的保护性因素,而脂肪含量是引发骨质疏松的危险因素。     

Surgery service learning in preclinical years improves medical student attitudes toward surgery,clinical confidence,and social determinants of health screening

IntroductionPatient health literacy is paramount for optimal outcomes. The Service Learning Project (SLP) aims to merge the need for patient education with the desire of medical students for early clinical experience.MethodsThis pretest-posttest study examined the effect of the SLP on medical students. First-year students spent 8 h each month educating inpatients and screening for social determinants of health (SDH). Students completed a 30-question survey pre- and post-SLP, and longitudinally throughout medical school. We used t-tests to assess changes in attitudes towards surgery, clinical confidence, and SDH screening.ResultsStudent self-perceived value on surgical teams increased significantly (2.49 vs 3.63 post-SLP, p < 0.001), as did their confidence interacting with patients (3.66–4.14, p = 0.002) and confidence assessing for SDH (3.13–4.75, p = 0.002). 100% of students continued to assess for SDH on clerkships.ConclusionsThe SLP model improves medical students’ skills and confidence working with patients and addressing SDH.     

Assessing impact of not attending lectures on osteopathic medical student performance: Brief survey of the literature and proposed future research

It has been anecdotally reported that osteopathic medical students often decrease their attendance of didactic lectures to increase time to study for the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA). There is no strong evidence supporting the impact of attending vs. not attending didactic lectures on osteopathic medical student performance in courses during the first two years of osteopathic medical education. This paper provides a brief survey of the literature. The paper also presents a proposal for future research to assess the impact of not attending didactic lectures on student performance. The proposed study would provide needed evidence toward answering this question in the osteopathic medical school environment.     

Upcoming female preponderance within surgery residents and the association of sex with the surgical career choice in the new millennium: results from a national survey in France

Background

Women are increasingly entering the medical studies. We aim to investigate surgical residents' career decision making among surgical residents and to assess the presence of women.

Methods

An “electronic questionnaire” of self-assessment was distributed to the 2,500 current French surgical residents. Items analyzed included population characteristics, demographics, educational experiences, and choice of career.

Results

There was a response rate of 31.2% (n = 779). The male/female ratio was 1.1 with 370 women (47.5%). The mean age was 28 ± 2 years. All surgical specialties were represented. Sex was significantly associated with the choice of surgical specialty (P < .05). Female residents had a shorter working time per week than male residents (P < .001). Sex was significantly linked with future career plans as female residents expected to work in a public hospital (P = .005) and as male residents expected to work in private practice (P = .001).

Conclusions

Female surgical residents are more populous than ever among surgical residents. Sex appears to be linked with where and how the residents expect to work in the future.