Faculty and resident evaluations of medical students on a surgery clerkship correlate poorly with standardized exam scores

Background

The clinical knowledge of medical students on a surgery clerkship is routinely assessed via subjective evaluations from faculty members and residents. Interpretation of these ratings should ideally be valid and reliable. However, prior literature has questioned the correlation between subjective and objective components when assessing students' clinical knowledge.

Methods

Retrospective cross-sectional data were collected from medical student records at The Johns Hopkins University School of Medicine from July 2009 through June 2011. Surgical faculty members and residents rated students' clinical knowledge on a 5-point, Likert-type scale. Interrater reliability was assessed using intraclass correlation coefficients for students with ≥4 attending surgeon evaluations (n = 216) and ≥4 resident evaluations (n = 207). Convergent validity was assessed by correlating average evaluation ratings with scores on the National Board of Medical Examiners (NBME) clinical subject examination for surgery. Average resident and attending surgeon ratings were also compared by NBME quartile using analysis of variance.

Results

There were high degrees of reliability for resident ratings (intraclass correlation coefficient, .81) and attending surgeon ratings (intraclass correlation coefficient, .76). Resident and attending surgeon ratings shared a moderate degree of variance (19%). However, average resident ratings and average attending surgeon ratings shared a small degree of variance with NBME surgery examination scores (ρ² ≤ .09). When ratings were compared among NBME quartile groups, the only significant difference was for residents' ratings of students with the lower 25th percentile of scores compared with the top 25th percentile of scores (P = .007).

Conclusions

Although high interrater reliability suggests that attending surgeons and residents rate students with consistency, the lack of convergent validity suggests that these ratings may not be reflective of actual clinical knowledge. Both faculty members and residents may benefit from training in knowledge assessment, which will likely increase opportunities to recognize deficiencies and make student evaluation a more valuable tool.     

Cutting too deep? Assessing the impact of a shorter surgery clerkship on students' clinical skills and knowledge

Background

The aim of this study was to compare the performance of students completing an 8-week versus a 6-week surgery clerkship on an objective structured clinical examination (OSCE) and the National Board of Medical Examiners (NBME) clinical science surgery examination.

Methods

One hundred fifteen students from the 8-week clerkship and 99 from the 6-week clerkship were included. Performance on a summative OSCE was assessed using behaviorally anchored checklists. NBME exams were graded using the NBME's standard scaled scores. Results were compared using 2-tailed, independent-samples, unequal-variance t tests.

Results

Mean OSCE scores for the 8-week and 6-week curricula were not statistically different. Mean NBME scores also did not statistically differ. Six-week students performed significantly better in the specific OSCE subdomains of blood pressure, orthostatic blood pressure, rectal exam, and fecal occult blood test.

Conclusions

Overall OSCE and NBME exam performance did not differ between 8-week and 6-week surgery clerkship students.     

Accuracy and content of medical student midclerkship self-evaluations

Background

Midclerkship self-evaluations (MCSEs) require students to reflect on their knowledge, skills, and behaviors. We hypothesized that MCSEs would be consistent with supervisor midpoint evaluations during a surgical clerkship.

Methods

MCSEs of 153 students who completed our surgery clerkship in 2 academic years were compared with supervisor midclerkship evaluations. The quantitative domains of the MCSE and supervisor evaluation were compared for accuracy. Identified areas of strengths and weakness were evaluated for thematic consistency.

Results

Student MCSE scoring was accurate across evaluated domains most of the time; when students were inaccurate, they tended to underrate themselves. Students and supervisors most often identified cognitive skills as areas for improvement and noncognitive skills predominated as student strengths.

Conclusions

Medical students can accurately identify their strengths and weaknesses in the context of an MCSE. Based on these findings, knowledge acquisition and application by medical students in the clinical setting should be emphasized in undergraduate medical education.     

Students’ expectations on the surgery clerkship exceed those of residents and faculty

Purpose

The perceptions and expectations of students on the surgery clerkship were evaluated and compared with those of surgical residents and faculty.

Methods

A voluntary and anonymous survey was distributed to third year medical students, general surgery residents, and surgery faculty. Statistical analysis was performed using the χ² and Cochran-Mantel-Haenszel tests (P value <0.05 was significant).

Results

Medical students, surgery residents and faculty largely agreed on student responsibilities during morning rounds. However, more students (96.9%) than residents (86.2%) and faculty (77.3%) believed they should be writing progress notes, and fewer students (85%) than residents (100%) and faculty (95.5%) thought they should be taught during morning rounds (P < 0.01 and P = 0.04, respectively). The expected and actual educational experience on the surgical clerkship was similar. The amount of instruction provided each week by residents and faculty was similar to the amount expected by students; each group believed that surgical residents were good educators. Students, residents, and faculty agreed on many essential skills for a student to learn by the clerkship’s end; however, more students (85.8%) than residents (58.6%) and faculty (68.2%) viewed identifying surgical complications as an essential skill (P < 0.01).

Conclusions

Overall, the expectations of the medical students, surgical residents, and surgical faculty on the surgery clerkship were well matched. However, there were some instances where the students thought they should be more involved in patient care than did the residents and the faculty. It is important for surgical educators to be aware of these discrepancies to fully maximize the educational experience of medical students.     

International surgical clerkship rotation: perceptions and academic performance

Background

Concerns about international training experiences in medical school curricula include the effect on student learning. We studied the educational effect of an international elective integrated into a traditional third-year (M3) surgical clerkship.

Methods

A 1-week surgical elective in Haiti was available to M3 students during the conventional 8-week surgical clerkship each year for the 4 academic years 2008 to 2011. The authors collected student and surgeon perceptions of the elective using a mixed-methods web-based survey. Statistical analysis compared the academic performance of participating M3s relative to nonparticipating peers.

Results

Twenty-eight (100%) students (41 trip weeks) and 3 (75%) surgeons responded. Twenty-five (89%) students believed the elective provided appropriate clinical training. Surgeon responses were consistent with students' reported perceptions.Strengths included unique clinical experiences and close interactions with faculty. Criticisms included recurring overwhelming clinical responsibilities and lack of local provider involvement.Academic performance of participants versus nonparticipants in the same clerkship term were statistically insignificant.

Conclusions

This study demonstrates the feasibility of integrating global health experiences into traditional medical student clinical curricula. The effects on less tangible attributes such as leadership skills, fostering teamwork, and cultural competency require future investigation.     

Palliative and end-of-life care training during the surgical clerkship

Background

In 2000, the Liaison Committee on Medical Education required that all medical schools provide experiential training in end-of-life care. To adhere to this mandate and advance the professional development of medical students, experiential training in communication skills at the end-of-life was introduced into the third-year surgical clerkship curriculum at Wright State University Boonshoft School of Medicine.

Materials and methods

In the 2007–08 academic year, 97 third-year medical students completed six standardized end-of-life care patient scenarios commonly encountered during the third-year surgical clerkship. Goals and objectives were outlined for each scenario, and attending surgeons graded student performances and provided formative feedback.

Results

All 97 students, 57.7% female and average age 25.6 ± 2.04 y, had passing scores on the scenarios: (1) Adult Hospice, (2) Pediatric Hospice, (3) Do Not Resuscitate, (4) Dyspnea Management/Informed Consent, (5) Treatment Goals and Prognosis, and (6) Family Conference. Scenario scores did not differ by gender or age, but students completing the clerkship in the first half of the year scored higher on total score for the six scenarios (92.8% ± 4.8% versus 90.5% ± 5.0%, P = 0.024).

Conclusions

Early training in end-of-life communication is feasible during the surgical clerkship in the third-year of medical school. Of all the scenarios, “Conducting a Family Conference” proved to be the most challenging.     

Porcine wet lab improves surgical skills in third year medical students

Background

Medical students desire to become proficient in surgical techniques and believe their acquisition is important. However, the operating room is a challenging learning environment. Small group procedural workshops can improve confidence, participation, and performance. The use of fresh animal tissues has been rated highly among students and improves their surgical technique. Greater exposure to surgical procedures and staff could positively influence students' interest in surgical careers. We hypothesized that a porcine “wet lab” course for third year medical students would improve their surgical skills.

Methods

Two skills labs were conducted for third year medical students during surgery clerkships in the fall of 2011. The students' surgical skills were first evaluated in the operating room across nine dimensions. Next, the students performed the following procedures during the skills lab: (1) laparotomy; (2) small bowel resection; (3) splenectomy; (4) partial hepatectomy; (5) cholecystectomy; (6) interrupted abdominal wall closure; (7) running abdominal wall closure; and (8) skin closure. After the skills lab, the students were re-evaluated in the operating room across the same nine dimensions. Student feedback was also recorded. Fifty-one participants provided pre- and post-lab data for use in the final analysis.

Results

The mean scores for all nine surgical skills improved significantly after participation in the skills lab (P ≤ 0.002). Cumulative post-test scores also showed significant improvement (P = 0.002). Finally, the student feedback was largely positive.

Conclusions

The surgical skills of third year medical students improved significantly after participation in a porcine wet lab, and the students rated the experience as highly educational. Integration into the surgery clerkship curriculum would promote surgical skill proficiency and could elicit interest in surgical careers.     

American College of Surgeons/Association for Surgical Education medical student simulation-based surgical skills curriculum needs assessment

Background

Simulation can enhance learning effectiveness, efficiency, and patient safety and is engaging for learners.

Methods

A survey was conducted of surgical clerkship directors nationally and medical students at 5 medical schools to rank and stratify simulation-based educational topics. Students applying to surgery were compared with others using Wilcoxon's rank-sum tests.

Results

Seventy-three of 163 clerkship directors (45%) and 231 of 872 students (26.5%) completed the survey. Of students, 28.6% were applying for surgical residency training. Clerkship directors and students generally agreed on the importance and timing of specific educational topics. Clerkship directors tended to rank basic skills, such as examination skills, higher than medical students. Students ranked procedural skills, such as lumbar puncture, more highly than clerkship directors.

Conclusions

Surgery clerkship directors and 4th-year medical students agree substantially about the content of a simulation-based curriculum, although 4th-year medical students recommended that some topics be taught earlier than the clerkship directors recommended. Students planning to apply to surgical residencies did not differ significantly in their scoring from students pursuing nonsurgical specialties.     

Surgical clerkship or medical clerkship first: Does it make a difference?

Introduction

This study compares NBME surgical clerkship scores of students who completed their medicine clerkship before their surgical clerkship with the performance of those who had not previously completed their medical clerkship.

Teaching across the continuum: variations in rankings and valued teaching components between surgery residents and medical students

Background

Surgical faculty teach medical students and residents within the same environment; however, each group may require different teaching methods. The aim of this study was to identify teaching components valued by these sets of learners.

Methods

Teaching evaluations for 43 surgical faculty members who received yearly evaluations from students and residents were analyzed. Highest and lowest ranked educators in medical student evaluations were traced to corresponding rankings in resident evaluations. Thematic analysis was conducted on written comments.

Results

Educators rated the highest in the medical student group were spread out among the residents' rankings, with several educators being rated the lowest. Similar patterns were observed in reverse for faculty rated highly by residents.

Conclusions

Residents and medical students value faculty teaching using different criteria, with residents more focused on operative autonomy and medical students focused on approachability. Using one group to define best teaching methods is insufficient as learners value different optimal attributes.     

Expanding surgical clerkships to remote community sites: the success of the Washington, Wyoming, Alaska, Montana, and Idaho experience

Background

The purpose of this analysis was to determine if the surgical clerkship model and site affect educational outcomes and student postclerkship perceptions.

Methods

Data from University of Washington students participating in surgical clerkships at traditional/academic or community/apprentice sites across Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI) between 2005 and 2007 were gathered retrospectively. Comparisons of final examination scores as well as postclerkship student evaluations of the educational experience were made between traditional and community training sites.

Results

The mean final examination scores at WWAMI sites were significantly higher than those at traditional academic sites. Furthermore, WWAMI sites were rated higher with respect to time spent by faculty in direct observation, quality of the clerkship as a whole, and overall contribution to medical education.

Conclusions

Community surgical clerkship sites remote from an academic institution can provide an excellent learning experience for students.     

A focused curriculum in surgical oncology for the third-year medical students

Background

Educating medical students in surgical subspecialty fields can be challenging, and the optimal timing and curriculum remain unknown. Despite advocacy for earlier exposure, competing core clerkship rotations often leave little time for subspecialty fields. We report our experience with a novel, short, and focused curriculum in surgical oncology for the third-year medical students.

Methods

A 2-wk (2009–2010) and a 4-wk (2010–2011) curriculum in surgical oncology were developed for the third-year students at a tertiary-referral cancer center, including formal didactics, rotation in clinical service of students' choosing (breast, gastrointestinal, endocrine, or melanoma), and case-based learning and presentation. Paired pre- and postrotation questionnaires were prospectively completed, including 20 items assessing knowledge and four items assessing experience. Grading was anonymous, and change in score was assessed by Wilcoxon signed-rank test.

Results

Paired questionnaires from 47 students (2-wk rotation, n = 26; 4-wk rotation, n = 21) showed a median improvement of three points (21.4%) from pre- to posttests (P < 0.001). The improvement did not differ by the length of rotation or by the specific clinical service. Nearly all (93%) reported a positive and inspiring experience. The most valuable avenue of learning was reported as the time spent with resident or fellow or attending (92%), followed by self-directed reading (62%) and didactic lectures (28%).

Conclusions

A short and focused curriculum in surgical oncology, including structured didactics and clinical rotation, had positive impact for the third-year students. Given the increasing work-hour limits, it is important to note that the time spent in the clinical setting continues to be ranked as the most educationally valuable by medical students.     

Influence of medical students' past experiences and innate dexterity on suturing performance

Background

Medical students often site their ability to excel at technical tasks as justification for choosing surgery as a career path. We sought to investigate how medical students' dexterity skills and past experiences correlated with suturing performance.

Methods

Sixty-four 3rd-year medical students were surveyed about previous experiences that involved manual dexterity. Technical skills were then measured using a validated test of manual dexterity and subcuticular closure of a pig's foot incision. Spearman's rank correlation coefficients determined correlation between variables.

Results

Previous experiences, self-assessment of dexterity, prior suturing, and current interest in surgery did not significantly correlate with manual dexterity or suturing skill scores. Innate manual dexterity score was the only significant correlating factor to suture skill score (Spearman's rank correlation coefficient = .336; P = .007).

Conclusions

Innate manual dexterity skills are predictive of initial surgical suturing performance regardless of past student experiences. Interventions aimed at improving early surgical technique should be optimally focused on dexterity training.     

Standardized and quality-assured video-recorded examination in undergraduate education: informed consent prior to surgery

Background

Communication skills combined with specialized knowledge are fundamental to the doctor–patient relationship in surgery. During a single-station video-recorded objective structured clinical examination (VOSCE), students were tasked with obtaining informed consent. Our aim was to develop a standardized and quality-assured assessment method in undergraduate education.

Methods

One hundred fifty-five students in their fifth year of medical school (78 videos) participated in a summative VOSCE within the framework of the teaching module “Operative Medicine.” They prepared for three clinical scenarios and the surgical procedures involved. The examination comprised participants having to obtain informed consent from simulated patients, video recording their performance. Students were assessed by two independent raters, the background of one of whom was nonsurgical. Results were statistically tested using SPSS.

Results

Students' scores were all beyond the pass mark of 70%, averaging 91.0% (±4.0%), 88.4% (±4.4%), and 87.0% (±4.7%) for the appendectomy, cholecystectomy, and inguinal hernia repair checklist, respectively. Most items (68%–89% of the checklists) were found to have fair to excellent discrimination values. Cronbach's α values ranged between 0.565 and 0.605 for the individual checklists. Interrater agreement was strong (Pearson correlation coefficient = 0.80, P < 0.01; intraclass correlation coefficient 2.1 = 0.78).

Conclusions

The VOSCE is both feasible and reliable as a method of assessing student communication skills and the application of clinical knowledge while obtaining informed consent in surgery. This method is efficient (flexible rating outside normal working hours possible with reductions in administrative load) and may be used for high-stakes evaluation of student performance.     

Do medical student's surgical examination scores correlate with performance markers?

Background

Some medical school training consists of oral examinations.

Methods

We conducted a 9-year review of third-year medical student examinations including oral examinations, National Board of Medical Examiners Surgery Subject Examination (SSE, ie, shelf), and United States Medical Licensing Examinations Step 1 and Step 2.

Results

Step 1 showed a moderate to strong association with Period 1 orals (Somers' D = .297, P < .001), but not Period 2 orals (Somers' D = .048, P = .053). Period 1 orals (percentage) had a strong association with SSE (Somers' D = .356, P < .001) and Step 2 (Somers' D = .368, P < .001). Period 2 orals (pass/fail) suggested a positive, but not statistically significant, association with SSE (Somers' D = .334, P = .085) and Step 2 (Somers' D = .370, P = .055). Step 1 shows a strong association with SSE (Somers' D = .490, P < .001). SSE showed a strong association with Step 2 (Somers' D = .506, P < .001).

Conclusions

Orals can be used to identify students who may have difficulty passing the SSE. Step 1 can be used to identify students at risk of poor performance on the SSE, and SSE can be used to identify students at risk for poor performance on Step 2.     

Is there a place for medical students as teachers in the education of junior residents?

Background

The aim of this study was to investigate a novel resident education model that turns the traditional surgical hierarchy upside down, termed a “reverse” peer-assisted learning curriculum.

Methods

Thirty surgical topics were randomized between medical students and chief residents on each clinical team, with 1 topic being presented briefly during morning rounds. An exam evaluating junior residents' knowledge of these topics was administered before and after 1 month of presentations. A questionnaire was distributed to evaluate the junior residents' perceptions of this teaching model.

Results

Thirty-four residents participated. There was a significant improvement in the mean examination score (54 vs 74, P < .05). No significant difference was noted in the mean score differentials of topics presented by either the medical students or the chief resident (21 vs 18, P = .22). More than 80% of the residents responded positively about the effectiveness of this exercise and agreed that they would like to see this model used on other services.

Conclusions

This study confirms the hypothesis that medical students can teach surgical topics to junior residents at least as effectively as their chief residents.     

When surgeons decide to become surgeons: new opportunities for surgical education

Background

When surgeons decide to become surgeons has important implications. If the decision is made prior to or early in medical school, surgical education can be more focused on surgical diseases and resident skills.

Methods

To determine when surgeons – compared with their nonsurgical colleagues – decide on their medical path, residents in surgery, internal medicine, obstetrics and gynecology, pediatrics, psychiatry, and emergency medicine were surveyed. Timing of residency choice, demographic data, personal goals, and reason for residency choice were queried.

Results

A total of 234 residents responded (53 surgical residents). Sixty-two percent of surgeons reported that they were “fairly certain” of surgery before medical school, 13% decided during their preclinical years, and 25% decided during their clerkship years. This compares with an aggregate 40%, 7%, and 54%, respectively, for the other 5 residency specialties. These differences were statistically significant (P = .001). When the 234 residents were asked about their primary motivation for choosing their field, 51% pointed to expected job satisfaction and 44% to intellectual curiosity, and only 3% mentioned lifestyle, prestige, or income.

Conclusions

General surgery residents decide on surgery earlier than residents in other programs. This may be advantageous, resulting in fast-tracking of these medical students in acquiring surgical knowledge, undertaking surgical research, and early identification for surgical residency programs. Surgical training in the era of the 80-hour work week could be enhanced if medical students bring much deeper knowledge of surgery to their first day of residency.     

Higher clinical performance during a surgical clerkship is independently associated with matriculation of medical students into general surgery

Background

The purpose of our study was to determine the predictive impact of individual academic measures for the matriculation of senior medical students into a general surgery residency.

Methods

Academic records were evaluated for third-year medical students (n = 781) at a single institution between 2004 and 2011. Cohorts were defined by student matriculation into either a general surgery residency program (n = 58) or a non–general surgery residency program (n = 723). Multivariate logistic regression was performed to evaluate independently significant academic measures.

Results

Clinical evaluation raw scores were predictive of general surgery matriculation (P = .014). In addition, multivariate modeling showed lower United States Medical Licensing Examination Step 1 scores to be independently associated with matriculation into general surgery (P = .007).

Conclusions

Superior clinical aptitude is independently associated with general surgical matriculation. This is in contrast to the negative correlation United States Medical Licensing Examination Step 1 scores have on general surgery matriculation. Recognizing this, surgical clerkship directors can offer opportunities for continued surgical education to students showing high clinical aptitude, increasing their likelihood of surgical matriculation.     

Performance of technology-driven simulators for medical students—a systematic review

Background

Simulation-based education has evolved as a key training tool in high-risk industries such as aviation and the military. In parallel with these industries, the benefits of incorporating specialty-oriented simulation training within medical schools are vast. Adoption of simulators into medical school education programs has shown great promise and has the potential to revolutionize modern undergraduate education.

Materials and methods

An English literature search was carried out using MEDLINE, EMBASE, and psychINFO databases to identify all randomized controlled studies pertaining to “technology-driven” simulators used in undergraduate medical education. A validity framework incorporating the “framework for technology enhanced learning” report by the Department of Health, United Kingdom, was used to evaluate the capabilities of each technology-driven simulator. Information was collected regarding the simulator type, characteristics, and brand name. Where possible, we extracted information from the studies on the simulators' performance with respect to validity status, reliability, feasibility, education impact, acceptability, and cost effectiveness.

Results

We identified 19 studies, analyzing simulators for medical students across a variety of procedure-based specialities including; cardiovascular (n = 2), endoscopy (n = 3), laparoscopic surgery (n = 8), vascular access (n = 2), ophthalmology (n = 1), obstetrics and gynecology (n = 1), anesthesia (n = 1), and pediatrics (n = 1). Incorporation of simulators has so far been on an institutional level; no national or international trends have yet emerged.

Conclusions

Simulators are capable of providing a highly educational and realistic experience for the medical students within a variety of speciality-oriented teaching sessions. Further research is needed to establish how best to incorporate simulators into a more primary stage of medical education; preclinical and clinical undergraduate medicine.