An Objective Evaluation of a Multi-Component, Competitive, Selection Process for Admitting Surgeons into Higher Surgical Training in a National Setting

Background

Changing work practices make it imperative that surgery selects candidates for training who demonstrate the spectrum of abilities that best facilitate learning and development of attributes that, by the end of their training, approximate the characteristics of a consultant surgeon.

Aims

The aim of our study was to determine the relative merits of components of a program used for competitive selection of trainees into higher surgical training (HST) in general surgery.

Methods

Applicants (N = 98, males 69, mean age 31 years [range 29–40]) to the Royal College of Surgeons in Ireland program for HST in general surgery between 2006 and 2008 were assessed. Clinical, basic surgical training, logbook, research performance, and reference scores were evaluated. A total of 51 candidates were shortlisted and completed a further objective assessment of their technical skills and interview performances.

Results

Shortlisted candidates performed better (p < 0.003) on all assessed parameters. Compared with candidates who were not selected for HST, those who were selected (N = 31) significantly outperformed on individual assessments and overall (p < 0.0001). Logistic regression analysis showed that clinical, technical skills, and research assessments, but not interview, predicted (92.2 %) HST selection outcomes.

Conclusions

Candidates selected for the national HST program in Ireland consistently outperformed those who were not. The assessments reliably and consistently distinguished between candidates, and all of the assessed parameters (except interview) contributed to a highly predictive selection model. This is the largest reported dataset from an objective, transparent, and fair assessment program for selection of the next generation of surgeons.     

Visuospatial and technical ability in the selection and assessment of higher surgical trainees in the London deanery

INTRODUCTION

Despite awareness of the limitations of current selection and competency assessments, there is little consensus and alternatives have not been readily accepted. Essential surgical skills include visuospatial and technical ability. The aim of this study was to survey current methods of higher surgical trainee selection and assessment. We suggest ways to improve the process.

MATERIALS AND METHODS

Nine surgical training programmes in the London deanery were surveyed through questionnaires to programme directors, existing trainees and examination of deanery publications.

RESULTS

Testing of visuospatial and technical ability was piloted at selection only in a single general surgical department. Practical skills were assessed in 3/9 (33%) specialties (ENT, plastic and general surgery). Once selected, no specialty tested visuospatial and technical ability. Practical skills were tested in only 1/9 (11%) specialties (plastic surgery). The remaining 8/9 (89%) were ‘assessed’ by interview.

CONCLUSIONS

Lack of visuospatial and technical ability assessment was identified at selection and during higher surgical training. Airlines have long recognised early identification of these qualities as critical for efficient training. There is a need for more objective methods in this area prior to selection as time to assess surgical trainees during long apprenticeships is no longer available. We advocate a suitably validated competency-based model during and at completion of training.     

The Canadian Urology Fair: a model for minimizing the financial and academic costs of the residency selection process.

INTRODUCTION: In 1994, the Canadian urology residency training programs designed the "Canadian Urology Fair"--a single-site (Toronto, Ont.), 1-day fair to conduct the personal interview portion of the residency selection process. The objective of the current study was to evaluate the success of the Urology Fair in achieving its original goals of decreasing the financial burden and minimizing time away from medical training for applicants and faculty. METHODS: Both candidates and Canadian urology training programs were surveyed regarding the financial and academic costs (days absent) of attending the 2001 Urology Fair. Data from the 2001 Canadian Resident Matching Service (CaRMS) was used to compare the financial and academic costs of attending personal interviews incurred by candidates declaring urology as their first-choice discipline to candidates interviewing with other surgical specialties throughout Canada. RESULTS: Financial costs incurred by candidates to attend the Urology Fair (mean Can dollar 367) were significantly lower than candidates' estimated costs of attending on-site interviews at the individual programs (mean Can dollar 2065). The financial costs of attending personal interviews by CaRMS applicants declaring urology as their first-choice discipline (mean Can dollar 2002) were significantly lower than the costs incurred by applicants interviewing with other surgical disciplines (mean Can dollar 2744). Financial costs to urology programs attending the fair (mean Can dollar 1931) were not significantly greater than the programs' estimated costs of conducting on-site interviews at their respective program locations (mean Can dollar 1825). Days absent from medical school to attend interviews were significantly lower among CaRMS applicants declaring urology as their first-choice discipline (3 d) compared with applicants who interviewed with other surgical specialties (9.1 d). CONCLUSION: The Canadian Urology Fair represents an innovative and efficient method for residency programs to conduct the personal interview portion of the residency selection process and should serve as a model for making the interview process less expensive and time-consuming for both candidates and faculty.     

Technical skills assessment as part of the selection process for a fellowship in minimally invasive surgery

Introduction

Selection of candidates for surgical fellowships has traditionally been based on subjective evaluations by the program directors and references from previous positions. The introduction of well-validated objective methods of assessment has allowed us to evaluate candidates’ technical skills and base the selection process on objective, reliable, and transparent criteria. The aim of the study was to assess the applicability of such methods in current practice.

Materials and methods

Prospective study. Eight surgeons, applying for a fellowship position in minimally invasive surgery (MIS), performed a previously validated assessment curriculum using a Virtual-Reality Laparoscopic Trainer (LapSim^® 3.0, Surgical Science, Gothenburgh, Sweden). Technical performance was evaluated using criteria registered by the simulator, i.e., time, error score, and efficiency of movements score. Candidates performed all the tasks in easy end medium level until reaching predefined criteria. If proficiency criteria were not achieved on easy or medium level after nine repetitions the test was considered as failed. Additionally, all applicants underwent an interview by two independent attending surgeons. Each applicant received a grade on a ten-point scale.

Results

Five out of the eight candidates failed the technical skills assessment test. One candidate failed to achieve proficiency criteria on easy level, one on medium, and three on difficult level. Evaluation scores, based on the interview of the candidates showed a good interrater reliability (Cronbach’s α = 0.8). There was no significant correlation between the interviewers rating, and the applicants technical skills demonstrated during the test on the VR trainer (Spearman’s ρ = 0.182, p = 0.696).

Conclusions

Evaluations by senior surgeons are reproducible and reliable. The introduction of technical skills assessment has the potential to improve the current method of candidate selection, making it more valid, objective, and transparent.     

ASSESSMENT OF LAPAROSCOPIC PSYCHOMOTOR SKILLS IN INTERNS USING THE MIST VIRTUAL REALITY SIMULATOR: A PREREQUISITE FOR THOSE CONSIDERING SURGICAL TRAINING?

Selection for surgical training in Australia is currently based on assessment of a structured curriculum vitae, referral reports from selected clinicians and an interview. The formal assessment of laparoscopic psychomotor skill and ability to attain skills is not currently a prerequisite for selection. The aim of this study was to assess the innate psychomotor skills of interns and also to compare interns with an interest in pursuing a surgical career to interns with those with no interest in pursuing a surgical career. Twenty-two interns were given the opportunity to carry out tasks on the Minimal Invasive Surgical Trainer, Virtual Reality (Mentice, Gothenburg, Sweden) Simulator. The candidates were required to complete six tasks, repeated six times each. Scores for each task were calculated objectively by the simulator software. Demographic data were similar between the two groups. Although some candidates who were interested in pursuing a surgical career performed poorly on the simulator, there was no significant difference when comparing the two groups. The Minimal Invasive Surgical Trainer, Virtual Reality (Mentice) Simulator provides an objective and comparable assessment of laparoscopic psychomotor skills. We can conclude that interns have varying inherent ability as judged by the simulator and this does not seem to have an influence on their career selection. There was no significant difference in the scores between the two groups. Interns with and without inherent abilities have aspirations to pursue surgical careers and their aptitude does not seem to influence this decision. Surgical colleges could use psychomotor ability assessments to recruit candidates to pursue a career in surgery. Trainees needing closer monitoring and additional training could be identified early and guided to achieve competency.     

Surgeons' non-technical skills

The importance of non-technical skills to surgical performance is gaining wide acceptance. This article discusses the core cognitive and social skills categories thought to underpin medical knowledge and surgical expertise, and describes the rise of non-technical skill models of assessment in surgery. Behavior rating systems such as NOTSS (Non-Technical Skills for Surgeons) have been developed to support education and assessment in this regard. We now understand more about these critical skills and how they impact surgery. The challenge in the future is to incorporate them into undergraduate teaching, postgraduate training, workplace assessment, and perhaps even selection.     

Assessment of microsurgery competency‐where are we now?

In the last decade surgical training is being revolutionized by two novel concepts that have been introduced to almost all branches of surgery including and most recently to microsurgery. These two concepts are: objective assessments of surgical skills and the nurturing of surgical skills in a simulation laboratory setting. Acquiring surgical skills in the laboratory setting can help move the microsurgical learning curve from the patient to the laboratory and this will in turn improve patient safety. In order to optimize microsurgical training through a competency based training programme, it is imperative for microsurgical educators to understand microsurgical skill acquisition. This requires accurate objective assessment tools that can define and quantify microsurgical competency. This article aims to review the current literature on the various objective assessment tools adapted for microsurgery and attempt to identify the gaps that need to be addressed by research in microsurgical education to establish the ideal objective assessment tool. © 2013 Wiley Periodicals, Inc. Microsurgery 33:406–415, 2013.     

Laparoscopic skills training and assessment

BACKGROUND: The introduction of laparoscopic techniques to general surgery was associated with many unnecessary complications, which led to the development of skills laboratories to train novice laparoscopic surgeons. This article reviews the tools currently available for training and assessment in laparoscopic surgery. METHODS: Medline searches were performed to identify articles with combinations of the following key words: laparoscopy, training, curriculum, virtual reality and assessment. Further articles were obtained by manually searching the reference lists of identified papers. RESULTS: Current training involves the use of box trainers with either innate models or animal tissues; it lacks objective assessment of skill acquisition. Virtual reality simulators have the ability to teach laparoscopic psychomotor skills, and objective assessment is now possible using dexterity-based and video analysis systems. CONCLUSION: The tools are now available for the development of a structured, competency-based, laparoscopic surgical training programme.     

Assessment of technical surgical skills.

OBJECTIVE: To review methods of assessment of technical surgical competence, and to point out the potential benefit of such assessment to training and safe practice. DESIGN: Review of publications about methods of evaluation of technical surgical competence. Studies were identified through MEDLINE using the keywords surgery, assessment, operative skills, training, and competence. The references cited in these studies were reviewed to find out whether any other trials fitted the selection criteria. SETTING: Teaching hospital, Denmark. RESULTS: There are several validated methods that allow objective assessment of operative skills for both open and laparoscopic surgery. CONCLUSIONS: Objective assessment of technical surgical skills is possible and reliable and valid methods are available. There are good reasons to implement evaluation programmes as a part of surgical education.     

Role and feasibility of psychomotor and dexterity testing in selection for surgical training

The practice of Surgery has undergone major changes in the past 20 years and this is likely to continue. Knowledge, judgement and good technical skills will no longer be enough to safely practice surgery and interventional procedures. Fundamental abilities (e.g. psychomotor skills, visuospatial ability and depth perception) are critically important for catheter‐based interventions, NOTES, robotic surgery and other procedural interventions of the future. Not all individuals possess the same amount of these innate fundamental abilities and those less endowed are likely to struggle during surgical training and thereafter in surgical practice. In contrast to other high‐skill professions/industries (e.g. aviation) we do not have a tradition of testing prospective surgical trainees for abilities/attributes that we now recognize as being important for surgical practice. Instead, we continue to rely on surrogate markers of future potential (e.g. academic record). However, many studies have shown that psychomotor ability is an important predictor of both learning rate and performance for complex laparoscopic tasks. Psychomotor skills, visuospatial ability and depth perception can all be tested objectively by validated tests. At the Royal College of Surgeons in Ireland, all short‐listed candidates for Higher Surgical Training now undergo formal testing of both technical skills and fundamental abilities (psychomotor skills, visuospatial ability and depth perception). Reports on each candidate’s performance are supplied to the interview committee. Furthermore, a prospective database is being kept for correlation with future surgical performance. We believe that selection into surgical training should take account of attributes that we know are important for safe and efficient surgical practice.     

Training for advanced laparoscopic surgery.

OBJECTIVE: To evaluate the efficiency of training for advanced minimally-invasive surgery in rats. SETTING: Teaching hospital, Germany. SUBJECTS: 10 surgical residents with few laparoscopic training skills had two days laparoscopic microsurgical training using rats. INTERVENTIONS: Increasingly difficult procedures started using a rubber model, followed by 'open' and 'closed' rat models simulating laparoscopic conditions. MAIN OUTCOME MEASURES: The surgical skills of all participants were assessed before and after the laparoscopic training course by applying 5 defined tasks in a simulation trainer. As a control group, 8 surgical residents with the same amount of minimal-invasive training were tested after a 2-day interval. RESULTS: All participants of the laparoscopic training programme significantly improved their surgical skills for all 5 tasks (p < 0.05). In the control group only one of the 5 tasks was significantly improved after repetition. Participants of the complete training programme consistently improved their surgical skills significantly compared with the controls before and after repetition (p < 0.05). CONCLUSION: Surgical training in minimal-invasive techniques in the rat model is effective and reproducible especially to improve laparoscopic skills such as bimanual tissue handling, knot tying and microsurgical suturing.     

Selecting the best and brightest: A comparison of residency match processes in the United States and Canada

BACKGROUND:

Selecting candidates for plastic surgery residency training remains a challenge. In the United States, academic measures (United States Medical Licensing Exam Step I scores, medical school class rank and publications) are used as primary criteria for candidate selection for residency. In contrast, Canadian medical education de-emphasizes academic measures by using a pass-fail grading system. As a result, choosing residents from many qualified applicants may pose a challenge for Canadian programs without objective measures of academic success.

METHODS:

A 25-question online survey was distributed to program directors of Canadian plastic surgery residency-training programs. Program directors commented on number of yearly residents and applicants; application sections (ranked in importance using a Likert scale); interview invitation and rank-order list determination; and their satisfaction with the selection process.

RESULTS:

Ten Canadian plastic surgery program directors responded (90.9% response rate). The most important application components determining invitation to interview were letters of reference from a plastic surgeon (mean importance of 5.0 on the Likert scale), clinical electives in plastic surgery (mean 4.6) and electives with their program (mean 4.5). Applicants invited for interview were assessed on the quality of their responses to questions, maturity and personality. The majority of program directors agreed that a clinical elective with their program was important for consideration on their rank-order list. Program directors were neutral on their satisfaction with the selection process.

CONCLUSION:

Canadian plastic surgery residency programs emphasize clinical electives with their program and letters of reference from colleagues when selecting applicants for interviews. In contrast to their American counterparts, Canadian program directors rely on clinical interactions with prospective residents in the absence of objective academic measures.     

Is the quality of surgical residency applicants deteriorating?

BACKGROUND: Among directors of general surgery residencies, there is a concern that the quality of medical students applying to surgical residencies is declining. METHODS: Quality of surgical applicants was assessed by several methods including subjective opinions determined by survey and by objective data including student United States Medical Licensing Examination (USMLE) scores of matched candidates. The number of applicants interviewed, total interviews granted, proportion of Alpha Omega Alpha (AOA) students, and the rank order of the candidates matched was obtained by survey. The survey included data on postgraduate year 1 (PGY-1) residents from July 1996 to July 1999. Three mailings were made to 226 US surgical residency programs. RESULTS: Data were obtained from 90 programs. Surgery program directors disagreed with a survey statement that overall quality of applicants had declined (P <0.01), but agreed with a statement that activities of medical schools to enroll graduating students into primary care had hurt recruitment (P <0.001). Objective data revealed no change in mean USMLE part I scores of PGY-1 residents over the 4 years (P = 0.265, power = 0.81). There was no change in proportion of matched residents who were AOA over time. The mean score of all new PGY-1 residents, the rank of the first matched resident, the rank of the last ranked resident, and proportion of AOA students was higher in programs with five or more categorical spots when compared with programs of at most four (P <0.001). Across all programs, there was a trend to go lower on the rank list to fill categorical positions over time (P <0.001). CONCLUSIONS: There is a perception that medical school policies act to discourage recruitment of quality medical students into general surgery programs, and surgery programs are going deeper into their rank lists to fill categorical positions. However, the average USMLE part I score of applicants to surgical residencies and proportion of AOA applicants has not decreased.     

Attitudes of applicants for surgical residency toward work hour limitations

BACKGROUND: There is an ongoing debate regarding the merits of resident work-hour limitations. We postulated that this issue would be a factor in the decision-making process of applicants to surgical residency. METHODS: Candidates for surgical residency at a university-based program completed an anonymous survey during their visit. Data was analyzed by analysis of variance and the chi-square test. RESULTS: Most candidates viewed work-hour limitations as being favorable to their future training. Nevertheless, work-hour limitations ultimately were not a critical factor in the decision-making process compared with issues such as quality of training and program reputation. Candidates ranked "reading in surgery" the most likely way they would spend the leisure time afforded by work-hour limitations. CONCLUSIONS: Most applicants for surgical residency consider work hour-limitations as being favorable to their training and view the extra free time as an opportunity for furthering their education. However, other issues take precedence when choosing a residency.     

Objective comparison of manual dexterity in physicians and surgeons

Recent interest in the assessment of manual dexterity of surgical residency applicants prompted an investigation of psychomotor skills in surgeons and physicians. The Purdue Pegboard and Minnesota Manual Dexterity tests were given to 57 subjects. Analysis of the data revealed no significant difference in dexterity between medical and surgical residents, suggesting that medical students do not select specialty training programs because of the presence or absence of manual skills. The data also revealed that surgical staffmen performed substantially worse than those in the other groups on some of the tests, possibly because of an older average age. The authors conclude from their data that manual dexterity tests should not be used in assessing candidates for surgical residency training positions.     

Nurturing of surgical careers by the wellington surgical interest club

The Royal Australasian College of Surgeons (RACS) has recently introduced the new Surgical Education and Training programme. The purpose of this was, in part, to help address the anticipated shortage of surgeons in the future, by streamlining the surgical training programme. The formation of the Wellington Surgical Interest Club (WSIC), a student‐led initiative, had several complementary goals. These included the desire to identify potential candidates for a career in surgery, promote a surgical career to students especially women, help students acquire basic surgical skills early, inform students about surgical careers, promote student involvement in surgical research and to create an effective mentorship model during undergraduate and junior surgical training. The strengths of WSIC are its goals, which are similar to those of the RACS with regard to promoting surgery as a career option; its easy reproducibility at other medical institutions; its ability to focus on issues of relevance to both students and junior doctors; and being a bridging solution at a time when early exposure to surgical specialties is both desired and necessary.     

Square pegs in round holes: has psychometric testing a place in choosing a surgical career? A preliminary report of work in progress.

Methods of selection of candidates for training in surgery has long been regarded as lacking explicit criteria and objectivity. Our purpose was to discover the aptitudes and personality types of applicants for surgical posts at the outset, in order to discover which were most likely to result in a satisfactory progression through training and which were associated with career difficulties. This longitudinal predictive validation study has been undertaken in a London Teaching Hospital since 1994. After short-listing, but immediately before interview, all candidates for senior house officer posts in basic surgical training and in geriatric medicine were asked to undertake psychometric tests of numerical (GMA) and spatial (SIT7) reasoning, personality type (MBTI), and self-rating of competency. There were no differences in ability scores between surgeons or geriatricians. Personality differences were revealed between the surgeons and the geriatricians, and between male and female surgeons. This study suggests that while there are no differences in ability between surgeons and geriatricians at the start of training, there are differences in personality. Long-term follow-up of the career development of this cohort of surgical SHOs is required to determine whether the psychometric measures described correlate with achievements of milestones in their surgical careers.     

Effects of a laparoscopic course on student interest in surgical residency.

BACKGROUND: The number of surgical residency applicants has been declining. Early introduction of the discipline of surgery is thought to stimulate early interest in surgical residency. This study investigated the hypothesis that a laparoscopic skills course introduced in preclinical years would stimulate student interest in entering surgical residency. METHODS: Preclinical medical students participated in a laparoscopic skills training course. All students underwent an animate laboratory at the beginning and at the end of the course. Students were divided into 4 separate groups: virtual reality, box trainer, both trainers, and control group. Before and after the course, students were asked their residency interest. First- and second-year medical students participated in the course. RESULTS: Before the course, 56% of the students desired to go into general surgery or a surgical subspecialty. After the course, 49% of the students expressed interest in entering general surgery or a surgical subspecialty. A decrease occurred in students who desired to go into surgical subspecialty residency from 31% to 15% (P = NS), and an increase occurred in students who desired to go into general surgery residency from 25% to 34% (P = NS). No statistically significant difference was seen in the 4 individual training subgroup analyses. CONCLUSIONS: Participation in a laparoscopic skills course does not affect medical student interest in entering surgical residency. A trend was noted in students choosing general surgery over surgical subspecialty training after this course. Surgical educators need to investigate methods to encourage preclinical medical student interest in surgical residencies.     

Training for infrainguinal bypass surgery.

BACKGROUND: Despite satisfactory results for surgery performed by trainees, vascular surgeons need to improve training methods to ensure that aspiring surgeons are adequately trained with less clinical exposure during fewer dedicated years of training. OBJECTIVES: To review the wide range of workshop, laboratory and seminar-room based methods available to train for the diverse range of skills required for distal arterial revascularisation. Training methods include anastomotic suturing skills with bench-top training apparatus, working with realistic plastic models and prosthetic conduits, cadaveric dissections and virtual-reality simulations. Many of these also provide excellent opportunities for objective assessment of technical skills and trainees' progress. DESIGN AND METHODS: A review of the literature on surgical education, surgical skills training and assessment. An evaluation of some of the apparatus, facilities, training curricula and courses, currently available to European trainees, is carried out. CONCLUSIONS: Many methods are now available to allow focused training for particular skills in non-clinical settings. Objective tools are also available that allow assessment of trainees at many levels or practicing surgeons. These technical skills assessment methods are important for trainees and surgeons who, in the future, will increasingly need to demonstrate competence in vascular surgery.     

Credentialing of surgical skills centers

Major imperatives regarding quality of patient care and patient safety are impacting surgical care and surgical education. Also, significant emphasis continues to be placed on education and training to achieve proficiency, expertise, and mastery in surgery. Simulation-based surgical education and training can be of immense help in acquiring and maintaining surgical skills in safe environments without exposing patients to risk. Opportunities for repetition of tasks can be provided to achieve pre-established standards, and knowledge and skills can be verified using valid and reliable assessment methods. Also, expertise and mastery can be attained through repeated practice, specific feedback, and establishment of progressively higher learning goals. Simulation-based education and training can help surgeons maintain their skills in infrequently performed procedures and regain proficiency in procedures they have not performed for a period of time. In addition, warm-ups and surgical rehearsals in simulated environments should enhance performance in real settings.Major efforts are being pursued to advance the field of simulation-based surgical education. New education and training models involving validation of knowledge and skills are being designed for practicing surgeons. A competency-based national surgery resident curriculum was recently launched and is undergoing further enhancements to address evolving education and training needs. Innovative simulation-based surgical education and training should be offered at state-of-the-art simulation centers, and credentialing and accreditation of these centers are key to achieving their full potential.