Development of a rating system for surgeons' non-technical skills

BACKGROUND: Analyses of adverse events in surgery reveal that many underlying causes are behavioural, such as communication failure, rather than technical. Non-technical (i.e. cognitive and interpersonal) skills are not addressed explicitly in surgical training. However, surgeons need to demonstrate these skills, which underpin their technical excellence, to maximise patient safety in the operating theatre. This paper describes the method used to identify surgeons' non-technical skills, and the development of a skills taxonomy and behavioural rating system to structure observation and feedback in surgical training. METHODS: Cognitive task analyses (critical incident interviews) were conducted with 27 consultant surgeons in general, cardiac and orthopaedic surgery. The interviews were coded and a multidisciplinary group of surgeons and psychologists used an iterative process to develop a skills taxonomy. This was supported by data gathered from an attitude survey, literature review, analysis of surgical mortality reports and observations in theatre. RESULTS: Five categories of non-technical skills were identified, including situation awareness, decision making, task management, leadership and communication and teamwork. This provided a structure for a prototype skill taxonomy (v1.1), which comprised 14 non-technical skill elements. Observable behaviours (markers) indicative of good and poor performance were developed for each element by 16 consultant surgeons to form a prototype behaviour rating system. CONCLUSIONS: The prototype skills taxonomy and behaviour rating system are grounded empirically in surgery. The reliability of the system is currently being tested using standardised scenarios. If this evaluation proves successful, the system could be used to structure feedback and guide non-technical skills training.     

How can we prepare medical students for theatre‐based learning?

CONTEXT: The quality of medical undergraduate operating theatre-based teaching is variable. Preparation prior to attending theatre may support student learning. Identifying and agreeing key skills, competences and objectives for theatre-based teaching may contribute to this process of preparation. METHODS: We carried out a cross-sectional survey of consultant surgeons and students using a forced choice questionnaire containing 16 skills and competences classified as 'essential', 'desirable' or 'not appropriate', and a choice of 6 different teaching methods, scored for perceived effectiveness on a 5-point Likert scale. Questionnaire content was based on the findings from an earlier qualitative study. RESULTS: Comparative data analyses (Mann- Whitney and Kruskal-Wallis tests) were carried out using SPSS Version 14. A total of 42 consultant surgeons and 46 students completed the questionnaire (46% and 100% response rates, respectively). Knowledge of standard theatre etiquette and protocols, ability to scrub up adequately, ability to adhere to sterile procedures, awareness of risks to self, staff and patients, and appreciation of the need for careful peri-operative monitoring were considered 'essential' by the majority. Student and consultant responses differed significantly on 5 items, with students generally considering more practical skills and competences to be essential. Differences between students on medical and surgical attachments were also identified. CONCLUSIONS: Consultant surgeons and medical students agree on many aspects of the important learning points for theatre-based teaching. Compared with their teachers, students, particularly those on attachment to surgical specialties, are more ambitious - perhaps overly so - in the level of practical skills and risk awareness they expect to gain in theatre.     

The relationship between medical students' attitudes towards communication skills learning and their demographic and education-related characteristics

INTRODUCTION: The General Medical Council (GMC) has stressed the importance of medical students' attitudes towards learning. However, few studies have explored medical students' attitudes towards communication skills learning. This study explores the relationship between the attitudes of medical students at two different schools and their demographic and education-related characteristics. METHODS: A total of 490 medical students from the Universities of Nottingham (Years 1 and 2) and Leicester (Year 1) completed the 26-item Communication Skills Attitude Scale (CSAS) and a personal details questionnaire satisfactorily. The relationships between students' attitudes and their demographic and education-related characteristics were analysed separately for Nottingham and Leicester students using both univariate and multivariate statistics. RESULTS: The attitudes of Nottingham and Leicester medical students towards communication skills learning were significantly associated with a number of demographic and education-related characteristics. Both Nottingham and Leicester students with more positive attitudes towards communication skills learning tended to be female, tended to think their communication skills needed improving and tended not to have parents who were doctors. Both Nottingham and Leicester students with more negative attitudes towards communication skills learning tended to think their communication skills did not need improving. DISCUSSION: The results indicate that medical students' attitudes towards communication skills learning are associated with their demographic and education-related characteristics. These findings have a number of implications for educational practice and further research and these are discussed in this paper.     

Reliability of peer and self-assessment scores compared with trainers' scores following third molar surgery

Context It is sometimes claimed that self‐assessment is inaccurate and that clinicians over‐rate their performance. There is a need to find out why this should be. Is poor self‐assessment caused by some clinicians' inability to accurately judge performance? Or does over‐scoring result from a desire to convey a more favourable impression? Peer assessment is widely advocated and is said to be of benefit to both assessor and assessee. Methods In this study, we wanted to see if postgraduates were able to peer‐assess and if this form of assessment was more reliable than self‐assessment when compared with assessment by a trainer. We used checklist and global rating scales to evaluate surgical skills in removing a mandibular third molar tooth. Results There was no statistically significant difference between peer‐assessed and trainer‐assessed scores. We found that, on average, peer assessment (especially global rating scales) reflected trainer scores more accurately than self‐assessment of surgical skills. Self‐assessment scores were significantly higher on average than those given in peer assessment. Discussion Although peers and trainee surgeons came from the same group, the surgeons were more likely to over‐score when measuring their own performances. The greatest variability (and over‐scoring) between assessor and trainee surgeon appeared to occur in those with lower mean scores. Formative peer assessment may be a useful and less stressful mechanism for encouraging reflection.     

Teaching motivational interviewing: using role play is as effective as using simulated patients

Context  Although several studies of motivational interviewing (MI) as an intervention have been conducted, there has been little research into how best to teach MI. Practice and rehearsal is often beneficial in helping practitioners to acquire communication skills, but there have been few studies into what types of practice and rehearsal are most effective.
Methods  Health care professionals (who attended a 2-day workshop in MI) were randomly assigned to conduct skills practice sessions with either a simulated patient (SP) or a fellow trainee. Their competence was assessed before and after training using the Behaviour Change Counselling Index, a validated rating scale. Participants also scored each practice session in terms of their affect and its perceived applicability to everyday clinical work.
Results  There was no significant difference in skill levels between groups following training and no significant difference between groups in their scoring of the affect and applicability of each practice session. There was little indication of an association between how participants felt about their practice sessions and their skill levels.
Conclusions  Trainees reached the same level of competence in MI following a 2-day workshop, regardless of whether they practised with an SP or a fellow trainee during training.     

Meeting the challenge of prescribing and administering medicines safely: structured teaching and assessment for final year medical students

OBJECTIVES: To promote safe prescribing and administration of medicines in the pre-registration house officer (PRHO) year through a programme of structured teaching and assessment for final year medical students. DESIGN: Forty final year medical students from two medical schools were randomly allocated either to participate in a pharmacist facilitated teaching session or to receive no additional teaching. Teaching comprised five practical exercises covering seven skills through which students rotated in small groups. One month later, a random sample of 16 taught and 16 non-taught students participated in a nine-station objective structured clinical examination (OSCE) to assess the impact of the teaching. SETTING: Manchester School of Medicine (MSM), and Kings College School of Medicine and Dentistry (KCSMD). PARTICIPANTS: Final year medical student volunteers. MAIN OUTCOME MEASURES: The need for teaching as indicated by student prior experience; questionnaire rating of student acceptability of teaching and assessment; self-rating of student confidence post-assessment, and student performance assessed by OSCE. RESULTS: The study demonstrated that the taught group achieved higher scores in eight OSCE stations. Four of these were statistically significant (P < or= 0.005). Taught students felt more confident performing the skills on five stations. From 0 to 47.5% students had prior experience of the skills taught. The post-teaching questionnaire evaluated exercises positively on several criteria, including provision of new information and relevance to future work. CONCLUSIONS: Structured teaching provided an effective and acceptable method of teaching the medicines management skills needed in the PRHO year. The structured approach complemented variable precourse clinical experience.     

Evaluation of self-directed clinical education: validation of an instrument

AIM: To explore the evaluation of self-directed, integrated clinical education. METHODS: We delivered a quantitative and qualitative, self-report questionnaire to students through their web-based learning management system. The questionnaire was distributed 4 times over 1 year, each time in 2 parts. A generic part evaluated boundary conditions for learning, teaching activities and "real patient learning". Factor analysis with varimax rotation was used to validate the constructs that made up the scale and to stimulate hypotheses about how they interrelated. A module-specific part evaluated real patient learning of the subject matter in the curriculum. RESULTS: A total of 101 students gave 380 of a possible 404 responses (94%). The generic data loaded onto 4 factors, corresponding to: firm quality; hospital-based teaching and learning; community and out-patient learning, and problem-based learning (PBL). A 5-item quality index had content, construct and criterion validity. Quality differed greatly between firms. Self-evaluation of module-specific, real patient learning was also valid. It was strongly influenced by the specialty interests of hospital firms. CONCLUSIONS: Quality is a multidimensional construct. Self-report evaluation of real patient learning is feasible, and could be capitalised on to promote reflective self-direction. The social and material context of learning is an important dimension of educational quality.     

Can undergraduate education have an effect on the ways in which pre‐registration house officers conceptualise communication?

AIMS: In 1994 Manchester Medical School introduced a learner-centred course using problem-based learning (PBL), which emphasises effective communication skills. This study explored how 2 cohorts of graduates conceptualised communication within their role as pre-registration house officer (PRHOs). METHODS: Graduates from the last year of the traditional and first year of the new course were interviewed 3 months after starting work. They were asked how well their courses had prepared them for working as PRHOs and were given a specific question about communication skills. Interviews were tape-recorded, transcribed, coded and analysed. RESULTS: In all, 24 traditional course graduates (TCGs) and 23 new course graduates (NCGs) were interviewed. Traditional course graduates tended to conceptualise communication as informing people rather than as involving negotiation or having therapeutic effects. Most TCGs considered good communicators 'were born that way' and did not think communication skills could be learned. Many NCGs had a richer concept of communication that recognised communication has therapeutic benefits for patients and involved negotiation. They understood it was possible to teach communication skills. However, a minority of NCGs conceptualised communication in a similar way to TCGs. CONCLUSION: Our study suggests that the new course has helped NCGs acquire a more complex concept of communication in their role as PRHOs.     

General competencies of problem-based learning (PBL) and non-PBL graduates

INTRODUCTION: Junior doctors have reported shortcomings in their general competencies, such as organisational skills and teamwork. We explored graduates' perceptions of how well their training had prepared them for medical practice and in general competencies in particular. We compared the opinions of graduates from problem-based learning (PBL) and non-PBL schools, because PBL is supposed to enhance general competencies. METHOD: We analysed the responses of 1159 graduates from 1 PBL and 4 non-PBL schools to a questionnaire survey administered 18 months after graduation. RESULTS: Compared with their non-PBL colleagues, the PBL graduates gave higher ratings for the connection between school and work, their medical training and preparation for practice. According to the graduates, the most frequently used competencies with sufficient coverage during medical training were expert knowledge, profession-specific skills and communication skills. The majority of the PBL graduates, but less than half of the non-PBL graduates, indicated that communication skills had been covered sufficiently. All the graduates called for more curriculum attention on working with computers, planning and organisation, and leadership skills. More PBL graduates than non-PBL graduates indicated that they had learned profession-specific methods, communication skills and teamwork in medical school. DISCUSSION: Overall, the graduates appeared to be satisfied with their knowledge and skills. The results suggest that the PBL school provided better preparation with respect to several of the competencies. However, both PBL and non-PBL graduates identified deficits in their general competencies, such as working with computers and planning and organising work. These competencies should feature more prominently in undergraduate medical education.     

A web-based presentation of an undergraduate clinical skills curriculum

AIMS: The aim of this study was to use information and communications technology to present a curriculum of clinical skills in a user-friendly format. SETTING: A UK undergraduate medical school with a problem-based curriculum and a strong emphasis on proficiency in clinical skills. STUDY DESIGN: Case study describing the qualitative analysis of users' requirements and development of a web-based learning portfolio. EVALUATION: The study involved direct observation of users during a 'think-aloud' protocol, a validated software users' measurement inventory and a 17-item questionnaire designed to test whether 'SkillsBase' met its users' requirements. RESULTS: Students wanted a clear and flexible presentation of their skills curriculum that was easy to navigate, offered instructional material and standards for self- and peer assessment, offered useful Internet links, allowed them to compare their progress with school standards and peer norms, and could be used as a learning portfolio. During the think-aloud protocol, students made very few errors in data interpretation or navigation, and found SkillsBase easy to learn and aesthetically pleasing to use. They rated it higher on all measures of usability than standard commercial software. The questionnaire showed that it met most aspects of its design specification, although many students were doubtful that they would use its reflective function. It is available for inspection at http://www.skillsbase.man.ac.uk/. CONCLUSIONS: SkillsBase meets the design specification for a training and reflective aid to learning clinical skills and is very usable.     

The effectiveness of a computer-assisted instruction programme on communication skills of medical specialists in oncology

BACKGROUND: Although doctor--patient communication is important in health care, medical specialists are generally not well trained in communication skills. Conventional training programmes are generally time consuming and hard to fit into busy working schedules of medical specialists. A computer-assisted instruction (CAI) programme was developed -- 'Interact-Cancer' -- which is a time-efficient learning method and easily accessible at the workplace. OBJECTIVE: To investigate the effect of the CAI training, 'Interact-Cancer', on the communication behaviour of medical specialists, and on satisfaction of patients about their physician interaction. DESIGN: Consultations of medical specialists with cancer outpatients were videotaped at 4 specific stages, 2 before and 2 after Interact-Cancer, with intervals of 4 weeks. PATIENTS/PARTICIPANTS: Participants were 21 medical specialists, mainly internists, working in 7 hospitals, and 385 cancer outpatients. METHODS: Communication behaviour was assessed on 23 observation categories derived from the course content. Frequencies were rated as well as judgements about the quality of the performance of each target skill. Satisfaction was measured by the Medical Interview Satisfaction Scale. Data were analyzed by means of multilevel statistical methods. RESULTS: The behavioural assessment showed course effects on ratings of the physicians' quality of performance. No course effects were found on the frequencies of physicians' behaviours and on the patient satisfaction ratings. CONCLUSIONS: CAI is a promising method to supply medical specialists with postgraduate training of communication skills. The application of judgement ratings of communication behaviour proved to be valuable to evaluate course effects in real-life patient encounters.     

Further challenges in measuring communication skills: accounting for actor effects in standardised patient assessments

Context  Subjective rating scales for communication skills may yield more personally meaningful responses than more standardised rating schemes. It is unclear, however, whether such evaluations may be overly biased by respondents' rating styles, which may lead to unreliable measurement of examinees' communication skills.
Methods  Our study involved 212 students from the classes of 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All students were rated by actors depicting standardised patients (SPs) on the same seven cases using the 19-item Rochester Communication Rating Scale (RCRS). Different students were assigned to different actors playing the same SP. We assessed the extent to which actors' personal rating styles influenced the scores they assigned to students. Main outcome measures were: between-actor variability in responses; the degree to which actors' response styles contribute to overall scores, and improvements in reliability achieved by standardising actors' ratings.
Results  There were statistically significant differences between actors in their mean assigned scores. Scores aggregated over 18 separate SP cases have an expected generalisability coefficient of 0.79. If raw RCRS scores are used, a total of 27 replications of the RCRS are required to achieve a Cronbach's alpha of 0.8; standardisation reduces this number to 18.
Conclusions  Although actors are variable in their use of a standardised subjective scale of communication, such differences contribute to an acceptably small proportion of the total variance if scores are combined across a large number of cases. Reliability can be markedly improved by standardising scores across raters.     

Impact of a new curriculum on medical students' interpersonal and interviewing skills

OBJECTIVES: In 1998 we reported on the rise and fall of medical student communication skills during the 4 years of medical school. Since then, the University of Connecticut School of Medicine has completed a major curriculum renewal project with an emphasis on early clinical work, lifelong learning and more ambulatory training. The goals of this study were to compare students' interviewing and interpersonal skills in standardised patient (SP) assessments in the old and new curricula and to assess the success of the new curriculum in preventing a decline in student skills in this domain. METHODS: The clinical skills of 202 students were measured longitudinally during encounters with SPs in each of their 4 years of medical school. Students in this study and the earlier study were evaluated using the Arizona Clinical Interviewing Rating (ACIR) Scale. RESULTS: Compared with students from the previous curriculum, students on the new curriculum in this study showed an improvement in ACIR scores. Year 1 mean ACIR scores (1 = poor to 5 = excellent) were, respectively, 3.6 for the old curriculum cohort and 4.0 for the new curriculum group. In Year 4 the mean score for the old curriculum cohort was 3.7 and that for the new curriculum group was 3.8. Students on the new curriculum still showed a decline in ACIR scores from Years 1 to 4, but it was not as severe a decline as it had been previously. CONCLUSIONS: Pre-clinical medical students perform better on measures of interpersonal communication than their clinical counterparts. The students who participated in the new curriculum demonstrated an earlier acquisition of and a less steep decline in interviewing and interpersonal skills during the course of medical school.     

Factor analysis can be a useful standard setting tool in a high stakes OSCE assessment

BACKGROUND: No method of standard setting for objective structured clinical examinations (OSCEs) is perfect. Using scores aggregated across stations risks allowing students who are incompetent in some core skills to pass an examination, which may not be acceptable for high stakes assessments. AIM: To assess the feasibility of using a factor analysis of station scores in a high stakes OSCE to derive measures of underlying competencies. METHODS: A 12-station OSCE was administered to all 192 students in the penultimate undergraduate year at the University of Aberdeen Medical School. Analysis of the correlation table of station scores was used to exclude stations performing unreliably. Factor analysis of the remaining station scores was carried out to characterise the underlying competencies being assessed. Factor scores were used to derive pass/fail cut-off scores for the examination. RESULTS: Four stations were identified as having unpredicted variations in station scores. Analysis of the content of these stations allowed the underlying problems with the station designs to be isolated. Factor analysis of the remaining 8 stations revealed 3 main underlying factors, accounting for 53% of the total variance in scores. These were labelled "examination skills", "communication skills" and "history taking skills". CONCLUSION: Factor analysis is a useful tool for characterising and quantifying the skills that are assessed in an OSCE. Standard setting procedures can be used to calculate cut-off scores for each underlying factor.     

The effect of PACS on the practice of orthopaedic surgeons

When new information and communication technologies are implemented there are organizational effects. Picture Archiving and Communication Systems (PACS) have been implemented not only within radiology but also within the orthopaedic context. We studied the impact of PACS on the work practice and professional role of orthopaedic surgeons in relation to radiologists. Qualitative interviews were performed with 15 surgeons at two hospitals. The interviews were transcribed, analysed and then coded using grounded theory. Three main categories were defined: professional role, diagnostic practice and technology used. There was a shift in orthopaedics from a single to a mixed profession, and from a relatively static practice in interpreting images to a more flexible practice, where every orthopaedic surgeon could view and access images from anywhere at any time, including three-dimensional imaging. It was easier for surgeons to see and interpret the images, and their diagnostic skills became accessible to everyone. The use of PACS also improved the dialogue with patients, according to the respondents. This was necessary in order to explain the treatment options for the patient or the details of the disease. PACS therefore acts as an information and communications technology and users required new knowledge and skills.     

Factors influencing prescribing: an intern's perspective

OBJECTIVES: To examine the self-reported influences on intern prescribing practice. DESIGN: Qualitative interviews with a cross-sectional cohort. PARTICIPANTS AND SETTING: Ten interns practising in two urban teaching hospitals in New South Wales, Australia. RESULTS: The interns identified a number of factors that improve their confidence and perceived competence and allow them to extend their existing skills. These were approachable, available and up-to-date teachers (most often registrars and subspecialty nurses and pharmacists); timely, relevant and practical teaching (such as interactive bedside teaching); concise and widely accepted resources (such as prescribing pocket guides); and a constructive manner on the part of senior staff for dealing with prescribing errors. Interns also identified influences that are detrimental to confidence, conflict with their perceptions of appropriate prescribing and inhibit learning and skills acquisition. These were unapproachable, physically and mentally remote teachers (most often consultants); theoretical, inconsistent and irrelevant teaching (such as grand rounds or didactic education sessions); inconsistent and inaccessible resources; and a confrontational and accusatory way of dealing with prescribing errors. The added pressures of time, hospital hierarchies and the indirect influence of drug company promotion also impeded acquisition of good prescribing habits. CONCLUSIONS: At a critical time in skills development, interns encounter many forces that can potentially impact on prescribing practices in both positive and negative ways. Our data contribute to the understanding of the multifaceted learning environment of interns and may be useful in providing a foundation for prescriber education programmes tailored to the specific needs of junior doctors.     

Clinical skills in junior medical officers: a comparison of self-reported confidence and observed competence

BACKGROUND: The intern year is a key time for the acquisition of clinical skills, both procedural and cognitive. We have previously described self-reported confidence and experience for a number of clinical skills, finding high levels of confidence among Australian junior doctors. This has never been correlated with an objective measure of competence. AIMS AND HYPOTHESIS: We aimed to determine the relationship between self-reported confidence and observed competence for a number of routine, procedural clinical skills. METHODS: A group of 30 junior medical officers in their first postgraduate year (PGY1) was studied. All subjects completed a questionnaire concerning their confidence and experience in the performance of clinical skills. A competency-based assessment instrument concerning 7 common, practical, clinical skills was developed, piloted and refined. All 30 PGY1s then completed an assessment using this instrument. Comparisons were then made between the PGY1s' self-reported levels of confidence and tutors' assessments of their competence. RESULTS: A broad range of competence levels was revealed by the clinical skills assessments. There was no correlation between the PGY1s' self-ratings of confidence and their measured competencies. CONCLUSIONS: Junior medical officers in PGY1 demonstrate a broad range of competence levels for several common, practical, clinical skills, with some performing at an inadequate level. There is no relationship between their self-reported level of confidence and their formally assessed performance. This observation raises important caveats about the use of self-assessment in this group.