Examiner fatigue in communication skills objective structured clinical examinations

CONTEXT: The assessment of undergraduates' communication skills by means of objective structured clinical examinations (OSCEs) is a demanding task for examiners. Tiredness over the course of an examining session may introduce systematic error. In addition, unsystematic error may also be present which changes over the duration of the OSCE session. AIM: To determine the strength of some sources of systematic and unsystematic error in the assessment of communication skills over the duration of an examination schedule. METHODS: Undergraduate first-year medical students completing their initial summative assessment of communication skills (a four-station OSCE) comprised the study population. Students from three cohorts were included (1996-98 intake). In all 3 years the OSCE was carried out identically. All stations lasted 5 minutes with a simulated patient. Students were assessed using an examiner (content expert) and a simulated-patient evaluation tool, the Liverpool Communication Skills Assessment Scale (LCSAS) and the Global Simulated-patient Rating Scale (GSPRS), respectively. Each student was assigned a time slot ranging from 1 to 24, where 1, for example, would denote that the student entered the exam first and 24 indicates the final slot for entry into the examination. The number of students who failed this exam was noted for each of the 24 time slots. A control set of marks from a communication skills written exam was also adopted for exploring a possible link with the time slot. Analysis was conducted using graphical display, covariate analysis and logistic regression. RESULTS: No significant relationship was found between the schedule point that the student entered the OSCE exam and their performance. The reliability of the content expert and simulated-patient assessments was stable throughout the session. CONCLUSION: No evidence could be found that duration of examining in a communication OSCE influenced examiners and the marks they awarded. Checks of this nature are recommended for routine inspection to confirm a lack of bias.     

Formative assessment of the consultation performance of medical students in the setting of general practice using a modified version of the leicester assessment package

OBJECTIVE: To evaluate the use of a modified version of the Leicester Assessment Package (LAP) in the formative assessment of the consultation performance of medical students with particular reference to validity, inter-assessor reliability, acceptability, feasibility and educational impact. DESIGN: 180 third and fourth year Leicester medical students were directly observed consulting with six general practice patients and independently assessed by a pair of assessors. A total of 70 practice and 16 departmental assessors took part. Performance scores were subjected to generalizability analysis and students' views of the assessment were gathered by questionnaire. RESULTS: Four of the five categories of consultation performance (Interviewing and history taking, Patient management, Problem solving and Behaviour and relationship with patients) were assessed in over 99% of consultations and Physical examination was assessed in 94%. Seventy-six percent of assessors reported that the case mix was 'satisfactory' and 20% that it was 'borderline'; 85% of students believed it to have been satisfactory. Generalizability analysis indicates that two independent assessors assessing the performance of students across six consultations would achieve a reliability of 0.94 in making pass or fail decisions. Ninety-eight percent of students perceived that their particular strengths and weaknesses were correctly identified, 99% that they were given specific advice on how to improve their performance and 98% believed that the feedback they had received would have long-term benefit. CONCLUSIONS: The modified version of the LAP is valid, reliable and feasible in formative assessment of the consultation performance of medical students. Furthermore, almost all students found the process fair and believed it was likely to lead to improvements in their consultation performance. This approach may also be applicable to regulatory assessment as it accurately identifies students at the pass/fail margin.     

Medical students' essay-writing skills: criteria-based self- and tutor-evaluation and the role of language background

OBJECTIVES: An exercise is described which aimed to make clear to first-year undergraduate medical students the expected writing skills required for an essay examination in one discipline. SUBJECTS: Many students were from a non-English speaking background and over one-third of students, regardless of language background, had limited experience in this type of essay writing. PROCEDURE: For this exercise, a practice essay was written by each student for formative assessment. The essay was rated by a tutor and by the student according to well-defined criteria. This allowed for comparisons to be made in a structured and objective way between the judgements of the student and the assessor. RESULTS: Students found the exercise to be very useful, although whether essay writing skills actually improved could not be established. Students from non-English speaking backgrounds tended to be most harsh in their self-evaluations, yet tutor-evaluations generally showed these students to have better writing skills than other students. Indeed, correlations between self- and tutor-evaluations were quite low. CONCLUSIONS: It is evident that students and their educators may be unclear about each others' expectations. By making explicit the requirements of an exercise, misunderstandings may be minimized and it is possible that student performance could improve, though further research is required to verify these hypotheses. It is suggested that students should be encouraged to evaluate their own work and should be instructed in writing skills throughout their medical degree education.     

Ethical reasoning and decision-making in the clinical setting: assessing the process

Clinical ethical reasoning and analysis are skills as central to good patient care as the efficient application of biomedical knowledge to diagnosis and prognosis. However, experience in teaching clinical ethics to senior medical students has indicated that simply trying to ‘apply’ the knowledge learnt about ethical theories, principles, concepts and rules in the clinical setting does not ensure ethical competence in clinical decision-making. In 1992, we developed and piloted a three-session programme that focused on a more systematic approach to the way students identified and attempted to manage ethical issues in their clinical practice. This programme was modified and improved in 1993 and further expanded in 1994. Our experience suggests that many students are now better able to bridge what has been called the ‘gap’ between the possession of ethical knowledge and its actual use in clinical decision-making. The remaining problem was assessment. How do you assess clinical ethical reasoning and decision-making? In the preclinical years of medical education, knowledge-based assessment tools, like the modified essay question (MEQ), provide a means for assessing the sensitivity of students to ethical issues. However, such tools permit neither an appraisal of how students actually make clinical ethical decisions, nor which factors students perceive as important in making an actual clinical decision. In order to make this type of appraisal, we developed a format for a written case report that facilitated our assessing the process as well as the end-product, the decision. The results obtained from the use of the written case report in the annual assessment of our senior medical students in 1993, while encouraging, also identified a number of unexpected problems.     

The Objective Structured Video Exam for assessment of communication skills

OBJECTIVES: (i) To design a new, quick and efficient method of assessing specific cognitive aspects of trainee clinical communication skills, to be known as the Objective Structured Video Exam (OSVE) (Study 1); (ii) to prepare a scoring scheme for markers (Study 2); and (iii) to determine reliability and evidence for validity of the OSVE (Study 3). METHODS: Study 1 describes how the exam was designed. The OSVE assesses the student's recognition and understanding of the consequences of various communication skills. In addition, the assessment taps the number of alternative skills that the student believes will be of assistance in improving the patient-doctor interaction. Study 2 outlines the scoring system that is based on a range of 50 marks. Study 3 reports inter-rater consistency and presents evidence to support the validity of the new assessment by associating the marks from 607 1st year undergraduate medical students with their performance ratings in a communication skills OSCE. SETTING: Medical school, The University of Liverpool. RESULTS: Preparation of a scoring scheme for the OSVE produced consistent marking. The reliability of the marking scheme was high (ICC=0.94). Evidence for the construct validity of the OSVE was found when a moderate predicted relationship of the OSVE to interviewing behaviour in the communication skills OSCE was shown (r=0.17, P < 0.001). CONCLUSION: A new video-based written examination (the OSVE) that is efficient and quick to administer was shown to be reliable and to demonstrate some evidence for validity.     

A simple eye model to objectively assess ophthalmoscopic skills of medical students

OBJECTIVES: Ophthalmoscopy is an important clinical skill that is essential for medical students to master. Competency in the performance of this skill needs to be assessed objectively. DESIGN: The development of a simple, cheap eye model for objective assessment of ophthalmoscopic skills of medical students is described. SETTING: University of Liverpool. SUBJECTS: Undergraduate medical students. RESULTS: The model was used in 803 assessments and showed a high level of student performance, based on both checklist marking of the general approach to the examination and objective marking of the ability of students to manipulate the light beam, focus the lens and systematically examine the model's fundus. CONCLUSIONS: The method described provides a simple, cost-effective, objective assessment of the performance of ophthalmoscopy.     

Comparison of methods for teaching clinical skills in assessing and managing drug-seeking patients

AIMS: New medical graduates lack clinical skills in assessing and managing patients seeking drugs of dependence. This study compares the effectiveness of three different clinical skills training methods, with similar content, which were developed to teach these skills to senior medical students. METHODS: A preliminary survey indicated that common problems seen by primary care practitioners included both new and previously known patients seeking either benzodiazepines or opiates. The common content of the teaching was determined from this survey. A didactic small group tutorial (DT), a video-based tutorial (VBT) using professional actors, and a computer-aided instruction package using digitized video (CAI) were developed with this common content, and trialled with undergraduate medical students over 2 years in a parallel-group design. Outcome was assessed by student feedback, performance on a case-based written examination and by a structured evaluation of interviews with simulated patients requesting drugs. Comparison was also made between methods on the basis of knowledge tests. RESULTS: No difference was seen in written examination and simulated patient outcomes between the three groups. However, the VBT was thought by the students to be preferable to other methods. The estimated development costs of CAI were higher, but total costs over a 6-year period were lower than for the DT and VBT. The results suggest that clinical skills can be taught equally effectively through several different methods. Collaboration between institutions in the development of widely applicable CAI tools should be an efficient and economical mode of teaching with a wide range of applications.     

Assessment in general practice: the predictive value of written-knowledge tests and a multiple-station examination for actual medical performance in daily practice

This study compares the predictive values of written-knowledge tests and a standardized multiple-station examination for the actual medical performance of general practitioners (GPs) in order to select effective assessment methods to be used in quality-improvement activities. A comprehensive assessment was performed in four phases. First, 100 GPs from the southern part of the Netherlands were assessed by a general medical knowledge test and by a knowledge test on technical skills. Second, in order to check for time-order effects, participants were randomly divided into two groups of 50 each, comparable on scores of both knowledge tests and on professional characteristics. Finally, both groups went through a multiple station examination using standardized patients and a practice video assessment of real surgery, but in opposite orders. Consultations were videotaped and assessed by well-trained peer observers. The drop-out rate was 10%. In both groups the predictive value of medical knowledge tests, ranging from 0.43 to 0.56 (Pearson correlation disattenuated), proved to be comparable with the predictive value of the multiple-station examination for actual performance (0.33-0.59). The overall explained variance of scores of the practice video assessment, measured by multiple regression analysis with performance scores as dependent variables and scores on the knowledge tests and the multiple-station examination as independent variables was moderate (19%). A time-order effect showed in only one direction: from practice video assessment to the multiple-station examination. The GP's professional characteristics did not contribute to the explanation of variation in performance. Medical knowledge tests can predict actual clinical performance to the same extent as a multiple-station examination. Compared with a station examination, a knowledge test may be a good alternative method for assessment the procedures of a large number of practising GPs.     

Use of a journal club and letter-writing exercise to teach critical appraisal to medical undergraduates

INTRODUCTION: There is growing interest in methods of teaching critical appraisal skills at undergraduate and postgraduate levels. We describe an approach using a journal club and subsequent letter writing to teach critical appraisal and writing skills to medical undergraduates. METHODS: The exercise occurs during a 3-week public health medicine attachment in the third year of the undergraduate curriculum. Students work in small groups to appraise a recently published research paper, present their findings to their peers in a journal club, and draft a letter to the journal editor. Evaluation took place through: informal and formal feedback from students; number of letters written, submitted and published, and a comparison of marks obtained by students submitting a literature review assignment with and without critical appraisal teaching during the public health attachment. RESULTS: Feedback from students was overwhelmingly positive. In the first 3(1/2) years, 26 letters have been published or accepted for publication, and 58 letters published on the Internet. There were no significant differences in overall marks or marks for the critical appraisal component of the literature review assignments between the two student groups. DISCUSSION: We believe our approach is an innovative and enjoyable method for teaching critical appraisal and writing skills to medical students. Lack of difference in marks in the literature review between the student groups may reflect its insensitivity as an outcome measure, contamination by other critical appraisal teaching, or true ineffectiveness.     

The use of an objective structured clinical examination (OSCE) for formative and summative assessment in a general practice clinical attachment and its relationship to final medical school examination performance

OBJECTIVES: To report the use of OSCEs for both formative and summative purposes within a general practice undergraduate clinical attachment and to compare student performance in the departmental OSCEs with that of their final medical school examinations. METHODS: Twenty-eight students rotated through the attachment and undertook pre- and post-attachment OSCEs of similar format but different content. Results were analysed to determine relationships between mean scores in the two OSCEs and student performance in their final medical school MBBS examinations. RESULTS: There was a marked improvement in all OSCE station scores. Pre-attachment scores for those stations measuring physical examination and problem-solving skills were unrelated to prior clinical experience. Post-attachment OSCE mean scores were significantly correlated with final examination OSCE and total mean scores. CONCLUSION: The general practice attachment appears to upgrade those clinical skills measured by the pre- and post-attachment OSCE, however, there was no control group of students. Problem-solving and focused physical examination skills need to be targeted by all undergraduate clinical departments. The department's post-attachment OSCE and total assessment results are predictors of final examination OSCE and total results. The use of pre- and post-attachment OSCEs facilitates both students' formative learning processes and the department's evaluation of its educational programme.     

Medical students proposing questions for their own written final examination: evaluation of an educational project

BACKGROUND: In order to emphasise learning more than control, from autumn 2000 we have invited medical students to propose questions for their own written examination in family medicine. One out of three student's proposals was guaranteed to be a part of their coming written examination, possibly somewhat modified. AIM: To evaluate how sixth year medical students experienced the project, and to what extent their performance in the examination was influenced. PARTICIPANTS: Sixth year medical students. MAIN OUTCOME MEASURES: The project was evaluated using (i) marks in examination; (ii) scores on self-administered questionnaires; and (iii) students' free text evaluation. RESULTS: Fifty-seven of 64 (89%) students taking their examination in autumn 2000, and 56 of 59 (95%) students taking the exam in spring 2001, responded. In autumn 2000, 34 (60%) students reported that the project had changed their learning strategies. During spring 2001, 46 of 56 students participated in producing questions, using a mean of 2.6 hours on the work. Students got 5-7% higher marks on their own questions on a scale ranging from 1 to 12. The students' free text evaluation showed that they had prepared especially thoroughly for the topics proposed by the students. They found it comforting to know at least one of the questions in the examination, and the students' questions were found relevant for general practice. CONCLUSION: Encouraging students to write questions for their own examination makes them feel more confident during the examination period, and may increase their reflective learning, without seriously limiting topics studied or violating the control function of the examination.     

Teaching and assessing clinical skills: a competency-based programme in China

The objective of this study was to develop a compentency-based clinical skills teaching and assessment programme in China utilizing modern teaching techniques. Medical teachers from three schools agreed on items for inclusion in the complete physical examination of an asymptomatic adult, an outline for an adult and paediatric history, and important interviewing skills. Lesson plans, performance checklists, and written and videotape training materials were developed. Standardized patients were trained at one school to assist with the teaching at that school and with the assessment at all three schools. A national, a provincial, and a local medical school in China were used. Before beginning the new curriculum for students in their first year of clinical training, baseline data were collected on skills of students at various levels of training in the previous curriculum at all three schools. Although in the previous curriculum there was some improvement in clinical skills among advanced compared to more junior students, performance was lower than expected by staff. One year after implementation of the new curriculum, students were evaluated. These students significantly outperformed their counterparts as well as the more senior level students tested the previous year. This project has established a competency-based teaching and assessment programme in China that allows for rapid improvement in the clinical skills of students. Within a short time, a sophisticated group of medical educators has been formed, who now function as consultants to other educators in their own country. Many aspects of this programme are being adapted throughout China and are applicable to medical schools throughout the world.     

A needs-based study and examination skills course improves students' performance

BACKGROUND: Adult learning theory suggests that learning is most effective when related to need, when driven by the learner and when it is flexible. We describe the effect of an educational intervention that was driven by student need, and largely designed by students. METHODS: We undertook a needs assessment of fifth year medical students' study needs. Based on this, we helped them design a course to meet these needs. This was predominantly related to study skills and a practice objective structured clinical examination (OSCE). We evaluated the course by asking for student opinion and by measuring the effect on student performance in a high stakes medical school examination (written examination and OSCE). FINDINGS: Despite the course being run voluntarily and in after-hours sessions, 80-90% of the medical student class attended each session. Student performance on the end of year examinations was significantly enhanced in the year of the intervention, compared with previous years and with students from other schools sitting identical examinations in the same year. INTERPRETATION: Learning activities that are directly based on student needs, that focus on study and examination techniques, and that are largely student-driven, result in effective and valuable outcomes.     

Teaching sexual history taking to medical students and examining it: experience in one medical school and a national survey

OBJECTIVES: To assess the feasibility and acceptability of training and examining medical students in taking a sexual history and to compare practice with other medical schools in the UK. DESIGN: A training programme involving group work, role play and clinical attachments was developed and applied to 131 students at the University of Bristol Medical School. They then underwent an objective structured clinical assessment using simulated patients. The practice of other medical schools was surveyed by postal questionnaire. RESULT: The students felt that the examination was a good test of their knowledge and skills. One student failed. Sexual history taking is taught in 17 of 22 medical schools but examined in only six. CONCLUSION: Both teaching and examining of sexual history taking skills are possible and are likely to occur increasingly in UK medical schools.     

Undergraduate surgical examinations: an appraisal of the clinical orals

The aims of this study of examinations in the final year of undergraduate surgery were to determine whether the long case, short cases/orals reflected students' knowledge of surgery effectively, whether their performance indicated the goals of the medical course relative to surgery were achieved, and whether these examinations gave a good and fair appraisal of student clinical skills. To test validity of these examinations, the scores of the long and short cases/oral were correlated with the aggregate score for the medical course, to the final-year total score, the final-year surgery mark, the written examination mark and the final-year internal medicine mark. Whether the surgical goals of the medical course were achieved was assessed by questionnaire to the surgical examiners, and whether there was a good and relevant appraisal of student clinical skills was assessed by questionnaire to students. The response rates to the questionnaires were 100% and 98%. The results showed a poor but significant correlation between long and short cases/oral (rho = 0.24, P = 0.01). The correlation of the long case and short cases/oral to internal medicine was rho = 0.45 and rho = 0.33 (both P < 0.001), and there was a better correlation of short cases/oral to the aggregate medical course score (rho = 0.53 vs. 0.42, both P < 0.001). The long case and short cases/oral had no correlation with the written paper (rho = 0.10 and 0.69, respectively). The goals of the medical course were considered achieved, but less so with assessment of the injured and acutely surgically ill; students considered the long and short cases/orals as a good and relevant appraisal of their clinical skills. The distribution of short cases marks shifted to the left, indicating lower scores for this part of the examination. In conclusion, this study has demonstrated that the long and short cases/orals tested different aspects of student's skills, and so reinforces the importance of retaining the short and long cases in undergraduate surgical examinations. It is suggested that a single long case may be insufficient to distinguish between adequate and inadequate performance.     

Structured packs for independent learning: a comparison of learning outcome and acceptability with conventional teaching

OBJECTIVES: The GMC recommends that students become independent learners, while tutor time is an increasingly precious resource. A set of structured learning materials requiring students to undertake and reflect on practical tasks in five learning areas was developed. DESIGN: The study used a randomized control trial to evaluate the effectiveness of using these structured learning materials in place of conventional teaching for 228 third-year undergraduate students and 55 teachers, on both hospital and community based medical and general practice firms. Evaluation involved assessing student performance on an examination question and a writing task, together with a student and tutor satisfaction questionnaire. SETTING: King's College School of Medicine and Dentistry, London. SUBJECTS: Third-year medical (first-year clinical) undergraduates. RESULTS: No significant difference in learning outcome was found for students on community- and hospital-based medical and general practice firms between students who had used the structured materials and those who had had conventional teaching on the same topic. The packs were acceptable to tutors and students. CONCLUSIONS: Such resources represent a mid-point between formal didactic teaching and self-directed learning. They may be particularly suitable for promoting independent learning for students on traditional medical courses. They offer an appropriate way to cover certain topics in the clinical curriculum and help to protect tutor time for topics which cannot be effectively taught in other ways.     

Providing early clinical experience in primary care

Undergraduate courses in British medical schools are changing following recommendations from the General Medical Council. Increasing emphasis has been placed on teaching in the community. Nottingham Medical School has pioneered the teaching of basic clinical skills in primary care during the pre-clinical course to help produce an integrated curriculum. This qualitative study evaluated the first two years of the new early clinical experience course at Nottingham by using interviews with 19 students and their GP tutors. Students claimed to have gained confidence in talking to patients, their understanding of the role of the doctor and the importance of the doctor-patient relationship. Students were less confident about examining patients and some reported having had little opportunity to practice examination skills. Half the students thought that the early clinical visits had helped them to understand and be more motivated to learn their basic medical sciences course. The newly recruited GP teachers were highly motivated, very positive about the early clinical teaching and all wanted to continue to teach the pre-clinical students. Difficulties in providing the course included communication with students and staff, organization of student travel and variation in the quality of teaching. However, the Nottingham early clinical experience course has shown that basic clinical skills can be successfully taught to pre-clinical students in primary care.     

Can medical schools rely on clerkships to train students in basic clinical skills?

OBJECTIVE: Many medical schools have drawn up lists of basic clinical skills that students are required to have mastered at the end of medical training. To determine whether undergraduate students actually perform these basic clinical skills during clerkships and whether different approaches to skills training led to different results, we surveyed 365 final-year medical students in 1996 and 1997. METHOD: A questionnaire containing items on 265 skills in eight body systems was administered to students from two conventional medical schools (Ghent and Antwerp, Belgium), and one Dutch medical school, Maastricht, which offers a problem-based curriculum and systematic skills training. RESULTS: Although quite a few skills were not performed by Maastricht students, the results of this school compared favourably to those of the Ghent and Antwerp medical schools. Significant differences between Ghent and Antwerp were found for surgery, paediatrics and gynaecology/obstetrics. In the non-obligatory clerkships in dermatology, otorhinolaryngology and ophthalmology a great percentage of skills were not performed. CONCLUSIONS: The main conclusion is that all three medical schools cannot rely on clerkship experiences alone to provide adequate basic skills training. A problem-based learning environment and training in a skills laboratory appear to result in students performing more skills during clerkships. Assessment of clinical skills, obligatory clerkships in specialties and general practice, and continuous monitoring of the quality of clerkships may also be strong determinants of the present findings.