The passing score in the Objective Structured Clinical Examination

The objective structured clinical examination (OSCE) now has an established place in the assessment of the medical undergraduate. While much has been written about the reliability of the OSCE, empirical work on the determination of the passing score which represents competence on the OSCE is rarely encountered. If the OSCE is to play its role in the 'high stakes' testing of clinical competence, it is important that this passing score be set reliably and defensibly. This article illustrates how a two-session modified Angoff standard-setting procedure is used to set the passing score on a 14 station Obstetrics and Gynaecology OSCE used to assess final year students at The Queen's University of Belfast. The Angoff methodology harnesses the professional judgement of expert judges to establish defensible standards. Four university teachers, five non-academic consultants and six junior clinical staff took part in a two-session Angoff standard-setting procedure. In the first session, the judges (individually and in silence) used their professional judgement to estimate the score which a minimally competent final year obstetrics and gynaecology student should achieve on each tested element of the OSCE. In the second session they revised their session 1 judgements in the light of the OSCE scores of real students and the opportunity for structured discussion. The passing score for the OSCE is reported together with the statistical measures which assure its reliability.     

Competency domains in an undergraduate Objective Structured Clinical Examination: their impact on compensatory standard setting

Context  Following a 15-week attachment in paediatrics and child health, general practice and dermatology medical students in their second clinical year at this medical school undertake a high-stakes assessment including an objective structured clinical examination (OSCE). There were 2 hypotheses. Firstly, groups of similar stations map to competency domains identifiable by factor analysis. Secondly, poor performance in individual domains is compensated for by achieving the required standard of performance across the whole assessment.
Methods  A total of 647 medical students were assessed by an OSCE during 5 individual examination sittings (diets) over 2 years. Ten scoring stations in the OSCE were analysed and confirmatory factor analysis performed comparing a 1-factor model (where all the stations are discrete entities related to one underlying domain) with a 3-factor model (where the stations load onto 3 domains from a previously reported exploratory factor analysis).
Results  The 3-factor model yielded a significantly better fit to the data (χ² = 15.3, P  <   0.01). Assessing the compensation data of 1 diet, 29 of 127 students failed in 1 or more domains described, whereas only 5 failed if compensation was allowed across all domains.
Discussion  Confirmatory factor analysis showed a significant fit of the data to previously described competency domains for a high-stakes undergraduate OSCE. Compensation within but not between competency domains would provide a more robust standard, improve validity, and substantially reduce the pass rate.     

The effectiveness of immediate feedback during the Objective Structured Clinical Examination

Using eight different physical examination or technical stations, 400 examinations were conducted to evaluate the effectiveness of immediate feedback during the Objective Structured Clinical Examination (OSCE). The test group comprised 50 medical students who underwent a standard 4-minute examination followed by 2 minutes of feedback. Immediately following feedback the students repeated an identical 4-minute examination scored by the same examiners. The control group consisted of 50 students from the same class who underwent an identical testing sequence, but instead of receiving feedback, they were instructed to continue their examinations for an additional 2 minutes before repeating the stations. Simple repetition of the task did not significantly improve score (mean increase 2.0%, NS). Extending the testing period from 4 to 6 minutes resulted in a small but significant increase in score (mean 6.7%, P less than 0.001). However, there was a much larger increase in the scores obtained following 2 minutes of immediate feedback compared to pre-feedback performance (mean 26.3%, P less than 0.0001). The majority of students and examiners felt that feedback, as administered in this study, was valuable both as a learning and teaching experience. Short periods of immediate feedback during an OSCE are practical and can improve competency in the performance of criterion-based tasks, at least over the short term. In addition, such feedback provides students with valuable self-assessment that may stimulate further learning.     

Providing feedback to students on clinical skills by using the Objective Structured Clinical Examination

Student opinions of some methods of how and when to provide feedback on clinical skills in the Objective Structured Clinical Examination (OSCE) were gathered. Feedback with their marked checklists plus a model videorecording was preferred to either on its own. Feedback was preferred during the examination rather than after it. No undue stress appeared to be associated with receiving feedback during the examination. Valuable extra instruction to students can be provided with minimal staff input.     

Structured teaching methods enhance skill acquisition but not problem-solving abilities: an evaluation of the 'silent run through'

OBJECTIVES: We aimed to evaluate whether a structured teaching technique (the silent run through) was of benefit in the learning of a skill sequence, and whether it would affect students' ability to improve their problem-solving skills or ability to make judgements. DESIGN: Students from one hospital were taught two different complex skills using a highly structured teaching method involving the breakdown of complex tasks into smaller components and the utilization of an internal, silent 'commentary'. Their subsequent ability to reproduce these skills was compared with a second group of students taught at another hospital. All students were assessed during a common end of attachment Objective Structured Clinical Examination on the two skills, by raters blind to the teaching techniques the students had received. SETTING: School of Medicine, University of Leeds, UK. SUBJECTS: Medical students. RESULTS: Students who received the structured teaching were significantly better at reproducing a complex, sequential clinical skill (examination of a squint). There was no demonstrable improvement in problem-solving skills as assessed on an X-ray interpretation station. Students expressed high levels of satisfaction with learning skills in such a structured way and reported increased confidence in their examination skills. CONCLUSIONS: This study provides some evidence to support the hypothesis that different teaching techniques may be more effective for improving different elements of skills learning. In particular, a highly structured technique involving breaking complex tasks down into smaller components and utilizing an internal 'commentary' may be an effective way of teaching the sequential motor components of complex clinical skills.     

The Objective Structured Video Exam for assessment of communication skills

OBJECTIVES: (i) To design a new, quick and efficient method of assessing specific cognitive aspects of trainee clinical communication skills, to be known as the Objective Structured Video Exam (OSVE) (Study 1); (ii) to prepare a scoring scheme for markers (Study 2); and (iii) to determine reliability and evidence for validity of the OSVE (Study 3). METHODS: Study 1 describes how the exam was designed. The OSVE assesses the student's recognition and understanding of the consequences of various communication skills. In addition, the assessment taps the number of alternative skills that the student believes will be of assistance in improving the patient-doctor interaction. Study 2 outlines the scoring system that is based on a range of 50 marks. Study 3 reports inter-rater consistency and presents evidence to support the validity of the new assessment by associating the marks from 607 1st year undergraduate medical students with their performance ratings in a communication skills OSCE. SETTING: Medical school, The University of Liverpool. RESULTS: Preparation of a scoring scheme for the OSVE produced consistent marking. The reliability of the marking scheme was high (ICC=0.94). Evidence for the construct validity of the OSVE was found when a moderate predicted relationship of the OSVE to interviewing behaviour in the communication skills OSCE was shown (r=0.17, P < 0.001). CONCLUSION: A new video-based written examination (the OSVE) that is efficient and quick to administer was shown to be reliable and to demonstrate some evidence for validity.     

Experiences of volunteer patients during undergraduate examinations: printed information can lead to greater satisfaction

BACKGROUND: Whilst most patients are happy to participate in medical examinations, previous studies have shown that some are dissatisfied with the information that they receive beforehand. OBJECTIVES: To demonstrate that patient satisfaction can be improved by the provision of written information. METHODS: Thirty-seven patients attending the final MBBS examination were sent written information about what to expect. The patients' experiences were assessed by means of an anonymous postal questionnaire 1 week after the examination. Their satisfaction with the level of information received beforehand was compared with that of an historical cohort who had received no written information. RESULTS: The patients had few complaints about their experience of participating in examinations and many enjoyed it. Only 5% of patients registered dissatisfaction with the level of information that was provided, compared with 18% in the previous study (P = 0.077). CONCLUSIONS: Written information led to a (near significant) improvement in patients satisfaction when compared to a previous cohort. We recommend that all patients attending for medical examinations should be provided with written information.     

客观结构式临床测验在双语教学中的应用

目的将简化的客观结构式临床测验（Objective Structured Clinical Examination,OSCE）引入临床双语教学,研究其提高双语教学效率的作用,以探讨建立一种新的双语教学模式的可行性。方法将OSCE中几种临床常见疾病的病史采集、体格检查等相关内容译成双语,以培训过的临床医师扮演标准化病人（Standardized Patients,SP）,训练学生来进行双语的病史采集和体格检查,最后对学生在规定时间内独立完成的相应项目进行评估。结果经过OSCE方式训练过的学生,能较好地运用英语进行问诊及体查。结论客观结构式临床测验对提高双语教学的效率有一定作用。     

An innovative problem-solving assessment for groups of first-year medical undergraduates – Think Tanks

A problem-solving assessment has been devised and used for groups of first-year medical students at Southampton University. Five cohorts of ≈160 students have taken the assessment since 1990. The assessment was part of a Locomotor system course that integrated biological and behavioural sciences and clinical specialities. The course also included lectures, practicals and tutorials and occupied the first 7 weeks of the last (10-week) term. Other assessments of the course involved essay writing and a spotter-type practical. For the problem-solving assessment, groups of about six students (`Think Tanks') received a problem which they had to investigate during the course. The students had some choice in the problem they were to study. The problems were designed, by staff, to cover aspects of movement in the context of health and disease. A staff advisor was assigned to each Think Tank group to provide general support. The results of the investigations of each Think Tank were displayed at the end of the course in the form of a poster aimed at informing their peers. Three members of staff gave each Think Tank group a communal oral examination and the performance of the group was given a grade. Each member of the group, normally, was allowed to add this grade to their marks for the whole year.     

An evaluation of the clinical problem-solving process using a simulation technique

Currently, research does not agree as to the extent to which medical content and problem-solving processes underlie clinical problem-solving. The results of research in this area fall into two categories: (1) clinical problem-solving is primarily dependent upon medical content specific within the case, and (2) clinical problem-solving is a skill, or series of skills, which can be applied to all clinical problems. In the study reported in this paper, seventy-one second-year medical students who had completed a 2-year, body-systems oriented curriculum were given an examination designed to measure clinical problem-solving. The results indicated that gathering data on patient history and hypotheses generation were specific skills common to clinical problem-solving, while hypotheses refinement, identification of relevant physical examinations, ordering laboratory investigations and making a diagnosis were case-related.     

An approach to medical curriculum evaluation

The advent of new trends in medical education makes it necessary for new and established medical schools to evaluate their curricula. The changing needs of society emphasize that this process should be continuous. There are, however, many difficulties which impede the initiation of curriculum evaluation. The College of Medicine and Allied Health Sciences at King Abdulaziz University initiated and carried out a plan for programme evaluation in 1984-85. The pathway selected for this experience was based on existing models which were modified to suit the local needs. The formulation of the plan was guided by general principles aiming at participation, communication, data-gathering from numerous sources and utilization of appropriate resources. The main direction of the evaluation was formative in nature and was carried out internally. The results were analysed and revealed major problems in the existing design and the teaching/learning environment. The recommendations for change were guided by educational criteria concerned with promotion of learning, problem-solving and community orientation. The main emphasis in the presentation of this report is on the process more than the content of curriculum evaluation.     

Assessing simulated patients in an educational setting: the MaSP (Maastricht Assessment of Simulated Patients)

CONTEXT: For more than two decades the Medical School in Maastricht, the Netherlands, has used simulated patients (SPs) to provide students with opportunities to practise their skills in communication and physical examination. In this educational setting a student meets a SP in a videotaped session. Feedback by the SP to the student at the end of the session is considered an important educational feature. We found no instruments to assess individual SP performance during those sessions. OBJECTIVE: To develop a valid, reliable and feasible instrument to evaluate the performance of SPs. METHODS: The content of the instrument was validated through interviews with students, teachers and experts who are involved with SPs. They were asked to indicate key features of good SP performance. Based on the interviews, a written checklist was developed to measure individual SP performance. The instrument was evaluated in a regular SP session at the medical school, involving 152 students and their teachers. MAIN OUTCOMES: All interviewees considered the scale to be satisfactory and the instrument to be valid. The feasibility and reliability of the checklists were investigated using the data of 398 returned checklists. Cronbach's alpha was found to be 0.73. Generalizability analysis showed that 12 completed checklists were required to obtain a reliable assessment of one SP. CONCLUSIONS: The Maastricht Assessment of Simulated Patients (MaSP) appears to be a valid, reliable and feasible tool to assess the performance of SPs in an educational setting.     

An Objective Structured Clinical Examination to assess preclinical skills

Aim: To describe the use of a preclinical Objective Structured Clinical Examination for the assessment of clinical skills in undergraduate dietetic students. Methods: The Objective Structured Clinical Examination is a well-established assessment tool in medicine, nursing and some allied health disciplines, but has been rarely used in either formative or summative assessment of dietetic students. This paper describes the development, implementation and evaluation of a preclinical Objective Structured Clinical Examination for dietetic students. Results: Students' performance in a preclinical Objective Structured Clinical Examination was assessed over a six-year period in successive student cohorts. Performance in the Objective Structured Clinical Examination was significantly related to their performance at their first clinical placement. Conclusions: A preclinical dietetic Objective Structured Clinical Examination allows problems arising during the teaching of basic clinical skills to be noted and corrected. In addition, a preclinical Objective Structured Clinical Examination identifies students requiring additional support on placement.     

An evaluation of a course for undergraduate teaching of general practice

This paper presents an evaluation of the undergraduate fifth year course of teaching in general practice in the Queen's University, Belfast. Two Modified Essay Question papers were randomly selected from those used for some years past in the Department of General Practice, as learning aids and class tests. These were administered to twelve randomly selected groups of students before and after completion of a five week course of teaching. Results show that the composite mean MEQ scores doubled between pre- and post-course tests. The correlation coefficients tabulated do not indicate that the marked improvement in mean scores before and after the course in general practice was influenced to any extent by knowledge acquired in other disciplines. The overall findings clearly establish the value of small group clinical teaching of general practice based on a problem orientated approach to learning. They suggest that the intellectual skills and attitudes required in the 'holistic' approach to problem solving in general practice may differ in some ways from the skills required in specialized areas of medicine. The evaluation justifies the time spent on clarifying the undergraduate educational objectives.     

A randomized controlled study of portfolio learning in undergraduate cancer education

Portfolio learning has not previously been reported for clinical undergraduate teaching. This open randomized study aimed to assess the effect of portfolio learning in the teaching of oncology to medical students. The project aimed to provide the student with a holistic understanding of the impact of the disease and its treatment on the patient and family, and the natural history of malignant disease, through long-term personal experience of a cancer patient. All undergraduate medical students entering Clinical Studies in October 1992 at the University of Wales College of Medicine were randomized to a study or control group. Both groups continued with the standard curriculum. Each study-group student followed a patient with cancer for 9 months, supported by bi-monthly small-group tutorials. Tutors were either general practitioners or hospital consultants, not necessarily oncologists; each was supplied with a tutor's resource pack of key oncology review papers. Students recorded triggers to learning and key items in a personal learning portfolio. Students' performances in clinical examinations and the contents of their portfolio were assessed. Final assessment was by hidden questions in the objective structured clinical examination (OSCE) in the final degree examination, when students in the study group showed higher marks in factual knowledge of oncology, particularly amongst the weaker students ( P  = 0·01). Those submitting portfolios for formative assessment had higher overall marks than those in the study group who did not ( P  = 0·04), representing the more motivated students. The whole study group showed a beneficial trend in their knowledge of oncology.     

Shared goals, shared learning: evaluation of a multiprofessional course for undergraduate students

Twenty-eight undergraduate degree students from seven health care professions attended a two-day pilot course. Using small multiprofessional groups, final-year students from occupational therapy, orthoptics, therapy radiography, nursing, physiotherapy, medicine and dentistry explored professional roles and clinical problem-solving using a theme-based approach. A balance of didactic and interactive small-group learning enabled them to identify issues surrounding multiprofessional teamworking and collaboration in the National Health Service. Evaluation results showed that the course increased knowledge and understanding of other health care professions, developed more positive attitudes and demonstrated the importance of multiprofessional teamwork and communication. Participating students believed that both early and regular opportunities for shared learning should be essential aspects of undergraduate courses.     

The relationship of tutors' content expertise to interventions and perceptions in a PBL medical curriculum

This study addressed three questions: (1) Do content-expert tutors differ from non-expert tutors in the extent to which they present/explain case content? (2) Do tutors who present/explain case content differ from those who almost never do in their ratings of various outcomes of a PBL curriculum? (3) Are tutors who present/explain case content rated differently by students from tutors who almost never do? Data were gathered from 88 tutors and 168 students in the first 2 years of a PBL medical curriculum. Students assessed their tutor after each unit, and tutors completed a questionnaire near the end of the academic year. In this study, `content expertise' was defined operationally as tutors' self-ratings on the question `To what extent could you teach (at the Med I level) the material covered in the cases?' Less than half of the tutors reported that they almost never presented/explained case content. As tutors' content expertise increased, they tended to present/explain case content more frequently. Tutors who almost never presented/explained case content rated PBL more highly than traditional methods. No differences were found in student ratings of tutors who almost never presented/explained case content, compared to tutors who did. The results suggest that tutors who are content experts find it difficult to maintain the `facilitator' role, but that those who maintain this role are more satisfied with PBL. It appears that other tutor behaviours may have a greater influence on students' ratings of their tutors.     

Gynaecological examination: a teaching package integrating assessment with learning

A self-directed learning package which includes assessment was developed to allow students to acquire the psychosocial and motor skills needed to conduct a gynaecological examination with sensitivity. The package integrates assessment with learning and uses the Assessment Form as an educational instrument. The introduction, including a video of a vaginal examination and pap smear, is followed by five learning stations for groups of 3 or 4 students to visit. At each station a series of tasks are given which enable the objectives of the station to be fulfilled. The materials necessary to complete the tasks are provided. Gynaecological assistants, themselves trained by participation in the learning sessions, guide the students throughout the session and act as adviser `patients' to help students acquire the necessary motor skills. In the assessment, students are required to integrate what they have learnt in the learning stations. The students assess themselves and the `patient' and an observer assesses the student using the same Assessment Form. The form assesses and gives examples of the psychosocial, including legal and motor skills required. The assistants–observers give feedback to the students about their performance and make suggestions for future improvement. Of the 232 medical students completing their clinical terms in Obstetrics and Gynaecology in 1995, those students receiving the learning package rated their practical experience as significantly better. They rated the learning method, content, enjoyment and package overall very highly (median 4, 5 point scale 1–5). Open-ended comments by students confirmed these findings.     

Structured packs for independent learning: a comparison of learning outcome and acceptability with conventional teaching

OBJECTIVES: The GMC recommends that students become independent learners, while tutor time is an increasingly precious resource. A set of structured learning materials requiring students to undertake and reflect on practical tasks in five learning areas was developed. DESIGN: The study used a randomized control trial to evaluate the effectiveness of using these structured learning materials in place of conventional teaching for 228 third-year undergraduate students and 55 teachers, on both hospital and community based medical and general practice firms. Evaluation involved assessing student performance on an examination question and a writing task, together with a student and tutor satisfaction questionnaire. SETTING: King's College School of Medicine and Dentistry, London. SUBJECTS: Third-year medical (first-year clinical) undergraduates. RESULTS: No significant difference in learning outcome was found for students on community- and hospital-based medical and general practice firms between students who had used the structured materials and those who had had conventional teaching on the same topic. The packs were acceptable to tutors and students. CONCLUSIONS: Such resources represent a mid-point between formal didactic teaching and self-directed learning. They may be particularly suitable for promoting independent learning for students on traditional medical courses. They offer an appropriate way to cover certain topics in the clinical curriculum and help to protect tutor time for topics which cannot be effectively taught in other ways.     

An initial evaluation of learning units to facilitate problem solving and self-directed study (portable patient problem pack)

This paper describes the initial evaluation of the portable patient problem pack, a learning unit designed to facilitate the development of problem-solving skills and stimulate self study.
A cross-over design using two neurological patient problems, each presented in two formats (simulated patient and P4), was used with a sample of twenty-nine volunteer nursing and medical students.
The results indicate that the P4 appears to be feasible and effective in stimulating problem solving and self-directed study. The suggested concurrent validity, low cost of production, and implementation make it a potentially potent tool for evaluation.