Formative assessment of the consultation performance of medical students in the setting of general practice using a modified version of the leicester assessment package

OBJECTIVE: To evaluate the use of a modified version of the Leicester Assessment Package (LAP) in the formative assessment of the consultation performance of medical students with particular reference to validity, inter-assessor reliability, acceptability, feasibility and educational impact. DESIGN: 180 third and fourth year Leicester medical students were directly observed consulting with six general practice patients and independently assessed by a pair of assessors. A total of 70 practice and 16 departmental assessors took part. Performance scores were subjected to generalizability analysis and students' views of the assessment were gathered by questionnaire. RESULTS: Four of the five categories of consultation performance (Interviewing and history taking, Patient management, Problem solving and Behaviour and relationship with patients) were assessed in over 99% of consultations and Physical examination was assessed in 94%. Seventy-six percent of assessors reported that the case mix was 'satisfactory' and 20% that it was 'borderline'; 85% of students believed it to have been satisfactory. Generalizability analysis indicates that two independent assessors assessing the performance of students across six consultations would achieve a reliability of 0.94 in making pass or fail decisions. Ninety-eight percent of students perceived that their particular strengths and weaknesses were correctly identified, 99% that they were given specific advice on how to improve their performance and 98% believed that the feedback they had received would have long-term benefit. CONCLUSIONS: The modified version of the LAP is valid, reliable and feasible in formative assessment of the consultation performance of medical students. Furthermore, almost all students found the process fair and believed it was likely to lead to improvements in their consultation performance. This approach may also be applicable to regulatory assessment as it accurately identifies students at the pass/fail margin.     

Childhood developmental examination: a novel approach to teaching

Childhood developmental assessment is an essential skill that medical students must attain during their education. The authors describe a new technique which is neither laborious nor time-consuming for both staff and students. Groups of 40 students were taught normal developmental examinations using a novel interactive approach during a lecture. The method proved successful in terms of student evaluation, cost effectiveness of medical teaching time and acceptability to the children used for the demonstration. Audio-visual equipment, available in most teaching centres, was used that was easily and cheaply installed for the purpose of the session.     

Medical students as standardized patients to assess interviewing skills for pain evaluation

PURPOSE: The use of medical students as standardized patients in a performance assessment of pain evaluation was studied. METHODS: Fifty-two pairs of second-year medical students participated. One student portrayed a patient presenting with cancer pain and was interviewed by the other medical student. The student-patient then rated the interview using a checklist of pain assessment and general interviewing skills. The interviews were audiotaped and also rated independently. RESULTS: Based on student-patient ratings, 36 (69%) students demonstrated 9 or more of the 11 pain-specific checklist items, compared to 34 (65%) students according to the trained rater. Highly specific pain-related items had higher agreement than broader interviewing skill items. There would be differences in the summary assessments of students depending on which rating data were used. DISCUSSION: Medical students represent a readily accessible resource as patients for clinical simulations. Students tended to overestimate the performance of fellow students, but acting as a standardized patient had educational value, and can be used to extend simulated patient encounters within the curriculum. Further investigation is needed to improve the reliability of the feedback provided by student-patients.     

The Objective Structured Video Exam for assessment of communication skills

OBJECTIVES: (i) To design a new, quick and efficient method of assessing specific cognitive aspects of trainee clinical communication skills, to be known as the Objective Structured Video Exam (OSVE) (Study 1); (ii) to prepare a scoring scheme for markers (Study 2); and (iii) to determine reliability and evidence for validity of the OSVE (Study 3). METHODS: Study 1 describes how the exam was designed. The OSVE assesses the student's recognition and understanding of the consequences of various communication skills. In addition, the assessment taps the number of alternative skills that the student believes will be of assistance in improving the patient-doctor interaction. Study 2 outlines the scoring system that is based on a range of 50 marks. Study 3 reports inter-rater consistency and presents evidence to support the validity of the new assessment by associating the marks from 607 1st year undergraduate medical students with their performance ratings in a communication skills OSCE. SETTING: Medical school, The University of Liverpool. RESULTS: Preparation of a scoring scheme for the OSVE produced consistent marking. The reliability of the marking scheme was high (ICC=0.94). Evidence for the construct validity of the OSVE was found when a moderate predicted relationship of the OSVE to interviewing behaviour in the communication skills OSCE was shown (r=0.17, P < 0.001). CONCLUSION: A new video-based written examination (the OSVE) that is efficient and quick to administer was shown to be reliable and to demonstrate some evidence for validity.     

The impact of formal instruction in clinical examination skills on medical student performance -- the example of peripheral nervous system examination

BACKGROUND: Peripheral nervous system examination is an essential part of the full medical clerking of a patient. We have investigated the effectiveness of formal instruction in peripheral nervous system examination compared to the traditional bedside ward teaching that our students usually receive. METHOD: We instructed an unselected group of 22 medical students in peripheral nervous system examination in a clinical skills centre and evaluated them with a 12 item marking schedule before and after instruction. The performance of this group was then compared to the rest of their year (220 students) in an end of year OSCE, which included a neurology station assessing sensory examination of the lower limbs. RESULTS: Students formally instructed in neurology significantly improved their scores after instruction and scored 15% higher marks (90% vs. 75%) than the rest of their year in the end of year neurology OSCE station 2 months later (P < 0.01, Mann Whitney U-test). They did not perform significantly better in the OSCE overall. CONCLUSIONS: Formal instruction in neurological examination resulted in a significant increase in the end of year neurology OSCE station score compared to traditional heterogeneous teaching methods.     

Benefiting from clinical experience: the influence of learning style and clinical experience on performance in an undergraduate objective structured clinical examination

OBJECTIVE: To assess the relationship between clinical experience, learning style and performance in an objective structured clinical examination (OSCE) in medical students at the end of their first clinical year. DESIGN: Prospective study of undergraduate students taking an OSCE examination at the end of their first clinical year. SUBJECTS: 194 undergraduate medical students (95 male). MAIN OUTCOME MEASURES: Performance in the OSCE examination, the Entwhistle Learning Style Inventory1 and a composite self-reported score of clinical activity during the students first clinical year. RESULTS: Performance in the OSCE examination was related to well-organized study methods but not to clinical experience. A significant relationship between clinical experience and organized deep-learning styles suggests that knowledge gained from clinical experience is related to learning style. CONCLUSIONS: The relationship between clinical experience and student performance is complex. Well-organized and strategic learning styles appear to influence the benefits of increased clinical exposure. Further work is required to elucidate the most beneficial aspects of clinical teaching.     

Teaching support in the behavioural sciences for non-English speaking background medical undergraduates

INTRODUCTION: This paper reports on a teaching strategy designed to support first year undergraduate medical students from non-English speaking backgrounds in a behavioural science course taught at an Australian university. METHOD: The strategy is described, as is the language profile of students from two successive cohorts who participated in the teaching programme. The effectiveness of the intervention in improving students' academic performance is explored, as gauged by summative assessment and student perceptions. CONCLUSIONS: Findings suggest that students disadvantaged by difficulties with language and/or knowledge of Australian culture were able to perform at least as well as other students on a number of summative criteria. Non-English speaking background students perceived the behavioural science course to be difficult, and indicated that they found the teaching support offered to them to be useful. The methodological difficulties and limitations involved in the evaluation of a teaching programme such as this are discussed and the conclusions that can be validly drawn are considered.     

Evaluating the outcome of communication skill teaching for entry‐level medical students: does knowledge of empathy increase?

BACKGROUND: While the literature shows the clinical value for medical practitioners of skill in communicating with patients in an empathetic manner, objective evaluations of methods to teach empathy are few. PURPOSES: This paper describes a method of teaching entry-level medical students the elements of effective communication with patients, in preparation for their first practical exercises. The paper focuses on how the outcomes of the teaching were evaluated with special attention to empathy. METHODS: Student evaluative ratings were collected after training, and students also completed a pencil-and-paper test of empathy, both before and after the training. While all data were anonymous, student pre- and post-training empathy scores could be compared to assess individual changes in knowledge of empathy after training. RESULTS: Most students (81%) felt better prepared to interview after the training. The pencil-and-paper measure of empathy has good reliability, both internal (alpha 0.83 and 0.91) and inter-rater (kappa 0.96). Overall, students made significant gains in their ability to make empathetic responses, although some (30%) showed no gains. CONCLUSIONS: Further research is required to identify students who fail to acquire skill in expressing empathy after undergoing training, and to validate the pencil-and- paper measure of empathy against real-life performance.     

Traditional finals and OSCEs in predicting consultant and self-reported clinical skills of PRHOs: a pilot study

INTRODUCTION: As we move from standard 'long case' final examinations to new objective structured formats, we need to ensure the new is at least as good as the old. Furthermore, knowledge of which examination format best predicts medical student progression and clinical skills development would be of value. METHODS: A group of medical students sat both the standard long case examination and the new objective structured clinical examination (OSCE) to introduce this latter examination to our Medical School for final MB. At the end of their pre-registration year, the group and their supervising consultants submitted performance evaluation questionnaires. RESULTS: Thirty medical students sat both examinations and 20 returned evaluation questionnaires. Of the 72 consultants approached, 60 (83%) returned completed questionnaires. No correlation existed between self- and consultant reported performance. The traditional finals examination was inversely associated with consultant assessment. Better performing students were not rated as better doctors. The OSCE (and its components) was more consistent and showed positive associations with consultant ratings across the board. DISCUSSION: Major discrepancies exist between the 2 examination formats, in data interpretation and practical skills, which are explicitly tested in OSCEs but less so in traditional finals. Standardised marking schemes may reduce examiner variability and discretion and weaken correlations across the 2 examinations. This pilot provides empirical evidence that OSCEs assess different clinical domains than do traditional finals. Additionally, OSCEs improve prediction of clinical performance as assessed by independent consultants. CONCLUSION: Traditional finals and OSCEs correlate poorly with one another. Objective structured clinical examinations appear to correlate well with consultant assessment at the end of the pre-registration house officer year.     

Medical students' essay-writing skills: criteria-based self- and tutor-evaluation and the role of language background

OBJECTIVES: An exercise is described which aimed to make clear to first-year undergraduate medical students the expected writing skills required for an essay examination in one discipline. SUBJECTS: Many students were from a non-English speaking background and over one-third of students, regardless of language background, had limited experience in this type of essay writing. PROCEDURE: For this exercise, a practice essay was written by each student for formative assessment. The essay was rated by a tutor and by the student according to well-defined criteria. This allowed for comparisons to be made in a structured and objective way between the judgements of the student and the assessor. RESULTS: Students found the exercise to be very useful, although whether essay writing skills actually improved could not be established. Students from non-English speaking backgrounds tended to be most harsh in their self-evaluations, yet tutor-evaluations generally showed these students to have better writing skills than other students. Indeed, correlations between self- and tutor-evaluations were quite low. CONCLUSIONS: It is evident that students and their educators may be unclear about each others' expectations. By making explicit the requirements of an exercise, misunderstandings may be minimized and it is possible that student performance could improve, though further research is required to verify these hypotheses. It is suggested that students should be encouraged to evaluate their own work and should be instructed in writing skills throughout their medical degree education.     

Development and integration of CAL: a case study in medicine

CONTEXT: The Faculty of Medicine and Medical Sciences at the University of Aberdeen viewed the use of Computer Assisted Leaning (CAL) and other IT based learning resources as a possible way of coping with an increase in student numbers whilst maintaining or increasing the quality of medical teaching. OBJECTIVES: Our primary objective was to develop and integrate Computer Assisted Learning (CAL) applications into the undergraduate medical curriculum. SUBJECTS/MATERIALS: A wide spectrum of CAL applications were developed dealing with many topics in the curriculum. METHODS: We formulated a structured approach to CAL development by establishing a team of professionals (forming a CAL Unit), using existing expertise and by implementing a process to ensure that the CAL had a maximum impact upon the curriculum. The CAL included multimedia tutorials, learning guides, computer aided assessment (CAA) and Model Patients. RESULTS: There are now over 150 IT based learning resources in our curriculum and course evaluation has showed that these have been well received by students. CONCLUSIONS: We conclude that with the wise use of the many skills and facilities usually available within an institution and by promoting collaborative projects with others, the production of high quality CAL is possible within most institutions.     

A needs-based study and examination skills course improves students' performance

BACKGROUND: Adult learning theory suggests that learning is most effective when related to need, when driven by the learner and when it is flexible. We describe the effect of an educational intervention that was driven by student need, and largely designed by students. METHODS: We undertook a needs assessment of fifth year medical students' study needs. Based on this, we helped them design a course to meet these needs. This was predominantly related to study skills and a practice objective structured clinical examination (OSCE). We evaluated the course by asking for student opinion and by measuring the effect on student performance in a high stakes medical school examination (written examination and OSCE). FINDINGS: Despite the course being run voluntarily and in after-hours sessions, 80-90% of the medical student class attended each session. Student performance on the end of year examinations was significantly enhanced in the year of the intervention, compared with previous years and with students from other schools sitting identical examinations in the same year. INTERPRETATION: Learning activities that are directly based on student needs, that focus on study and examination techniques, and that are largely student-driven, result in effective and valuable outcomes.     

Can student tutors act as examiners in an objective structured clinical examination?

Context  The dissemination of objective structured clinical examinations (OSCEs) is hampered by requirements for high levels of staffing and a significantly higher workload compared with multiple-choice examinations. Senior medical students may be able to support faculty staff to assess their peers. The aim of this study is to assess the reliability of student tutors as OSCE examiners and their acceptance by their peers.
Methods  Using a checklist and a global rating, teaching doctors (TDs) and student tutors (STs) simultaneously assessed students in basic clinical skills at 4 OSCE stations. The inter-rater agreement between TDs and STs was calculated by kappa values and paired t -tests. Students then completed a questionnaire to assess their acceptance of student peer examiners.
Results  All 214 Year 3 students at the University of Göttingen Medical School were evaluated in spring 2005. Student tutors gave slightly better average grades than TDs (differences of 0.02–0.20 on a 5-point Likert scale). Inter-rater agreement at the stations ranged from 0.41 to 0·64 for checklist assessment and global ratings; overall inter-rater agreement on the final grade was 0.66. Most students felt that assessment by STs would result in the same grades as assessment by TDs (64%) and that it would be similarly objective (69%). Nearly all students (95%) felt confident that they could evaluate their peers themselves in an OSCE.
Conclusions  On the basis of our results, STs can act as examiners in summative OSCEs to assess basic medical skills. The slightly better grades observed are of no practical concern. Students accepted assessment performed by STs.     

Patients teach students: partners in arthritis education

CONTEXT: A large metropolitan teaching hospital within The Northern Clinical School, University of Sydney. OBJECTIVE: To assess whether students taught by trained patients (Patient Partners) acquire the same levels of competence in musculoskeletal examination skills for arthritis as students taught by Consultant Rheumatologists. SUBJECTS: Year four medical students in a six-year Undergraduate Medical Programme. METHOD: Students randomized to eight tutorial groups were taught musculoskeletal examination skills in a 75-90 minute tutorial. Four groups were taught by Consultants with an untrained patient present and four groups were taught by Patient Partners. RESULTS: Students' mean self-ratings of skill before and after their tutorial were summed. For both groups, self-ratings before the tutorial were similar. After the tutorial both groups showed substantial gains in levels of skill. Patient Partners' ratings of students' taught by either Consultants or Partners were comparable. CONCLUSIONS: Patient Partners are at least equal to Consultant Rheumatologists in the teaching of musculoskeletal examination techniques for arthritis.     

Preliminary evaluation of "interpreter" role plays in teaching communication skills to medical undergraduates

RATIONALE AND OBJECTIVES: Multiculturalism presents linguistic obstacles to health care provision. We explored the early introduction of "interpreter" role-play exercises in teaching medical undergraduates communication skills. The interpreter role creates a natural barrier in communication providing an active prompt for recognizing learning needs in this area. METHODS: Bilingual Cantonese first-year medical students (n=160) were randomly allocated to either "Observer" or "Interpreter" role plays at a small-group introductory communication skills workshop using a quasi experimental design, counterbalanced across tutors. Students assessed their own skill competence before and, together with their perceptions of the different role plays' effectiveness, again after the workshop, using an anonymous 16 item Likert-type scale, analysed using ANOVA and MANOVA. RESULTS: Students' assessments of their skills improved significantly following the workshop (F=73.19 [1,156], P=0.0009). Students in the observer group reported greater changes in their scores following the workshop than did students in the interpreter group (F=4.84 [1,156], P=0.029), largely due to improvement in perceived skill (F=4.38 [1,156], P=0.038) rather than perceived programme effectiveness (F=3.13 [1,156], P > 0.05). Subsequent MANOVA indicated no main effect of observer/interpreter conditions, indicating these differences could be attributed to chance alone (F=1.41 [16 141], P > 0.05). CONCLUSION: The workshop positively influenced students' perceived communication skills, but the "Interpreter" role was less effective than the "Observer" role in achieving this. Future studies should examine whether interpreter role plays introduced later in the medical programme are beneficial.     

Health promotion projects: skill and attitude learning for medical students

OBJECTIVES: The aim of this health promotion project is to introduce students to appropriate skills and attitudes--as well as knowledge about health promotion strategies and methods. As part of this process, standardized procedures have been established to ensure that the projects are scientifically and ethically appropriate and adequately supervised. This project-centred course introduces the discipline of health promotion to third-year medical students at Monash University. It is aimed at introducing students to the range of health promotion concepts, providing them with experience of health promotion activities and involving them in consideration of the scientific, political and ethical issues arising from doctors' participation in health promotion. DESIGN: As the major learning and assessment component of the unit, students participate in self-selected project groups of three to five students. Each group develops a topic for a health promotion activity in the community, carries out that project and presents the results as a poster as well as a written report. SETTING: Monash University. SUBJECTS: Third-year medical students. RESULTS: Sixty per cent of each student's mark for the unit is based on the project. The posters produced by the project groups are placed on public display in a major teaching hospital for a week at the end of the unit. Public display of the posters helps each student to appreciate the variety of possible health promotion activities, and to appreciate health promotion as a scientific discipline. It also makes the project findings available to the public. CONCLUSIONS: Student evaluation of the project, and community response to the projects--especially the poster display-- indicate that the project is both a highly effective learning experience and a health-promoting activity in its own right.     

Comparison of didactic lecture, self-reading and self-instruction as learning methods in medical students of western India

A controlled study was conduced on a batch of fifty-four medical students in the paraclinical period of study, to compare the relative effectiveness of self-reading of books, didactic lecture and use of self-instruction kits as teaching-learning experiences. The assessment was done by an immediate and delayed objective type examination as well as questionnaire.
The results indicate self-reading to be relatively ineffective. Didactic lectures and use of self-instruction kits have a similar rating in objective assessment while, subjectively, the students expressed a marginal preference for self-instruction kits.
Comparison of the performance of the students grouped according to merit reveals that there is no significant difference between the three methods of learning in the lower group while the upper group did perceptibly better with self-instruction kits than with didactic lecture.     

Learning from lives: a model for health and social care education in the wider community context

OBJECTIVES: To develop and evaluate a model for medical education which draws upon the whole frontline workforce in primary care and which enables medical students to develop skills and competencies in patient-centred teamworking across organisational boundaries. METHODS: Over a period of 3 years, 517 undergraduate medical students undertook a 4-week community hospital-based attachment developed in partnership with frontline staff. Pre- and post-course questionnaires and qualitative evaluation were sought from students, patients, tutors and frontline staff. RESULTS: The performance of students in assessment was very good. Before receiving the assessment results, students perceived a high degree of achievement of the specified learning outcomes and reported significant changes in attitude. Qualitative comments were overwhelmingly positive, with clear indications that students appreciated the unique learning opportunities available in this model. Patients were very positive about continuing involvement and valued the opportunity to influence future doctors. Staff were enthusiastic and committed despite competing service pressures. CONCLUSIONS: We have successfully developed a partnership model of community-based education and shown that it leads to significant changes in attitude in students and enables them to learn in an active, patient-focused way about the complexities of delivery of care outside the secondary care environment. We have identified the key requirements for implementation of the model in other centres.     

Not knowing that they do not know: self-assessment accuracy of third-year medical students

PURPOSE: The development of self-regulated learning is a major focus of our problem-based learning (PBL) medical programme. Students who are unsuccessful in assessments often seem to lack insight into the standard of their own performance, yet the ability to self-assess accurately is essential for the effective self-management of learning. The aim of this project was to evaluate the accuracy of self- and peer-assessment according to academic performance. METHOD: In 2004, 175 3rd-year students undertook an integrated, case-based, short-essay, formative assessment. After the assessment they were provided with model answers and marking criteria. Students marked their own assessment paper and the paper of one of their peers. Assessment papers were subsequently marked by faculty members. The following data was available for each student: self-mark, faculty-mark, score awarded by a peer and the score that they awarded to their peer. Self-assessment and peer-assessment ability was compared to overall academic performance. RESULTS: Low-achieving students score themselves and their peers generously. High-achieving students score themselves more harshly than faculty. However, they score their peers accurately. CONCLUSION: In the 3rd year of the programme low-achieving students are unable to assess accurately the quality of their own work or the work of their peers in a formative written assessment. The PBL curriculum does not guarantee the appropriate development of self-assessment skills.     

Brief problem-solving questions in medical school examinations: is it necessary for students to explain their answers?

OBJECTIVE: To compare outcomes when answers to objective type problem-solving questions are marked with and without consideration of students' explanations of their answers. DESIGN: Students answered six multiple-part problem-solving questions on the final examination for a course in clinical biochemistry. Each part required a choice from a list, a number or a few words, plus an explanation or justification. Scores were determined independently for the answers alone and for the answers plus explanations. The outcomes of the two marking methods were compared. SETTING: Department of Pathology, University of Otago Medical School, Dunedin, New Zealand. SUBJECTS: Final year preclinical medical students. RESULTS: Marks were slightly higher for answers alone than for answers plus explanations (overall mean, 82.1%; standard deviation, 9.8% vs. 77.1%; 9.3%; P < 0.001 by paired t-test). There was a very high correlation between scores derived from the two methods (r=0. 87). Consideration of the answers alone (without explanations) was about as good as answers plus explanations in identifying students who performed at different levels overall on the examination and in identifying the weakest students. CONCLUSIONS: The data suggest that when students arrived at the correct answers, their information and reasons were usually correct and that answers alone discriminate adequately among students with different levels of knowledge and ability. By dispensing with the requirement for explanations, we would be able to ask more questions and mark objectively, so competent students should not be disadvantaged by the lack of opportunity to explain their reasoning.