The effectiveness of immediate feedback during the Objective Structured Clinical Examination

Using eight different physical examination or technical stations, 400 examinations were conducted to evaluate the effectiveness of immediate feedback during the Objective Structured Clinical Examination (OSCE). The test group comprised 50 medical students who underwent a standard 4-minute examination followed by 2 minutes of feedback. Immediately following feedback the students repeated an identical 4-minute examination scored by the same examiners. The control group consisted of 50 students from the same class who underwent an identical testing sequence, but instead of receiving feedback, they were instructed to continue their examinations for an additional 2 minutes before repeating the stations. Simple repetition of the task did not significantly improve score (mean increase 2.0%, NS). Extending the testing period from 4 to 6 minutes resulted in a small but significant increase in score (mean 6.7%, P less than 0.001). However, there was a much larger increase in the scores obtained following 2 minutes of immediate feedback compared to pre-feedback performance (mean 26.3%, P less than 0.0001). The majority of students and examiners felt that feedback, as administered in this study, was valuable both as a learning and teaching experience. Short periods of immediate feedback during an OSCE are practical and can improve competency in the performance of criterion-based tasks, at least over the short term. In addition, such feedback provides students with valuable self-assessment that may stimulate further learning.     

The consistency of students'' self-assessments in short-essay subject matter examinations

This longitudinal study compares the accuracy of self-assessments of 22 students across four examinations during their first 2 years of medical school. The four examinations used a similar short-essay format and covered many of the same basic science disciplines at similar levels of difficulty. Immediately after answering an average of 20 questions on each examination, students predicted their performance on those questions. After assigned subject matter experts had scored the questions, the differences between students' predictions and the experts' scores were calculated for each question. The degree to which students had over- and underestimated their performance across all questions was determined by separately averaging all positive and negative differences between students' and experts' assessments on each examination. The results of the study indicated that accuracy in self-assessment improved from examination 1 to examination 3 (with less overestimation) and dropped on examination 4 (with more underestimation). The results revealed no relationship between self-assessment estimations and actual scores received. Furthermore, the self-assessment estimations tended to be statistically correlated between contiguous examinations (i.e., examinations 1 and 2, 2 and 3, etc.) but not between non-contiguous ones (i.e., examinations 1 and 3, etc.). The results of the study are interpreted to suggest that the students in the study have a self-assessment tendency towards over- or underestimation that is somewhat stable but that gradually evolves over time with experience, maturity and self-assessment practice. The most frequent direction of change is towards decreased overestimation and increased underestimation. These results are consistent with the findings of other recent longitudinal self-assessment studies.     

Competency domains in an undergraduate Objective Structured Clinical Examination: their impact on compensatory standard setting

Context  Following a 15-week attachment in paediatrics and child health, general practice and dermatology medical students in their second clinical year at this medical school undertake a high-stakes assessment including an objective structured clinical examination (OSCE). There were 2 hypotheses. Firstly, groups of similar stations map to competency domains identifiable by factor analysis. Secondly, poor performance in individual domains is compensated for by achieving the required standard of performance across the whole assessment.
Methods  A total of 647 medical students were assessed by an OSCE during 5 individual examination sittings (diets) over 2 years. Ten scoring stations in the OSCE were analysed and confirmatory factor analysis performed comparing a 1-factor model (where all the stations are discrete entities related to one underlying domain) with a 3-factor model (where the stations load onto 3 domains from a previously reported exploratory factor analysis).
Results  The 3-factor model yielded a significantly better fit to the data (χ² = 15.3, P  <   0.01). Assessing the compensation data of 1 diet, 29 of 127 students failed in 1 or more domains described, whereas only 5 failed if compensation was allowed across all domains.
Discussion  Confirmatory factor analysis showed a significant fit of the data to previously described competency domains for a high-stakes undergraduate OSCE. Compensation within but not between competency domains would provide a more robust standard, improve validity, and substantially reduce the pass rate.     

A multi-format Interdisciplinary Final Examination

Limitations of the traditional final medical examination for the assessment of clinical competence led to such developments as simulated patients and the Objective Structured Clinical Examination (OSCE). An interdisciplinary OSCE incorporating simulated patients and involving nine disciplines was introduced into the final examination in the Auckland School of Medicine to supplement the written papers and the long case. Six-hundred and eight students were assessed over a 6-year period. Each of the three examination modes provided good discriminatory power. Significant correlations were found between the tests, but this does not mean one or more is redundant. Principal component analysis showed that a single significant factor accounted for over half the variance in the final assessment. This factor was equally weighted to the three examinations. A variety of evaluative methods are necessary to assess a student's competence and greater emphasis should be placed on those methods which encourage the learning of clinical skills and concurrently provide an appropriate mechanism for assessing them. The changes introduced have been supported by students and teachers and have fostered the learning of important clinical skills. Efforts to standardize the single long case have not overcome the criticisms surrounding its use, particularly in summative assessment.     

Standard setting in an objective structured clinical examination: use of global ratings of borderline performance to determine the passing score

BACKGROUND: Objective structured clinical examination (OSCE) standard-setting procedures are not well developed and are often time-consuming and complex. We report an evaluation of a simple 'contrasting groups' method, applied to an OSCE conducted simultaneously in three separate schools. SUBJECTS: Medical students undertaking an end-of-fifth year multidisciplinary OSCE. METHODS: Using structured marking sheets, pairs of examiners independently scored student performance at each OSCE station. Examiners also provided a global rating of overall performance. The actual scores of any borderline candidates at each station were averaged to provide a passing score for each station. The passing scores for all stations were combined to become the passing score for the whole exam. Validity was determined by making comparisons with performance on other fifth-year assessments. Reliability measures comprised interschool agreement, interexaminer agreement and interstation variability. RESULTS: The approach was simple and had face validity. There was a stronger association between the performance of borderline candidates on the OSCE and their in-course assessments than with their performance on the written exam, giving a weak measure of construct validity in the absence of a better 'gold standard'. There was good agreement between examiners in identifying borderline candidates. There were significant differences between schools in the borderline score for some stations, which disappeared when more than three stations were aggregated. CONCLUSION: This practical method provided a valid and reliable competence-based pass mark. Combining marks from all stations before determining the pass mark was more reliable than making decisions based on individual stations.     

The Objective Structured Video Exam for assessment of communication skills

OBJECTIVES: (i) To design a new, quick and efficient method of assessing specific cognitive aspects of trainee clinical communication skills, to be known as the Objective Structured Video Exam (OSVE) (Study 1); (ii) to prepare a scoring scheme for markers (Study 2); and (iii) to determine reliability and evidence for validity of the OSVE (Study 3). METHODS: Study 1 describes how the exam was designed. The OSVE assesses the student's recognition and understanding of the consequences of various communication skills. In addition, the assessment taps the number of alternative skills that the student believes will be of assistance in improving the patient-doctor interaction. Study 2 outlines the scoring system that is based on a range of 50 marks. Study 3 reports inter-rater consistency and presents evidence to support the validity of the new assessment by associating the marks from 607 1st year undergraduate medical students with their performance ratings in a communication skills OSCE. SETTING: Medical school, The University of Liverpool. RESULTS: Preparation of a scoring scheme for the OSVE produced consistent marking. The reliability of the marking scheme was high (ICC=0.94). Evidence for the construct validity of the OSVE was found when a moderate predicted relationship of the OSVE to interviewing behaviour in the communication skills OSCE was shown (r=0.17, P < 0.001). CONCLUSION: A new video-based written examination (the OSVE) that is efficient and quick to administer was shown to be reliable and to demonstrate some evidence for validity.     

The challenge of creating new OSCE measures to capture the characteristics of expertise

PURPOSE: Although expert clinicians approach interviewing in a different manner than novices, OSCE measures have not traditionally been designed to take into account levels of expertise. Creating better OSCE measures requires an understanding of how the interviewing style of experts differs objectively from novices. METHODS: Fourteen clinical clerks, 14 family practice residents and 14 family physicians were videotaped during 2 15-minute standardized patient interviews. Videotapes were reviewed and every utterance coded by type including questions, empathic comments, giving information, summary statements and articulated transitions. Utterances were plotted over time and examined for characteristic patterns related to level of expertise. RESULTS: The mean number of utterances exceeded one every 10 s for all groups. The largest proportion was questions, ranging from 76% of utterances for clerks to 67% for experts. One third of total utterances consisted of a group of 'low frequency' types, including empathic comments, information giving and summary statements. The topic was changed often by all groups. While utterance type over time appeared to show characteristic patterns reflective of expertise, the differences were not robust. Only the pattern of use of summary statements was statistically different between groups (P < 0.05). CONCLUSIONS: Measures that are sensitive to the nature of expertise, including the sequence and organisation of questions, should be used to supplement OSCE checklists that simply count questions. Specifically, information giving, empathic comments and summary statements that occupy a third of expert interviews should be credited. However, while there appear to be patterns of utterances that characterise levels of expertise, in this study these patterns were subtle and not amenable to counting and classification.     

The objective structured clinical examination in undergraduate psychiatry

Inadequate attention has been given to verifying the psychometric attributes of the objective structured clinical examination (OSCE), yet its popularity has been increasing in recent years. Our 6 years' experience in Nigeria showed that OSCE is practicable in undergraduate psychiatry assessment and there is evidence over consecutive years that it has satisfactory reliability and criterion-based validity. The importance of students' feedback in assessing the quality of examination is reinforced, and subtle, less tangible elements which determine students' performance, such as social interactional mystique and some personality traits, are worthy of evaluative research.     

Portfolios as a learning tool in obstetrics and gynaecology undergraduate training

CONTEXT: We developed a structured portfolio for medical students to use during their obstetrics and gynaecology undergraduate training. The main objective was to support the learning process of the students. We also wanted feedback information to enhance teaching. METHODS: The study population consisted of 91 medical students who completed the portfolio during their training course. The portfolio consisted of a 28-page A5-size booklet. The students entered all the clinical procedures they had performed and all the deliveries they had attended. After each group session, they answered questions about what they had learned and evaluated the performance of the teacher. They also indicated their general evaluation of the course and the portfolio itself. The teachers listed the 13 most important skills to be learned during the course. The students were asked to evaluate their own development on a scale of 0-5 before and after the course. A content analysis was performed on all the texts the students produced, and all quantitative variables were coded. RESULTS: The amount of text written in the portfolio correlated (P < 0.001, F-value 4.2) with success in the final exam. In addition to acting as a logbook, use of the portfolio enhanced the learning process during the course. Students' attitudes towards the portfolio were mainly positive. Students appreciated the departmental interest in their learning process. CONCLUSION: Portfolios support the personal and professional development of medical students. A portfolio clarifies the learning goals and helps students to monitor how these goals are achieved. A portfolio encourages constant self-reflection.     

The effect of a Structured Question Grid on the validity and perceived fairness of a medical long case assessment

PROBLEM: A perception that the reliability of our oral assessments of clinical competence was vitiated by lack of consistency in questioning. DESIGN: Parallel group controlled trial of a Structured Question Grid for use in clinical assessments. The Structured Question Grid required assessors to see the patient personally in advance of the student and to write down for each case the points they wished to examine. The Structured Question Grid limited assessors to two questions on each point, one designated a pass question and one at a higher level. Three basic science and three clinical reasoning issues were required, so that a total of 12 questions was allowed. SETTING: Small (70 students/year) undergraduate medical school with an integrated, problem-based curriculum. SUBJECTS: Sixty-seven students in the fourth year of a 5-year course were assessed, each seeing one patient and being examined by a pair of assessors. Assessor pairs were allocated to use the Structured Question Grid or to assess according to their usual practice. RESULTS: After the assessment but before being informed of the result the students completed a questionnaire on their experience and gave their performance a score between 0 and 100. The questions asked were based on focus group discussions with a previous student cohort, and concerned principally the perceived fairness and subjective validity of the assessment. The assessors independently completed a similar questionnaire, gave the student's performance a score between 0 and 100, and assigned an overall pass/fail grade. CONCLUSIONS: No difference was detected between students' or assessors' views of the fairness of the assessment for assessors who had used the Structured Question Grid compared to those who had not. Students whose assessors used the Structured Question Grid considered the assessment less representative of their ability. No difference was detected in the chance of students being assessed as failing or on the likelihood of a discrepancy between students' and assessors' ratings of students as passing or failing.     

The effect on reliability of adding a separate written assessment component to an objective structured clinical examination

PURPOSE: To determine the effect on test reliability when a separate written assessment component is added to an objective structured clinical examination (OSCE). METHOD: Volunteers (n=38) from Maastricht Medical School were recruited to take a skills-related knowledge test in addition to their regular end-of-year OSCE. The OSCE scores of these volunteers did not differ from those of the other students of their class. Multivariate generalizability theory was used to investigate the combined reliability of the two test formats as well as their respective contributions to overall reliability. RESULTS: Combining the two formats has an added value. The loss of reliability due to the use of fewer stations in the OSCE can be fully compensated by lengthening the written test component. CONCLUSION: From the perspective of test reliability, it is possible to economize on the resources needed for performance-based assessment by adding a separate written test component.     

Providing feedback to students on clinical skills by using the Objective Structured Clinical Examination

Student opinions of some methods of how and when to provide feedback on clinical skills in the Objective Structured Clinical Examination (OSCE) were gathered. Feedback with their marked checklists plus a model videorecording was preferred to either on its own. Feedback was preferred during the examination rather than after it. No undue stress appeared to be associated with receiving feedback during the examination. Valuable extra instruction to students can be provided with minimal staff input.     

The impact of curricular change on medical students' knowledge of anatomy

BACKGROUND: In recent years, following the publication of Tomorrow's Doctors, the undergraduate medical curriculum in most UK medical schools has undergone major revision. This has resulted in a significant reduction in the time allocated to the teaching of the basic medical sciences, including anatomy. However, it is not clear what impact these changes have had on medical students' knowledge of surface anatomy. AIM: This study aimed to assess the impact of these curricular changes on medical students' knowledge of surface anatomy. SETTING: Medical student intakes for 1995-98 at the Queen's University of Belfast, UK. METHODS: The students were invited to complete a simple examination paper testing their knowledge of surface anatomy. Results from the student intake of 1995, which undertook a traditional, 'old' curriculum, were compared with those from the student intakes of 1996-98, which undertook a new, 'systems-based' curriculum. To enhance linear response and enable the use of linear models for analysis, all data were adjusted using probit transformations of the proportion (percentage) of correct answers for each item and each year group. RESULTS: The student intake of 1995 (old curriculum) were more likely to score higher than the students who undertook the new, systems-based curriculum. CONCLUSION: The introduction of the new, systems-based course has had a negative impact on medical students' knowledge of surface anatomy.     

Formative assessment of the consultation performance of medical students in the setting of general practice using a modified version of the leicester assessment package

OBJECTIVE: To evaluate the use of a modified version of the Leicester Assessment Package (LAP) in the formative assessment of the consultation performance of medical students with particular reference to validity, inter-assessor reliability, acceptability, feasibility and educational impact. DESIGN: 180 third and fourth year Leicester medical students were directly observed consulting with six general practice patients and independently assessed by a pair of assessors. A total of 70 practice and 16 departmental assessors took part. Performance scores were subjected to generalizability analysis and students' views of the assessment were gathered by questionnaire. RESULTS: Four of the five categories of consultation performance (Interviewing and history taking, Patient management, Problem solving and Behaviour and relationship with patients) were assessed in over 99% of consultations and Physical examination was assessed in 94%. Seventy-six percent of assessors reported that the case mix was 'satisfactory' and 20% that it was 'borderline'; 85% of students believed it to have been satisfactory. Generalizability analysis indicates that two independent assessors assessing the performance of students across six consultations would achieve a reliability of 0.94 in making pass or fail decisions. Ninety-eight percent of students perceived that their particular strengths and weaknesses were correctly identified, 99% that they were given specific advice on how to improve their performance and 98% believed that the feedback they had received would have long-term benefit. CONCLUSIONS: The modified version of the LAP is valid, reliable and feasible in formative assessment of the consultation performance of medical students. Furthermore, almost all students found the process fair and believed it was likely to lead to improvements in their consultation performance. This approach may also be applicable to regulatory assessment as it accurately identifies students at the pass/fail margin.     

Predictive validity and estimated cut score of an objective structured clinical examination (OSCE) used as an assessment of clinical skills at the end of the first clinical year

BACKGROUND: Assessment plays a key role in the learning process. The validity of any given assessment tool should ideally be established. If an assessment is to act as a guide to future teaching and learning then its predictive validity must be established. AIM: To assess the ability of an objective structured clinical examination (OSCE) taken at the end of the first clinical year of an undergraduate medical degree to predict later performance in clinical examinations. METHODS: Performance of two consecutive cohorts of year 3 medical undergraduates (n=138 and n=128) in a 23 station OSCE were compared with their performance in 5 subsequent clinical examinations in years 4 and 5 of the course. RESULTS: Poor performance in the OSCE was strongly associated with later poor performance in other clinical examinations. Students in the lowest three deciles of OSCE performance were 6 times more likely to fail another clinical examination. Receiver operating characteristic curves were constructed as a method to criterion reference the cut point for future examinations. CONCLUSION: Performance in an OSCE taken early in the clinical course strongly predicts later clinical performance. Assessing subsequent student performance is a powerful tool for assessing examination validity. The use of ROC curves represents a novel method for determining future criterion referenced examination cut points.     

An evaluation of the Team Objective Structured Clinical Examination (TOSCE)

OBJECTIVES: To evaluate the validity, reliability and feasibility of the TOSCE, a new means of formative assessment for medical students, from the perspectives of examiners, simulated patients and students. DESIGN: Teams of five students rotate through five clinical stations, performing one of four tasks in turn, whilst the fifth member of the team 'rests'. SETTING: St George's Hospital Medical School, London. SUBJECTS: Third-year medical students, their examiners and simulated patients. RESULTS: All participants felt the TOSCE session had high validity although some students were unfamiliar with some subjects. Stations were double-marked and agreement rates and Cohen's Kappa ranged from 67 to 94% and 0.04-0.88, respectively. Analysis of the causes of disagreement led to improved marking schedules and enhanced reliability. CONCLUSIONS: The feasibility of the exercise as a formative assessment was demonstrated by the confidence of the staff participating both as examiners and simulated patients and from the majority of students who welcomed the learning experience.     

A conference on ethics for obstetric and gynaecological clerkship students

Integrating the teaching of medical ethics into medical students' clinical education is challenging, given the competing demands on students' time and the need for teaching to be clinically relevant. This paper describes a model programme for incorporating ethics teaching into the obstetrics and gynaecology clerkship for third-year medical students. The programme is taught by two attending teachers and a medical ethicist with experience teaching in the clinical setting of obstetrics and gynaecology. Objective pretests and posttests showed substantial improvement in students' knowledge, and student feedback has been very positive.     

Clinical skills assessment with standardized patients

Previous projects (Combell I & II) to assess clinical skills were conducted in medical schools in Catalonia, in order to introduce a model of such an assessment using standardized patients (SP). The aim of this study (Combell III) was to measure selected characteristics of our model. Seventy-three medical students in the final year at the Bellvitge teaching unit of the University of Barcelona participated in a clinical skills assessment (CSA) project that used 10 SP cases. The mean group scores for the four components of clinical skills for each day of testing were studied, and ratings for each student in the 10 sequential encounters were checked. The study also compared the clinical skills scores with their academic grades. The total case mean score (mean score of history-taking, physical examination and patient notes scores) was 51.9%, and the mean score for communication skills was 63.6%. The clinical skills scores over the 8 testing days showed no day-to-day differences. The study did not find differences among the sequential encounters for each student (training effect). There was a lack of correlation between clinical skill scores and academic grades. The project demonstrated the feasibility of the method for assessing clinical skills, confirmed its reliability, and showed that there is no correlation between scores with this method and academic examinations that mainly reflect knowledge.