The objective structured clinical examination in undergraduate psychiatry

Inadequate attention has been given to verifying the psychometric attributes of the objective structured clinical examination (OSCE), yet its popularity has been increasing in recent years. Our 6 years' experience in Nigeria showed that OSCE is practicable in undergraduate psychiatry assessment and there is evidence over consecutive years that it has satisfactory reliability and criterion-based validity. The importance of students' feedback in assessing the quality of examination is reinforced, and subtle, less tangible elements which determine students' performance, such as social interactional mystique and some personality traits, are worthy of evaluative research.     

An oral practical examination in emergency clinical surgery

A body of students were asked about their views regarding an oral practical examination in clinical surgery that had been in practice for over 12 years. This is an examination with the planned objective to test problem-solving abilities and professional attitudes in emergency surgery. Verbal emergency simulation aided by appropriate pathological specimens, surgical instruments and X-rays are put forward by the examiners, and the candidate is asked to respond. One hundred and ninety-four students returned their questionnaire out of a total of 196 students (99% response rate). Candidates who were satisfied with their answers in the examination numbered 127 (65%), while 67 candidates were dissatisfied (35%). Reasons for dissatisfaction were cited as difficult and complex emergency simulation questions, expressed by 44 candidates (23%), and anxiety generated by confrontation of the examiners, expressed by 23 candidates (12%). One hundred and thirteen students (58%) thought a change in the format of the examination was not needed, while 47 students (24%) suggested a change mostly towards standardization of the examination. Emergency clinical surgery is an important area of the surgical curriculum that needs to be assessed properly. A more standardized version of this oral practical examination could perhaps provide a useful tool of assessment.     

Scoring the objective structured clinical examination using a microcomputer

The objective structured clinical examination (OSCE) is being used increasingly to assess students' clinical competence in a variety of controlled settings. The OSCE consists of multiple stations composed of a variety of clinically relevant problems (e.g. examining simulated patients, diagnosing X-rays, etc.) Generally, three types of performance data are collected: answers to multiple choice or true/false questions, written short answers, and performance check-lists completed by observers. In most OSCEs these student performance measures are scored by hand. This is time-consuming, increases the probability of mistakes and reduces the amount of data available for analysis. This paper describes a method of computer scoring OSCEs with over 100 students using statistical and test-scoring software regularly used for multiple choice examinations. During the examination, students, markers and raters code answers and performance data directly on optical mark-sheets which are read into the computer using an optical mark reader. The resultant computer data can be efficiently scored and rescored, grouped into different types of subscales, weighted to reflect questions' relative importance, and easily printed in a variety of report formats.     

Psychometric characteristics of the objective structured clinical examination

The objective structured clinical examination (OSCE) is increasingly being used as a method of clinical assessment yet its measurement characteristics have not been well documented. Evidence is accumulating that many OSCEs may be too short to achieve reliable results. This paper reports detailed psychometric analyses of OSCEs which were administered as part of a well-established final-year examination. Generalizability theory guided investigation of test reliability. At the present test length the OSCE components showed low reliabilities relative to written components. Satisfactory reliabilities could potentially be achieved if test length was increased to approximately 6 hours, a time which would create significant logistic problems for most medical schools. Several strategies for dealing with this practical problem have been explored. Firstly, it was shown that more careful selection of stations based on their psychometric characteristics can significantly improve reliability. Secondly, where rater availability is a limiting factor to increasing test length, more can be gained by using one rater per station and having more stations than using two raters per station. Finally, OSCE scores can, with advantage, be combined with other test scores which are obtained by using less resource-intensive methods. By adopting such strategies, a reliable assessment of clinical competence could be obtained in about 4 hours of testing time which was equally divided between an OSCE constructed of practical and clinical stations and a written test.     

The direct clinical examination: an alternative method for the assessment of clinical psychiatry skills in undergraduate medical students

Summary: Because of dissatisfaction with the traditional long case procedure as a method of examining the clinical competence of medical students undertaking a psychiatry term, an alternative 'direct' method whereby two examiners observe the interaction between student and patient has been developed and is described. This method of examining allows the examiners to set and evaluate case-specific tasks. It is demonstrated that the two examiners achieve satisfactory inter-rater reliability both with respect to the mark awarded and the difficulty the patient presents and that, as one would wish, these two measures do not correlate. Students' opinions regarding the examination were assessed pre- and post-examination using visual analogue scales. The students found the examination stressful but rated the method as an appropriate form of clinical assessment both before and after their examination. The method is seen as having several advantages which must be set against the disadvantage of its being relatively expensive of examiners' time.     

Predictive validity and estimated cut score of an objective structured clinical examination (OSCE) used as an assessment of clinical skills at the end of the first clinical year

BACKGROUND: Assessment plays a key role in the learning process. The validity of any given assessment tool should ideally be established. If an assessment is to act as a guide to future teaching and learning then its predictive validity must be established. AIM: To assess the ability of an objective structured clinical examination (OSCE) taken at the end of the first clinical year of an undergraduate medical degree to predict later performance in clinical examinations. METHODS: Performance of two consecutive cohorts of year 3 medical undergraduates (n=138 and n=128) in a 23 station OSCE were compared with their performance in 5 subsequent clinical examinations in years 4 and 5 of the course. RESULTS: Poor performance in the OSCE was strongly associated with later poor performance in other clinical examinations. Students in the lowest three deciles of OSCE performance were 6 times more likely to fail another clinical examination. Receiver operating characteristic curves were constructed as a method to criterion reference the cut point for future examinations. CONCLUSION: Performance in an OSCE taken early in the clinical course strongly predicts later clinical performance. Assessing subsequent student performance is a powerful tool for assessing examination validity. The use of ROC curves represents a novel method for determining future criterion referenced examination cut points.     

Effects of basic clinical skills training on objective structured clinical examination performance

OBJECTIVES: The aim of curriculum reform in medical education is to improve students' clinical and communication skills. However, there are contradicting results regarding the effectiveness of such reforms. METHODS: A study of internal medicine students was carried out using a static group design. The experimental group consisted of 77 students participating in 7 sessions of communication training, 7 sessions of skills-laboratory training and 7 sessions of bedside-teaching, each lasting 1.5 hours. The control group of 66 students from the traditional curriculum participated in equally as many sessions but was offered only bedside teaching. Students' cognitive and practical skills performance was assessed using Multiple Choice Question (MCQ) testing and an objective structured clinical examination (OSCE), delivered by examiners blind to group membership. RESULTS: The experimental group performed significantly better on the OSCE than did the control group (P < 0.01), whereas the groups did not differ on the MCQ test (P < 0.15). This indicates that specific training in communication and basic clinical skills enabled students to perform better in an OSCE, whereas its effects on knowledge did not differ from those of the traditional curriculum. CONCLUSION: Curriculum reform promoting communication and basic clinical skills are effective and lead to an improved performance in history taking and physical examination skills.     

The effect on reliability of adding a separate written assessment component to an objective structured clinical examination

PURPOSE: To determine the effect on test reliability when a separate written assessment component is added to an objective structured clinical examination (OSCE). METHOD: Volunteers (n=38) from Maastricht Medical School were recruited to take a skills-related knowledge test in addition to their regular end-of-year OSCE. The OSCE scores of these volunteers did not differ from those of the other students of their class. Multivariate generalizability theory was used to investigate the combined reliability of the two test formats as well as their respective contributions to overall reliability. RESULTS: Combining the two formats has an added value. The loss of reliability due to the use of fewer stations in the OSCE can be fully compensated by lengthening the written test component. CONCLUSION: From the perspective of test reliability, it is possible to economize on the resources needed for performance-based assessment by adding a separate written test component.     

Eight years'' experience with a structured clinical examination

A structured clinical examination has been an integral component of the final-year examinations conducted by the Departments of Medicine and Surgery at the University of Adelaide for the last 8 years. It has been used as an alternative to the traditional clinical examination. This paper describes the results of ongoing student and examiner surveys carried out to determine their views and satisfaction with this new approach. It also briefly discusses the feasibility of introducing such an examination into a conventional medical school environment. The surveys showed a remarkable level of acceptance and support by both students and examiners. This positive reaction has been maintained over the 8-year period. The main reasons seem to be its perceived relevance and fairness. Students also appear to be directing their learning in a direction thought desirable by teachers. No significant problems have been encountered with the practical implementation of the method.     

Assessing professionalism in the context of an objective structured clinical examination: an in-depth study of the rating process

INTRODUCTION: Professionalism is fundamental to the practice of medicine. Objective structured clinical examinations (OSCEs) have been proposed as appropriate for assessing some aspects of professionalism. This study investigated how raters assign professionalism ratings to medical students' performances in OSCE encounters. METHODS: Three standardised patients, 3 doctor preceptors, and 3 lay people viewed and rated 20 videotaped encounters between 3rd-year medical students and standardised patients. Raters recorded their thoughts while rating. Qualitative and quantitative analyses were conducted. Comments about observable behaviours were coded, and relative frequencies were computed. Correlations between counts of categorised comments and overall professionalism ratings were also computed. RESULTS: Raters varied in which behaviours they attended to, and how behaviours were evaluated. This was true within and between rater type. Raters also differed in the behaviours they consider when providing global evaluations of professionalism. CONCLUSIONS: This study highlights the complexity of the processes involved in assigning ratings to doctor-patient encounters. Greater emphasis on behavioural definitions of specific behaviours may not be a sufficient solution, as raters appear to vary in both attention to and evaluation of behaviours. Reliance on global ratings is also problematic, especially if relatively few raters are used, for similar reasons. We propose a model highlighting the multiple points where raters viewing the same encounter may diverge, resulting in different ratings of the same performance. Progress in assessment of professionalism will require further dialogue about what constitutes professional behaviour in the medical encounter, with input from multiple constituencies and multiple representatives within each constituency.     

Development of a three-centre simultaneous objective structured clinical examination

OBJECTIVE: To describe the development, organization, implementation and evaluation of a yearly multicentre, identical and simultaneous objective structured clinical examination (OSCE). SUBJECTS: All fifth-year medical students in a 6-year undergraduate medical programme. SETTING: The Christchurch, Dunedin and Wellington Schools of Medicine of the University of Otago, New Zealand. METHOD: One practice and two full 18-station OSCEs have been completed over 2 years, for up to 72 students per centre, in three centres. The process of development and logistics is described. Data are presented on validity, reliability and fairness. RESULTS: Face and content validity were established. Internal consistency was 0.83-0. 86 and interexaminer reliability, as assessed by the coefficient of correlation, averaged 0.78. Students rated the OSCE highly on relevance. Of the total variance in total OSCE marks, the schools contributed 6.9%, and the students 93.1%, in the first year. In the second year the schools contributed 6.2% and the students 93.8%. CONCLUSION: Implementation of a psychometrically sound, multicentre, simultaneous and identical OSCE is possible with a low level of interschool variation.     

The assessment of an examination in pathology consisting of multiple-choice, practical and short essay questions

The rationale for assessing medical students' achievement in pathology with examinations consisting of multiple-choice, practical and short essay items is provided. In addition, these three-part examinations are assessed with reference to their reliabilities and their content, criterion and construct validities.     

The modified essay question: an evaluation of its use in a family medicine clerkship

This paper describes the extensive experience of the Department of Family Medicine at Jefferson Medical College in utilizing the modified essay question (MEQ) as a final examination for its required third-year clerkship in family medicine. The results of a preliminary study are presented, comparing the MEQ as a method of evaluation with the multiple choice question (MCQ) format on similar content material. MCQ performance was found to be a better predictor of National Board Part I and II scores, but neither format was found to be a good predictor of postgraduate performance. The data were statistically significant (P less than 0.05) that MEQ performance was related to postgraduate performance in the area of professional attitude while the MCQ was not; however, the number of students in the analysis was small. Additional studies are planned to test this hypothesis further.     

Reliability and validity of the objective structured clinical examination in paediatrics

The assessment of clinical competence has traditionally been carried out through standard evaluations such as multiple choice question and bedside oral examinations. The attributes which constitute clinical competence are multidimensional, and we have modified the objective structured clinical examination (OSCE) to measure these various competencies. We have evaluated the validity and reliability of the OSCE in a paediatric clinical clerkship. We divided the examination into the four components of competence (clinical skills, problem-solving, knowledge, and patient management) and evaluated the performance of 77 fourth-year medical students. The skill and content domains of the OSCE were carefully defined, agreed upon, sampled and reproduced. This qualitative evaluation of the examination was both adequate and appropriate. We achieved both acceptable interstation and intertask reliability. When correlated with concurrent methods of evaluation we found the OSCE to be an accurate measure of paediatric knowledge and patient management skills. The OSCE did not correlate, however, with traditional measures of clinical skills including history-taking and physical examination. Our OSCE, as outlined, offers an objective means of identifying weaknesses and strengths in specific areas of clinical competence and is therefore an important addition to the traditional tools of evaluation.     

Objective structured practical examination: a new concept in assessment of laboratory exercises in preclinical sciences

The objective structured practical examination (OSPE) was used as an objective instrument for assessment of laboratory exercises in preclinical sciences, particularly physiology. It was adapted from the objective structured clinical examination (OSCE). The OSPE was administered to two consecutive classes in conjunction with the conventional examination in which the candidate is expected to perform a given experiment. The scores of the students in the two components of the examination were used to compare the OSPE with the conventional examination and to evaluate the new instrument of assessment. The OSPE appears to be a reliable device with a good capacity for discriminating between different categories of students. It is better in these respects than the conventional practical examination. Moreover, it has scope for being structured in such a way that all the objectives of laboratory teaching can be tested and each aspect can be assigned the desired weightage.     

A comparative study of the traditional long case with the objective structured clinical examination in Lagos, Nigeria

An apparent difference in the results of the clinical examination of the final M.B., B.S. was observed following replacement of the traditional long case (TLC) with the Objective Structured Clinical Examination (OSCE) in 1979. This led to a study of the results of two consecutive years of each method, 1976 and 1978 (TLC), and 1979 and 1980 (OSCE). The OSCE pass rate of 61% was found to be significantly lower than TLC pass rate of 93% (P less than 0.05). Using the analysis of variance and the critical difference (CD) of the mean scores of the different types of examination, no significant difference was found to exist between the two TLC examinations or between the two OSCE examinations. However, significant difference exists between the TLC of 1978 and the OSCE of 1979, P less than 0.05. A comparison of the CD of MCQ to those of TLC and OSCE suggests that less differences exist between MCQ and OSCE scores compared to MCQ and TLC scores, and by 1980 no significant difference exists between MCQ and OSCE. OSCE, like MCQ, will therefore appear an acceptable method of examination and perhaps a more effective method of clinical examination than TLC.     

Benefiting from clinical experience: the influence of learning style and clinical experience on performance in an undergraduate objective structured clinical examination

OBJECTIVE: To assess the relationship between clinical experience, learning style and performance in an objective structured clinical examination (OSCE) in medical students at the end of their first clinical year. DESIGN: Prospective study of undergraduate students taking an OSCE examination at the end of their first clinical year. SUBJECTS: 194 undergraduate medical students (95 male). MAIN OUTCOME MEASURES: Performance in the OSCE examination, the Entwhistle Learning Style Inventory1 and a composite self-reported score of clinical activity during the students first clinical year. RESULTS: Performance in the OSCE examination was related to well-organized study methods but not to clinical experience. A significant relationship between clinical experience and organized deep-learning styles suggests that knowledge gained from clinical experience is related to learning style. CONCLUSIONS: The relationship between clinical experience and student performance is complex. Well-organized and strategic learning styles appear to influence the benefits of increased clinical exposure. Further work is required to elucidate the most beneficial aspects of clinical teaching.     

The attitudes of medical students to the objective structured practical examination

Summary. The objective structured practical examination (OSPE) together with the classical practical examination (CPE) form the basis for evaluation of laboratory teaching at the All India Institute of Medical Sciences, New Delhi. Students' attitudes to OSPE and CPE were assessed by preparing a questionnaire containing 32 item statements on Likert's 5-point scale (LS) and 11 bipolar adjectives on Osgood's 7-point Semantic Differential Scale (OSDS). The questionnaire was administered to 50 medical undergraduates before their final examination. Forty-two (84%) students responded to the questionnaire. The LS showed high internal consistency and validity. Attitude scoring on LS and OSDS revealed a high degree of correlation. The majority of students showed a positive attitude to OSPE, and high-rank students had a greater intensity of positive attitude. OSPE was rated as a reliable, effective, useful, interesting and challenging examination, although considered taxing, mentally and physically. The students preferred a combination of OSPE and CPE as in the present system to a complete change-over to the OSPE. The item analysis of the questionnaire is discussed.     

Reliability and learning from the objective structured clinical examination

The difficulties in measurement of the clinical performance of students in the health professions are well known by educators. One innovative measure incorporated in several of the educational programmes, including the BSc in Nursing programme, in the Faculty of Health Sciences, at McMaster University, Hamilton, Ontario, Canada is the objective structured clinical examination (OSCE). The purpose of this study was to determine the reliability of this evaluation method, both within and between stations. One problem that has been noted by users of the OSCE method is that performance on individual OSCE stations is poorly correlated across stations, apparently regardless of the particular content of the station. A number of hypotheses have been advanced to attempt to explain this phenomenon: performance of any skill is sufficiently variable that the correlation is poor; different skills have little common basis, so that there is no generalizability from one to another, or reliability of assessment in any one station is low. To test these hypotheses, a study was designed for test-retest and interrater reliability. Students undergoing a 10-station OSCE also repeated their starting OSCE station at the end of the examination circuit. In addition, several stations were rated by more than one observer (interrater). This study of 71 first-year BScN students showed that the interrater reliability was high (ICC = 0.80 to 0.99), and test-retest reliability on the same station was good (ICC = 0.66 to 0.86); however, correlation across stations was low (alpha = 0.198). Thus it is apparent that there is high consistency of repeated performance of a skill but little consistency of performance on different skills.     

The development, validity and reliability of a multimodality objective structured clinical examination in psychiatry

OBJECTIVES: To evaluate the development, validity and reliability of a multimodality objective structured clinical examination (OSCE) in undergraduate psychiatry, integrating interactive face-to-face and telephone history taking and communication skills stations, videotape mental state examinations and problem-oriented written stations. METHODS: The development of the OSCE on a restricted budget is described. This study evaluates the validity and reliability of 4 15-18-station OSCEs for 128 students over 1 year. Face and content validity were assessed by a panel of clinicians and from feedback from OSCE participants. Correlations with consultant clinical 'firm grades' were performed. Interrater reliability and internal consistency (interstation reliability) were assessed using generalisability theory. RESULTS: The OSCE was feasible to conduct and had a high level of high perceived face and content validity. Consultant firm grades correlated moderately with scores on interactive stations and poorly with written and video stations. Overall reliability was moderate to good, with G-coefficients in the range 0.55-0.68 for the 4 OSCEs. CONCLUSIONS: Integrating a range of modalities into an OSCE in psychiatry appears to represent a feasible, generally valid and reliable method of examination on a restricted budget. Different types of stations appear to have different advantages and disadvantages, supporting the integration of both interactive and written components into the OSCE format.