Traditional finals and OSCEs in predicting consultant and self-reported clinical skills of PRHOs: a pilot study

INTRODUCTION: As we move from standard 'long case' final examinations to new objective structured formats, we need to ensure the new is at least as good as the old. Furthermore, knowledge of which examination format best predicts medical student progression and clinical skills development would be of value. METHODS: A group of medical students sat both the standard long case examination and the new objective structured clinical examination (OSCE) to introduce this latter examination to our Medical School for final MB. At the end of their pre-registration year, the group and their supervising consultants submitted performance evaluation questionnaires. RESULTS: Thirty medical students sat both examinations and 20 returned evaluation questionnaires. Of the 72 consultants approached, 60 (83%) returned completed questionnaires. No correlation existed between self- and consultant reported performance. The traditional finals examination was inversely associated with consultant assessment. Better performing students were not rated as better doctors. The OSCE (and its components) was more consistent and showed positive associations with consultant ratings across the board. DISCUSSION: Major discrepancies exist between the 2 examination formats, in data interpretation and practical skills, which are explicitly tested in OSCEs but less so in traditional finals. Standardised marking schemes may reduce examiner variability and discretion and weaken correlations across the 2 examinations. This pilot provides empirical evidence that OSCEs assess different clinical domains than do traditional finals. Additionally, OSCEs improve prediction of clinical performance as assessed by independent consultants. CONCLUSION: Traditional finals and OSCEs correlate poorly with one another. Objective structured clinical examinations appear to correlate well with consultant assessment at the end of the pre-registration house officer year.     

Teaching sexual history taking to medical students and examining it: experience in one medical school and a national survey

OBJECTIVES: To assess the feasibility and acceptability of training and examining medical students in taking a sexual history and to compare practice with other medical schools in the UK. DESIGN: A training programme involving group work, role play and clinical attachments was developed and applied to 131 students at the University of Bristol Medical School. They then underwent an objective structured clinical assessment using simulated patients. The practice of other medical schools was surveyed by postal questionnaire. RESULT: The students felt that the examination was a good test of their knowledge and skills. One student failed. Sexual history taking is taught in 17 of 22 medical schools but examined in only six. CONCLUSION: Both teaching and examining of sexual history taking skills are possible and are likely to occur increasingly in UK medical schools.     

Can student tutors act as examiners in an objective structured clinical examination?

Context  The dissemination of objective structured clinical examinations (OSCEs) is hampered by requirements for high levels of staffing and a significantly higher workload compared with multiple-choice examinations. Senior medical students may be able to support faculty staff to assess their peers. The aim of this study is to assess the reliability of student tutors as OSCE examiners and their acceptance by their peers.
Methods  Using a checklist and a global rating, teaching doctors (TDs) and student tutors (STs) simultaneously assessed students in basic clinical skills at 4 OSCE stations. The inter-rater agreement between TDs and STs was calculated by kappa values and paired t -tests. Students then completed a questionnaire to assess their acceptance of student peer examiners.
Results  All 214 Year 3 students at the University of Göttingen Medical School were evaluated in spring 2005. Student tutors gave slightly better average grades than TDs (differences of 0.02–0.20 on a 5-point Likert scale). Inter-rater agreement at the stations ranged from 0.41 to 0·64 for checklist assessment and global ratings; overall inter-rater agreement on the final grade was 0.66. Most students felt that assessment by STs would result in the same grades as assessment by TDs (64%) and that it would be similarly objective (69%). Nearly all students (95%) felt confident that they could evaluate their peers themselves in an OSCE.
Conclusions  On the basis of our results, STs can act as examiners in summative OSCEs to assess basic medical skills. The slightly better grades observed are of no practical concern. Students accepted assessment performed by STs.     

A study of pre-registration house officers' clinical skills

BACKGROUND: Little is known about the ability of pre-registration house officers (PRHOs) to perform basic clinical skills just prior to entering the medical register. OBJECTIVES: To find out whether PRHOs have deficiencies in basic clinical skills and to determine if the PRHOs themselves or their consultants are aware of them. METHOD: All 40 PRHOs at the Chelsea and Westminster and Whittington Hospitals were invited to undertake a 17 station OSCE of basic clinical skills. Each station was marked by one examiner completing an overall global score after completing an itemised checklist. An adequate station performance was the acquisition of a pass/borderline pass grade. Prior to the OSCE, a questionnaire was given to each PRHO asking them to rate their own abilities (on a 5-point scale) in the skills tested. A similar questionnaire was sent to the educational supervisors of each PRHO asking them to rate their house officer's ability in each of the same skills. RESULTS: Twenty-two PRHOs participated. Each PRHO failed to perform adequately a mean of 2.4 OSCE stations (SD 1.8, range 1-8). There were no significant correlations between OSCE performance and either self- or educational supervisor ratings. The supervisor felt unable to give an opinion on PRHO abilities in 18% of the skills assessed. DISCUSSION: This study suggests that PRHOs may have deficiencies in basic clinical skills at the time they enter the medical register. Neither the PRHOs themselves nor their consultants identified these deficiencies. A large regional study with sufficient power is required to explore the generalizability of these concerns in more detail.     

Patients' global ratings of student competence. Unreliable contamination or gold standard?

Purpose To determine whether global ratings by patients are valid and reliable enough to be used within a major summative assessment of medical students' clinical skills. Method In 11 stations of an 18‐station objective structured clinical examination (OSCE), where a student was asked to educate or take a history from a patient, the patient was asked, ‘How likely would you be to come back and discuss your concerns with this student again?’ These 11 opinions were aggregated into a single patient opinion mark and correlated with other measures of student competence. The patients were not experienced in student assessment. Results A total of 204 students undertook the OSCE. Reliability of patient opinion across all 11 stations revealed a Cronbach alpha of 0·65. The correlation coefficient between the patient ratings and the total OSCE score was good (r = 0·74; P < 0·001) and was better than the correlation between any single OSCE station and the total OSCE score. It was also better than the correlation between the aggregated patient opinion and tests of student knowledge (r = 0·47). Conclusion It is known that patients can reliably complete checklists of clinical skills and that doctors can reliably provide global ratings of students. We have now shown that, by controlling the context, asking the right question and aggregating several opinions, untrained patients can provide a reliable and valid global opinion that contributes to the assessment of a student's clinical skills.     

Teaching in the new NHS: the specialised ward based teacher

OBJECTIVES: To evaluate the effects of introducing specialised ward based teachers (WBTs) who had a broad remit to improve third year medical undergraduates' clinical experience. DESIGN: Quantitative and qualitative methods including interviews with WBTs, participating consultants and SIFT co-ordinator; student questionnaire and evaluations; analysis of Objective Structured Clinical Examination (OSCE) scores to ascertain if exposure to WBTs affected OSCE scores. SETTING: Two university teaching hospitals. PARTICIPANTS: Third year undergraduate medical students from one school of medicine; four WBTs; 25 consultants; SIFT co-ordinator. MAIN OUTCOME MEASURES: Student evaluations. Student questionnaires. Student OSCE scores. Interview data. RESULTS: WBTs had a demonstrable effect on student performance in OSCE examinations. 94% of students either agreed or strongly agreed that WBTs had helped them develop their examination skills and 87% either agreed or strongly agreed that WBTs had helped them develop their history taking skills. Interview data indicated that the consultants and SIFT co-ordinator considered that WBTs made an important contribution to clinical teaching. CONCLUSIONS: This study suggests that specialised WBTs are one way to manage clinical experience and enhance learning of undergraduate medical students. As clinical teaching moves into earlier parts of the undergraduate curriculum and into the community there is potential for this role to be developed.     

Factor analysis can be a useful standard setting tool in a high stakes OSCE assessment

BACKGROUND: No method of standard setting for objective structured clinical examinations (OSCEs) is perfect. Using scores aggregated across stations risks allowing students who are incompetent in some core skills to pass an examination, which may not be acceptable for high stakes assessments. AIM: To assess the feasibility of using a factor analysis of station scores in a high stakes OSCE to derive measures of underlying competencies. METHODS: A 12-station OSCE was administered to all 192 students in the penultimate undergraduate year at the University of Aberdeen Medical School. Analysis of the correlation table of station scores was used to exclude stations performing unreliably. Factor analysis of the remaining station scores was carried out to characterise the underlying competencies being assessed. Factor scores were used to derive pass/fail cut-off scores for the examination. RESULTS: Four stations were identified as having unpredicted variations in station scores. Analysis of the content of these stations allowed the underlying problems with the station designs to be isolated. Factor analysis of the remaining 8 stations revealed 3 main underlying factors, accounting for 53% of the total variance in scores. These were labelled "examination skills", "communication skills" and "history taking skills". CONCLUSION: Factor analysis is a useful tool for characterising and quantifying the skills that are assessed in an OSCE. Standard setting procedures can be used to calculate cut-off scores for each underlying factor.     

The use of an objective structured clinical examination (OSCE) for formative and summative assessment in a general practice clinical attachment and its relationship to final medical school examination performance

OBJECTIVES: To report the use of OSCEs for both formative and summative purposes within a general practice undergraduate clinical attachment and to compare student performance in the departmental OSCEs with that of their final medical school examinations. METHODS: Twenty-eight students rotated through the attachment and undertook pre- and post-attachment OSCEs of similar format but different content. Results were analysed to determine relationships between mean scores in the two OSCEs and student performance in their final medical school MBBS examinations. RESULTS: There was a marked improvement in all OSCE station scores. Pre-attachment scores for those stations measuring physical examination and problem-solving skills were unrelated to prior clinical experience. Post-attachment OSCE mean scores were significantly correlated with final examination OSCE and total mean scores. CONCLUSION: The general practice attachment appears to upgrade those clinical skills measured by the pre- and post-attachment OSCE, however, there was no control group of students. Problem-solving and focused physical examination skills need to be targeted by all undergraduate clinical departments. The department's post-attachment OSCE and total assessment results are predictors of final examination OSCE and total results. The use of pre- and post-attachment OSCEs facilitates both students' formative learning processes and the department's evaluation of its educational programme.     

A discourse analysis study of 'good' and 'poor' communication in an OSCE: a proposed new framework for teaching students

BACKGROUND: There is still a great deal to be learnt about teaching and assessing undergraduate communication skills, particularly as formal teaching in this area expands. One approach is to use the summative assessments of these skills in formative ways. Discourse analysis of data collected from final year examinations sheds light on the grounds for assessing students as 'good' or 'poor' communicators. This approach can feed into the teaching/learning of communication skills in the undergraduate curriculum. SETTING: A final year UK medical school objective structured clinical examination (OSCE). METHODS: Four scenarios, designed to assess communication skills in challenging contexts, were included in the OSCE. Video recordings of all interactions at these stations were screened. A sample covering a range of good, average and poor performances were transcribed and analysed. Discourse analysis methods were used to identify 'key components of communicative style'. FINDINGS: Analysis revealed important differences in communicative styles between candidates who scored highly and those who did poorly. These related to: empathetic versus 'retractive' styles of communicating; the importance of thematically staging a consultation, and the impact of values and assumptions on the outcome of a consultation. CONCLUSION: Detailed discourse analysis sheds light on patterns of communicative style and provides an analytic language for students to raise awareness of their own communication. This challenges standard approaches to teaching communication and shows the value of using summative assessments in formative ways.     

Unintentional failure to assess for experience in senior undergraduate OSCE scoring

Context One goal of undergraduate assessment is to test students’ (future) performance. In the area of skills testing, the objective structured clinical examination (OSCE) has been of great value as a tool with which to test a number of skills in a limited time, with bias reduction and improved reliability. But can OSCEs measure undergraduate internship expertise in basic clinical skills? Methods Undergraduate students (n = 32) were given a questionnaire listing 182 basic clinical skills. We asked them to score the number of times they had performed each skill during their internships (a 12‐month period in Year 6). We assessed the students at the end of Year 5 (before the start of their internships) and again at the start of Year 7 (undergraduate training takes 7 years in Belgium, with internships during Year 6), using a 14‐station OSCE assessing basic clinical skills. Global ratings were used to score performance. The relationship between internship experience and the OSCE Year 7 score was analysed using a linear regression model, controlling for variation in OSCE scores from Year 5. A multi‐level analysis was performed considering students as level‐1 units and stations as level‐2 units. Results Year 7 OSCE scores (post‐internships) were not affected by the number of times that students practised basic medical skills during their internships. Discussion Scores on OSCEs do not seem to reflect clinical expertise acquired during internships. Other more integrated assessment methods may prove to be more valid for testing final undergraduate skills levels.     

Effectiveness of medical school admissions criteria in predicting residency ranking four years later

BACKGROUND: Medical schools across Canada expend great effort in selecting students from a large pool of qualified applicants. Non-cognitive assessments are conducted by most schools in an effort to ensure that medical students have the personal characteristics of importance in the practice of Medicine. We reviewed the ability of University of Toronto academic and non-academic admission assessments to predict ranking by Internal Medicine and Family Medicine residency programmes. METHODS: The study sample consisted of students who had entered the University of Toronto between 1994 and 1998 inclusive, and had then applied through the Canadian resident matching programme to positions in Family or Internal Medicine at the University of Toronto in their graduating year. The value of admissions variables in predicting medical school performance and residency ranking was assessed. RESULTS: Ranking in Internal Medicine correlated significantly with undergraduate grade point average (GPA) and the admissions non-cognitive assessment. It also correlated with 2-year objective structured clinical examination (OSCE) score, clerkship grade in Internal Medicine, and final grade in medical school. Ranking in Family Medicine correlated with the admissions interview score. It also correlated with 2nd-year OSCE score, clerkship grade in Family Medicine, clerkship ward evaluation in Internal Medicine and final grade in medical school. DISCUSSION: The results of this study suggest that cognitive as well as non-cognitive factors evaluated during medical school admission are important in predicting future success in Medicine. The non-cognitive assessment provides additional value to standard academic criteria in predicting ranking by 2 residency programmes, and justifies its use as part of the admissions process.     

The development, validity and reliability of a multimodality objective structured clinical examination in psychiatry

OBJECTIVES: To evaluate the development, validity and reliability of a multimodality objective structured clinical examination (OSCE) in undergraduate psychiatry, integrating interactive face-to-face and telephone history taking and communication skills stations, videotape mental state examinations and problem-oriented written stations. METHODS: The development of the OSCE on a restricted budget is described. This study evaluates the validity and reliability of 4 15-18-station OSCEs for 128 students over 1 year. Face and content validity were assessed by a panel of clinicians and from feedback from OSCE participants. Correlations with consultant clinical 'firm grades' were performed. Interrater reliability and internal consistency (interstation reliability) were assessed using generalisability theory. RESULTS: The OSCE was feasible to conduct and had a high level of high perceived face and content validity. Consultant firm grades correlated moderately with scores on interactive stations and poorly with written and video stations. Overall reliability was moderate to good, with G-coefficients in the range 0.55-0.68 for the 4 OSCEs. CONCLUSIONS: Integrating a range of modalities into an OSCE in psychiatry appears to represent a feasible, generally valid and reliable method of examination on a restricted budget. Different types of stations appear to have different advantages and disadvantages, supporting the integration of both interactive and written components into the OSCE format.     

The effect of an extended hospital induction on perceived confidence and assessed clinical skills of newly qualified pre-registration house officers

BACKGROUND: Recent studies raise concerns over the preparedness of newly qualified doctors for the role of the pre-registration house officer (PRHO). This study aimed to assess self-perception of preparedness, objective assessment of core clinical skills and the effect of an extended clinical induction programme prior to commencing full duties. METHODS: A group of 26 newly qualified doctors from 1 district general hospital underwent an extended 5-day, ward-based induction programme. The participants completed questionnaires on their own perceptions of their preparedness for PRHO duties and underwent an objective structured clinical examination (OSCE) of 4 core clinical skills prior to induction, on completion of induction and 1 month into working life. RESULTS: At the outset PRHOs had low perceptions of their own capabilities in all clinical scenarios and skills. Most perceptions improved after induction, although in 2 clinical areas they felt even less confident. One month into post there were significant improvements in all areas. Only 1 PRHO passed all 4 clinical skills assessments at the pre-induction assessment. Seven (26%) failed on 1 or more skills at the post-induction assessment. However, all participants were deemed competent in all skills at the 1-month assessment. CONCLUSION: Newly qualified doctors do not feel prepared for PRHO duties and objectively are not competent in basic clinical skills. An extended induction improves preparedness in some but not all clinical areas and improves performance of objectively assessed clinical skills.     

One year's experience with a clinical skills resource centre

OBJECTIVES: To describe our experience of teaching clinical skills to first-year medical students in a new problem-based curriculum. DESIGN: Prospective evaluation of clinical skills acquisition using objective measures. SETTING: Students were taught in a purpose-designed clinical skills resource centre in weekly structured sessions. SUBJECTS: All 210 first year medical undergraduates in the first year of a new problem-based and clinically oriented course. RESULTS: Student performance in structured objective examinations improved over 1 year. In all but one of nine matched stations, involving history, examination and procedural skills, improvements were significant. Ninety-three per cent thought that the appropriateness of the skills training course to other course elements was 'good' or 'very good'. CONCLUSIONS: First-year medical students can learn a wide range of clinical skills. Integrated teaching of clinical skills improves satisfaction with undergraduate studies.     

Benefiting from clinical experience: the influence of learning style and clinical experience on performance in an undergraduate objective structured clinical examination

OBJECTIVE: To assess the relationship between clinical experience, learning style and performance in an objective structured clinical examination (OSCE) in medical students at the end of their first clinical year. DESIGN: Prospective study of undergraduate students taking an OSCE examination at the end of their first clinical year. SUBJECTS: 194 undergraduate medical students (95 male). MAIN OUTCOME MEASURES: Performance in the OSCE examination, the Entwhistle Learning Style Inventory1 and a composite self-reported score of clinical activity during the students first clinical year. RESULTS: Performance in the OSCE examination was related to well-organized study methods but not to clinical experience. A significant relationship between clinical experience and organized deep-learning styles suggests that knowledge gained from clinical experience is related to learning style. CONCLUSIONS: The relationship between clinical experience and student performance is complex. Well-organized and strategic learning styles appear to influence the benefits of increased clinical exposure. Further work is required to elucidate the most beneficial aspects of clinical teaching.     

Structured packs for independent learning: a comparison of learning outcome and acceptability with conventional teaching

OBJECTIVES: The GMC recommends that students become independent learners, while tutor time is an increasingly precious resource. A set of structured learning materials requiring students to undertake and reflect on practical tasks in five learning areas was developed. DESIGN: The study used a randomized control trial to evaluate the effectiveness of using these structured learning materials in place of conventional teaching for 228 third-year undergraduate students and 55 teachers, on both hospital and community based medical and general practice firms. Evaluation involved assessing student performance on an examination question and a writing task, together with a student and tutor satisfaction questionnaire. SETTING: King's College School of Medicine and Dentistry, London. SUBJECTS: Third-year medical (first-year clinical) undergraduates. RESULTS: No significant difference in learning outcome was found for students on community- and hospital-based medical and general practice firms between students who had used the structured materials and those who had had conventional teaching on the same topic. The packs were acceptable to tutors and students. CONCLUSIONS: Such resources represent a mid-point between formal didactic teaching and self-directed learning. They may be particularly suitable for promoting independent learning for students on traditional medical courses. They offer an appropriate way to cover certain topics in the clinical curriculum and help to protect tutor time for topics which cannot be effectively taught in other ways.     

A needs-based study and examination skills course improves students' performance

BACKGROUND: Adult learning theory suggests that learning is most effective when related to need, when driven by the learner and when it is flexible. We describe the effect of an educational intervention that was driven by student need, and largely designed by students. METHODS: We undertook a needs assessment of fifth year medical students' study needs. Based on this, we helped them design a course to meet these needs. This was predominantly related to study skills and a practice objective structured clinical examination (OSCE). We evaluated the course by asking for student opinion and by measuring the effect on student performance in a high stakes medical school examination (written examination and OSCE). FINDINGS: Despite the course being run voluntarily and in after-hours sessions, 80-90% of the medical student class attended each session. Student performance on the end of year examinations was significantly enhanced in the year of the intervention, compared with previous years and with students from other schools sitting identical examinations in the same year. INTERPRETATION: Learning activities that are directly based on student needs, that focus on study and examination techniques, and that are largely student-driven, result in effective and valuable outcomes.     

Assessment of emergency medicine: a comparison of an experimental objective structured clinical examintion with a practical examination

An emergency medicine and trauma programme was implemented at Ben Gurion University Medical School in Israel. Clinical performance assessment of the first-year course in emergency medicine and trauma was done using a practical examination (PE). In the continuous process of critically reviewing the course objectives and assessment methods the objective structured clinical examination (OSCE) was chosen, for the first time in our medical school, as a tool for course development as well as evaluation of the existing PE. Seven experimental OSCE stations were designed which covered some of the course and practical examination topics. Twenty-six first-year medical students have taken both examinations concurrently. Twenty-three students answered an attitude questionnaire regarding both examinations. Results have indicated that the OSCE provided additional and crucial information on students' deficiencies in clinical performance which were not available from the PE. Those differences were probably due to realistic OSCE station content, highly simulated set-ups, and the objectivity of the examiner, all of which have contributed to a more challenging examination, as compared to the PE. The OSCE in emergency medicine and trauma seemed to have a relatively high level of acceptance by both staff and students. In our opinion it seems that the OSCE is a better tool for first-year level final assessment in emergency medicine and trauma. is a better tool for first-year level final assessment     

Identifying core skills for the medical curriculum

A survey was undertaken at the University of Sheffield Medical School to identify clinical and other professional skills which are required on graduation and to assess whether and when these skills have been acquired. The purpose of the survey was to identify core practical skills, defined as skills which the majority of clinical firms require and which, in addition, were used by the majority of Pre-registration house officers (PRHOs). The survey was conducted by questionnaires which were circulated to the following four sample groups (number in sample: % response rate): (1) consultants in clinical firms taking undergraduates and post-graduates from Sheffield Medical School (63: 75%); (2) house officers starting their pre-registration year (118: 52%); (3) house officers ending their pre-registration year (120: 43%); and (4) undergraduates starting their final year (110: 65%). The questionnaire contained a stimulus list of 31 clinical and four personal skills with an opportunity for respondents to list other skills they felt to be important. Of the 35 skills listed on the questionnaire, 26 were identified as core practical skills. Fifteen of the core skills had been acquired by the majority of students by the time they graduated, 13 of which were acquired prior to the final undergraduate year. Core skills not acquired as an undergraduate were acquired as a PRHO. The survey acted as a quality control mechanism for teaching at both undergraduate and post graduate levels and provided the basis of a medical school core of practical skills on the basic medical education continuum.