Assessing performance in final year medical students. Can a postgraduate measure be used in an undergraduate setting?

BACKGROUND: The transition between medical school and graduate performance should be a continuum. This study aimed to evaluate an assessment tool developed for practising doctors when applied to undergraduates. METHODS: A 12-item rating form was developed from that used for practising doctors by the Royal Australasian College of Physicians. Over a 2-year period, senior doctors, junior doctors and nurses completed the rating form on final year medical students. Some students completed self-assessments. We performed factor analysis and correlated scores between raters and attachments. Correlating ratings with concurrent traditional assessment results across the year tested construct validity. RESULTS: Ten forms per student were distributed for all 123 students and 856 were returned (70%). Internal consistency was very high. In all, 71.1% of the variance was accounted for by two factors (clinical skills and humanistic). This factor structure is unchanged when restricted to different raters and is the same as that noted previously when rating practising doctors. There were good correlations between raters (including self) and between attachments. Nurse ratings were reliable but nurses rated students significantly lower on humanistic qualities. Correlations with traditional assessments were high when all traditional assessments were combined. Women scored more favourably than men on humanistic qualities. CONCLUSION: A rating instrument for doctors in practice retains the same factor structure and a high degree of reliability and validity for senior medical students. Reliable ratings by nurses have implications for measures of collegiality and teamwork. We believe the instrument could be a useful outcome measure for medical programmes and employers.     

Developing professional clinical skills for practice – the results of a feasibility study using a reflective approach to intimate examination

Objectives  To develop a student-selected component (SSC) for junior medical students, to evaluate the feasibility of incorporating the development of skills in carrying out an intimate examination, whilst developing professional thinking skills using a reflective approach.
Subjects  The student selected component was designed for a maximum of 6 students over a 4-week period in Phase 2 (year 2 and 3) of the undergraduate medical curriculum.
Setting  The Clinical Skills Centre, the Faculty of Medicine Nursing and Dentistry, University of Dundee, Scotland, UK.
Methods  The SSC consisted of a weekly patient clerking from the ward, an analysis of the clinical and communication skills for any intimate examinations the patient would require, and practice using simulators and simulated patients. Students were supported by timed periods of structured reflection, which enabled them to discuss ethical issues and their own professional conduct related to carrying out an intimate examination. The SSC was evaluated on 3 levels; student satisfaction, learning professional thinking using a reflective approach, and observing behaviour change in relation to skill development, using a ward simulation exercise.
Results  The evaluation at the levels of student satisfaction, learning professional thinking and changes in behaviour associated with intimate examination demonstrated that the SSC had been well received by the students, who felt they had improved their skills in intimate examinations in the context of a ward simulation exercise, in parallel with their professional thinking skills.
Conclusion  It is possible to develop the professional thinking of junior medical students at the same time as developing their technical competence in relation to intimate examinations. It may be advantageous to introduce students early to this reflective approach, using simulation to promote the integration of theory with practice.     

Low- to high-fidelity simulation – a continuum of medical education?

Context  Changes in medical training and culture have reduced the acceptability of the traditional apprenticeship style training in medicine and influenced the growth of clinical skills training. Simulation is an educational technique that allows interactive, and at times immersive, activity by recreating all or part of a clinical experience without exposing patients to the associated risks. The number and range of commercially available technologies used in simulation for education of health care professionals is growing exponentially. These range from simple part-task training models to highly sophisticated computer driven models.
Aim  This paper will review the range of currently available simulators and the educational processes that underpin simulation training. The use of different levels of simulation in a continuum of training will be discussed. Although simulation is relatively new to medicine, simulators have been used extensively for training and assessment in many other domains, most notably the aviation industry. Some parallels and differences will be highlighted.     

An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation

PURPOSE: This investigation aimed to explore the measurement properties of scores from a patient simulator exercise. METHODS: Analytic and holistic scores were obtained for groups of medical students and residents. Item analysis techniques were used to explore the nature of specific examinee actions. Interrater reliability was calculated. Scores were contrasted for third year medical students, fourth year medical students and emergency department residents. RESULTS: Interrater reliabilities for analytic and holistic scores were 0.92 and 0.81, respectively. Based on item analysis, proper timing and sequencing of actions discriminated between low- and high-ability examinees. In general, examinees with more advanced training obtained higher scores on the simulation exercise. CONCLUSION: Reliable and valid measures of clinical performance can be obtained from a trauma simulation provided that care is taken in the development and scoring of the scenario.     

Assessment of patient management skills and clinical skills of practising doctors using computer-based case simulations and standardised patients

CONTEXT: Standardised assessments of practising doctors are receiving growing support, but theoretical and logistical issues pose serious obstacles. OBJECTIVES: To obtain reference performance levels from experienced doctors on computer-based case simulation (CCS) and standardised patient-based (SP) methods, and to evaluate the utility of these methods in diagnostic assessment. SETTING AND PARTICIPANTS: The study was carried out at a military tertiary care facility and involved 54 residents and credentialed staff from the emergency medicine, general surgery and internal medicine departments. MAIN OUTCOME MEASURES: Doctors completed 8 CCS and 8 SP cases targeted at doctors entering the profession. Standardised patient performances were compared to archived Year 4 medical student data. RESULTS: While staff doctors and residents performed well on both CCS and SP cases, a wide range of scores was exhibited on all cases. There were no significant differences between the scores of participants from differing specialties or of varying experience. Among participants who completed both CCS and SP testing (n = 44), a moderate positive correlation between CCS and SP checklist scores was observed. There was a negative correlation between doctor experience and SP checklist scores. Whereas the time students spent with SPs varied little with clinical task, doctors appeared to spend more time on communication/counselling cases than on cases involving acute/chronic medical problems. CONCLUSION: Computer-based case simulations and standardised patient-based assessments may be useful as part of a multimodal programme to evaluate practising doctors. Additional study is needed on SP standard setting and scoring methods. Establishing empirical likelihoods for a range of performances on assessments of this character should receive priority.     

Predictive validity of a written knowledge test of skills for an OSCE in postgraduate training for general practice

PURPOSE: To examine the validity of a written knowledge test of skills for performance on an OSCE in postgraduate training for general practice. METHODS: A randomly-selected sample of 47 trainees in general practice took a knowledge test of skills, a general knowledge test and an OSCE. The OSCE included technical stations and stations including complete patient encounters. Each station was checklist rated and global rated. RESULTS: The knowledge test of skills was better correlated to the OSCE than the general knowledge test. Technical stations were better correlated to the knowledge test of skills than stations including complete patient encounters. For the technical stations the rating system had no influence on the correlation. For the stations including complete patient encounters the checklist rating correlated better to the knowledge test of skills than the global rating. CONCLUSION: The results of this study support the predictive validity of the knowledge test of skills. In postgraduate training for general practice a written knowledge test of skills can be used as an instrument to estimate the level of clinical skills, especially for group evaluation, such as in studies examining the efficacy of a training programme or as a screening instrument for deciding about courses to be offered. This estimation is more accurate when the content of the test matches the skills under study. However, written testing of skills cannot replace direct observation of performance of skills.     

The interactive examination: assessing students' self-assessment ability

BACKGROUND: The ability to self-assess one's competence is a crucial skill for all health professionals. The interactive examination is an assessment model aiming to evaluate not only students' clinical skills and competence, but also their ability to self-assess their proficiency. METHODS: The methodology utilised students' own self-assessment, an answer to a written essay question and a group discussion. Students' self-assessment was matched to the judgement of their instructors. As a final task, students compared their own essay to one written by an "expert". The differences pointed by students in their comparison documents and the accompanying arguments were analysed and categorised. Students received individual feedback on their performance and learning needs. The model was tested on 1 cohort of undergraduate dental students (year 2001, n = 52) in their third semester of studies, replacing an older form of examination in the discipline of clinical periodontology. RESULTS: Students' acceptance of the methodology was very positive. Students tended to overestimate their competence in relation to the judgement of their instructors in diagnostic skills, but not in skills relevant to treatment. No gender differences were observed, although females performed better than males in the examination. Three categories of differences were observed in the students' comparison documents. The accompanying arguments may reveal students' understanding and methods of prioritising. CONCLUSIONS: Students tended to overestimate their competence in diagnostic rather than treatment skills. The interactive examination appeared to be a convenient tool for providing deeper insight into students' ability to prioritise, self-assess and steer their own learning.     

Standardized patients: a long-station clinical examination format

Summary. Data from the first 20 periods of a long-station clinical performance examination for a 4-week required clerkship in family medicine were examined in order to assess the reliability and validity of the examination. Data from 304 students were examined for station, case scenario and examiner effects and results compared to short-station formats. A significant examiner effect was found but there were no differences in student performance for station or case scenario. These findings reflect examiner specificity cited in the literature for short station examinations, but not case specificity. The source of variability for this examination appears to be primarily examiner effect. There was a significant correlation between student scores on the two cases, and raters tended to rank order students similarly in spite of variability in mean rater score. Scores on the CPE correlated with other measures of clinical performance as well as other methods of student evaluation for the clerkship providing some evidence for construct and criterion-related validity. CPE cases were developed from clerkship objectives but examination of the test blueprint revealed some gaps in the extent to which the CPE covers the course content. CPE developers are working to increase interrater reliability through examiner training and further standardize case scenarios through checklists and patient training. Additional cases are being developed to increase the content validity of the examination.     

Portfolio use in general practice vocational training: a survey of GP registrars

BACKGROUND: Portfolios are increasingly advocated in medical education. Qualitative studies have suggested their value in stimulating experiential learning, promoting deep learning and encouraging reflection. This study explored the use of educational portfolios in reflective learning by general practice (GP) registrars in Yorkshire, England. METHOD: A postal questionnaire was sent to the 92 registrars of a deanery in November 2001, after a pilot study with registrars in a single district had been carried out. The questionnaire explored the registrars' use of the portfolio to collect information and for reflection, as well as registrars' attitudes towards the portfolio. It was returned by 71 registrars, representing a 77% response rate. Structured in-depth interviews were used to support the results obtained. RESULTS: Of the registrars who responded, 65% recorded information on a regular basis and 42% used the portfolio in reflective learning. Experienced registrars used the portfolio least. Those with supportive trainers used the portfolio more in reflection. CONCLUSIONS: The study suggests that the role of the trainer/supervisor is vital in portfolio-based learning. It raises questions about the acceptability of portfolio learning. It adds weight to the suggestion that careful introduction of portfolios and training of supervisors is vital. Further work to establish the role of portfolios in reflective learning is recommended.     

The development, validity and reliability of a multimodality objective structured clinical examination in psychiatry

OBJECTIVES: To evaluate the development, validity and reliability of a multimodality objective structured clinical examination (OSCE) in undergraduate psychiatry, integrating interactive face-to-face and telephone history taking and communication skills stations, videotape mental state examinations and problem-oriented written stations. METHODS: The development of the OSCE on a restricted budget is described. This study evaluates the validity and reliability of 4 15-18-station OSCEs for 128 students over 1 year. Face and content validity were assessed by a panel of clinicians and from feedback from OSCE participants. Correlations with consultant clinical 'firm grades' were performed. Interrater reliability and internal consistency (interstation reliability) were assessed using generalisability theory. RESULTS: The OSCE was feasible to conduct and had a high level of high perceived face and content validity. Consultant firm grades correlated moderately with scores on interactive stations and poorly with written and video stations. Overall reliability was moderate to good, with G-coefficients in the range 0.55-0.68 for the 4 OSCEs. CONCLUSIONS: Integrating a range of modalities into an OSCE in psychiatry appears to represent a feasible, generally valid and reliable method of examination on a restricted budget. Different types of stations appear to have different advantages and disadvantages, supporting the integration of both interactive and written components into the OSCE format.     

Learning the skills of flexible sigmoidoscopy – the wider perspective

Background  Nurse-led gastrointestinal endoscopy is a priority clinical area in the UK. Endoscopic procedures are challenging to learn, requiring a combination of technical competence (manipulating a flexible endoscope and interpreting the findings) and interpersonal skills (engaging effectively with a conscious patient who is frequently apprehensive).
This paper explores the potential of an innovative, scenario-based approach which links a simulated patient with a computer-driven virtual reality (VR) training device for flexible sigmoidoscopy. Within this safe yet realistic quasi-clinical environment, learners carry out the procedure while interacting with the 'patient'. Communication skills are assessed by simulated patients, while quantitative performance data relating to the procedure is generated automatically by the VR simulator.
Methods  This pilot study took place within a nurse practitioner endoscopy course. A mixed methodology combined qualitative and quantitative data (observation and interview studies, communication rating scales and a range of computer-generated output measures from the VR simulator) in a multifaceted evaluation.
Results  Seven nurses took part in the study. Participants found the scenarios to be a convincing and powerful learning experience. All experienced high levels of anxiety. Simulated patients identified strengths in participants' communication skills, together with areas for development. Simulator-based practice led to an improvement in objective performance measures.
Discussion  Scenario-based training provides a powerful learning experience, allowing participants to build their technical expertise and apply it within a holistic clinical context without the risk of causing harm.
We used this pilot study as a springboard for discussions over wider implications of procedure-based skills training, locating it within the literature on expertise and situated learning.     

Setting a standard for performance assessment of doctor-patient communication in general practice

CONTEXT: Continuing professional development (CPD) of general practitioners. OBJECTIVE: Criterion-referenced standards for assessing performance in the real practice of general practitioners (GPs) should be available to identify learning needs or poor performers for CPD. The applicability of common standard setting procedures in authentic assessment has not been investigated. METHODS: To set a standard for assessment of GP-patient communication with video observation of daily practice, we investigated 2 well known examples of 2 different standard setting approaches. An Angoff procedure was applied to 8 written cases. A borderline regression method was applied to videotaped consultations of 88 GPs. The procedures and outcomes were evaluated by the applicability of the procedure, the reliability of the standards and the credibility as perceived by the stakeholders, namely, the GPs. RESULTS: Both methods are applicable and reliable; the obtained standards are credible according to the GPs. CONCLUSIONS: Both modified methods can be used to set a standard for assessment in daily practice. The context in which the standard will be used - i.e. the specific purpose of the standard, the moment the standard must be available or if specific feedback must be given - is important because methods differ in practical aspects.     

Reliability and validity of the direct observation clinical encounter examination (DOCEE)

CONTEXT: The College of Medicine and Medical Sciences at the Arabian Gulf University, Bahrain, replaced the traditional long case/short case clinical examination on the final MD examination with a direct observation clinical encounter examination (DOCEE). Each student encountered four real patients. Two pairs of examiners from different disciplines observed the students taking history and conducting physical examinations and jointly assessed their clinical competence. OBJECTIVES: To determine the reliability and validity of the DOCEE by investigating whether examiners agree when scoring, ranking and classifying students; to determine the number of cases and examiners necessary to produce a reliable examination, and to establish whether the examination has content and concurrent validity. SUBJECTS: Fifty-six final year medical students and 22 examiners (in pairs) participated in the DOCEE in 2001. METHODS: Generalisability theory, intraclass correlation, Pearson correlation and kappa were used to study reliability and agreement between the examiners. Case content and Pearson correlation between DOCEE and other examination components were used to study validity. RESULTS: Cronbach's alpha for DOCEE was 0.85. The intraclass and Pearson correlation of scores given by specialists and non-specialists ranged from 0.82 to 0.93. Kappa scores ranged from 0.56 to 1.00. The overall intraclass correlation of students' scores was 0.86. The generalisability coefficient with four cases and two raters was 0.84. Decision studies showed that increasing the cases from one to four improved reliability to above 0.8. However, increasing the number of raters had little impact on reliability. The use of a pre-examination blueprint for selecting the cases improved the content validity. The disattenuated Pearson correlations between DOCEE and other performance measures as a measure of concurrent validity ranged from 0.67 to 0.79. CONCLUSIONS: The DOCEE was shown to have good reliability and interrater agreement between two independent specialist and non-specialist examiners on the scoring, ranking and pass/fail classification of student performance. It has adequate content and concurrent validity and provides unique information about students' clinical competence.     

The long case

BACKGROUND: The long case has been gradually replaced by the objective structured clinical examination (OSCE) as a summative assessment of clinical skills. Its demise occurred against a paucity of psychometric research. This article reviews the current status of the long case, appraising its strengths and weaknesses as an assessment tool. ISSUES: There is a conflict between validity and reliability. The long case assesses an integrated clinical reaction between doctor and real patients and has high face validity. Intercase reliability is the prime problem. As most examinations traditionally used a single case only, problems of content specificity and standardisation were not addressed. DISCUSSION: Recent research suggests that testing across more cases does improve reliability. Better structuring of tests and direct observation increases validity. Substituting standardised cases for real patients may be of little benefit compared to increasing the sample of cases. CONCLUSIONS: Observed long cases can be useful for assessment depending on the sample size of cases and examiners. More research is needed into the exact nature of intercase and interexaminer variance and consequential validity. Feasibility remains a key problem. More exploration of combined assessments using real patients with OSCEs is suggested.     

Responding to the implications of the genetics revolution for the education and training of doctors: a medical humanities approach

BACKGROUND: Rapid advances in the field of genetics continue to present medical educators with significant challenges. Whilst there is undoubtedly a pressing need to educate doctors about genetic disorders, research and therapies, there is a parallel need to provide a context for all of these. CONTENT: An interdisciplinary, arts and humanities based approach, responding to this need, is described. This teaching has been successfully delivered both as optional and core undergraduate teaching, and as part of continuing professional development. THE HUMAN PERSPECTIVE: STORIES NOT HISTORIES: Understanding of the patient's perspective can be significantly improved by drawing on both written and oral stories of illness. THE HISTORICAL PERSPECTIVE: LEARNING FROM THE PATIENT: Experiential learning provides insights into the social history of developments in genetics, thereby placing the current concern and debate about the new genetics in context. THE ROLE OF THE MEDIA: THE POWER TO PERSUADE: Critical reading skills can be developed and the power of the popular press to influence the reader acknowledged by analysing and employing the skills of the journalist when reporting developments in biotechnology. LEARNER ASSESSMENT AND EVALUATION: Assessment, both formative and summative, demonstrates sophisticated insights and perspectives into the lived experience of genetic illness. Learner evaluation of the teaching is high. CONCLUSION: Medical humanities offers a powerful way to convey an understanding of how genetic disorders impact on the lives of patients and families, and to set this against the background of a history rich in the uses, and abuses, of knowledge of heredity.     

Validity threats: overcoming interference with proposed interpretations of assessment data

CONTEXT: Factors that interfere with the ability to interpret assessment scores or ratings in the proposed manner threaten validity. To be interpreted in a meaningful manner, all assessments in medical education require sound, scientific evidence of validity. PURPOSE: The purpose of this essay is to discuss 2 major threats to validity: construct under-representation (CU) and construct-irrelevant variance (CIV). Examples of each type of threat for written, performance and clinical performance examinations are provided. DISCUSSION: The CU threat to validity refers to undersampling the content domain. Using too few items, cases or clinical performance observations to adequately generalise to the domain represents CU. Variables that systematically (rather than randomly) interfere with the ability to meaningfully interpret scores or ratings represent CIV. Issues such as flawed test items written at inappropriate reading levels or statistically biased questions represent CIV in written tests. For performance examinations, such as standardised patient examinations, flawed cases or cases that are too difficult for student ability contribute CIV to the assessment. For clinical performance data, systematic rater error, such as halo or central tendency error, represents CIV. The term face validity is rejected as representative of any type of legitimate validity evidence, although the fact that the appearance of the assessment may be an important characteristic other than validity is acknowledged. CONCLUSIONS: There are multiple threats to validity in all types of assessment in medical education. Methods to eliminate or control validity threats are suggested.     

Comparison of a rational and an empirical standard setting procedure for an OSCE. Objective structured clinical examinations

PURPOSE: Earlier studies of absolute standard setting procedures for objective structured clinical examinations (OSCEs) show inconsistent results. This study compared a rational and an empirical standard setting procedure. Reliability and credibility were examined first. The impact of a reality check was then established. METHODS: The OSCE included 16 stations and was taken by trainees in their final year of postgraduate training in general practice and experienced general practitioners. A modified Angoff (independent judgements, no group discussion) with and without a reality check was used as a rational procedure. A method related to the borderline group procedure, the borderline regression (BR) method, was used as an empirical procedure. Reliability was assessed using generalisability theory. Credibility was assessed by comparing pass rates and by relating the passing scores to test difficulty. RESULTS: The passing scores were 73.4% for the Angoff procedure without reality check (Angoff I), 66.0% for the Angoff procedure with reality check (Angoff II) and 57.6% for the BR method. The reliabilities (expressed as root mean square errors) were 2.1% for Angoffs I and II, and 0.6% for the BR method. The pass rates of the trainees and GPs were 19% and 9% for Angoff I, 66% and 46% for Angoff II, and 95% and 80% for the BR method, respectively. The correlation between test difficulty and passing score was 0.69 for Angoff I, 0.88 for Angoff II and 0.86 for the BR method. CONCLUSION: The BR method provides a more credible and reliable standard for an OSCE than a modified Angoff procedure. A reality check improves the credibility of the Angoff procedure but does not improve its reliability.