The effect of in-training assessment on clinical confidence in postgraduate education

INTRODUCTION: The literature on how in-training assessment (ITA) works in practice and what educational outcomes can actually be achieved is limited. One of the aims of introducing ITA is to increase trainees' clinical confidence; this relies on the assumption that assessment drives learning through its content, format and programming. The aim of this study was to investigate the effect of introducing a structured ITA programme on junior doctors' clinical confidence. The programme was aimed at first year trainees in anaesthesiology. METHODS: The study involved a nationwide survey of junior doctors' self-confidence in clinical performance before (in 2001) and 2 years after (in 2003) the introduction of an ITA programme. Respondents indicated confidence on a 155-item questionnaire related to performance of clinical skills and tasks reflecting broad aspects of competence. A total of 23 of these items related to the ITA programme. RESULTS: The response rate was 377/531 (71%) in 2001 and 344/521 (66%) in 2003. There were no statistically significant differences in mean levels of confidence before and 2 years after the introduction of the ITA programme - neither in aspects that were related to the programme nor in those that were unrelated to the programme. DISCUSSION: This study demonstrates that the introduction of a structured ITA programme did not have any significant effect on trainees' mean level of confidence on a broad range of aspects of clinical competence. The importance of timeliness and rigorousness in the application of ITA is discussed.     

Setting a standard for performance assessment of doctor-patient communication in general practice

CONTEXT: Continuing professional development (CPD) of general practitioners. OBJECTIVE: Criterion-referenced standards for assessing performance in the real practice of general practitioners (GPs) should be available to identify learning needs or poor performers for CPD. The applicability of common standard setting procedures in authentic assessment has not been investigated. METHODS: To set a standard for assessment of GP-patient communication with video observation of daily practice, we investigated 2 well known examples of 2 different standard setting approaches. An Angoff procedure was applied to 8 written cases. A borderline regression method was applied to videotaped consultations of 88 GPs. The procedures and outcomes were evaluated by the applicability of the procedure, the reliability of the standards and the credibility as perceived by the stakeholders, namely, the GPs. RESULTS: Both methods are applicable and reliable; the obtained standards are credible according to the GPs. CONCLUSIONS: Both modified methods can be used to set a standard for assessment in daily practice. The context in which the standard will be used - i.e. the specific purpose of the standard, the moment the standard must be available or if specific feedback must be given - is important because methods differ in practical aspects.     

Computer‐aided learning in the real world of medical education: does the quality of interaction with the computer affect student learning?

BACKGROUND: Computerised learning clearly offers exciting potential for improving student learning, either as an aid to or as a replacement for traditional formats, or for the development of innovative approaches. However, rigorous evaluation of the utility of computer-aided learning (CAL) in enhancing student learning can be difficult. Many studies have compared CAL to more traditional learning formats, but there is little evidence to show which style of CAL leads to the best learning outcomes. AIM: This study aimed to test the hypothesis that a CAL tutorial, in which the learner actively interacts with the computer, will result in superior learning (ability to apply and retain knowledge) to that obtained in more passive CAL formats. METHODS: Third year medical undergraduates at Adelaide University, South Australia were randomly assigned to 4 groups. Following a pretest, only students in the "didactic", "problem-based" and "free text" groups had 2 weeks of free access to a neuroradiology CAL tutorial in their assigned format. Tutorial access was denied to all students 2 weeks before post-testing. Learning was quantified by comparing the post- to pretest scores for each of the 4 groups. RESULTS: After active interaction with the computer material, students in the free text group demonstrated a statistically significant improvement in their ability to apply and retain knowledge compared to the control group, but no advantage compared to the didactic group. CONCLUSIONS: While users of an interactive CAL tutorial demonstrated significant learning gains compared to non-CAL users, these gains were not superior to those achieved from non-interactive CAL. When evaluating education interventions such as CAL packages, it is important to use a valid assessment tool to measure learning.     

Portfolios for assessment of paediatric specialist registrars

INTRODUCTION: In 1997 the Royal College of Paediatrics and Child Health introduced portfolios to guide and monitor the learning of specialist registrars. We studied their value for assessment. METHODS: Using Bigg's SOLO criteria we devised a marking scheme based on 6 domains of competence: clinical, communication, teaching and learning, ethics and attitudes, management and evaluation, and creation of evidence. We rated portfolios according to quality of evidence presented and expectations by year of training. We similarly assessed trainee performance in the annual record of in-training assessment (RITA) interview. Specific advice based on the results of the first portfolio assessments was circulated to all trainees, instructing them to increase the structure and decrease the bulk of portfolios. A second sample of portfolios was reviewed a year later, using similar evaluations, to determine the effects. RESULTS: A total of 76 portfolios were assessed in year 1 by a single rater; 30 portfolios were assessed in year 2 by 2 independent raters. The quality of documentation improved from year 1 to year 2 but there was no significant increase in portfolio scores. The inter-rater correlation coefficient of the portfolio assessment method was 0.52 (Cohen's kappa 0.35). The inter-rater correlation coefficient of the RITA interview was 0.71 (Cohen's kappa 0.38). There was moderate inter-assessment correlation between portfolios and RITA interviews (kappa 0.26 in year 1 and 0.29 in year 2). Generalisability analysis suggested that 5 successive ratings by a single observer or independent ratings by 4 observers on the same occasion would be needed to yield a generalisability coefficient > 0.8 for overall portfolio rating. CONCLUSIONS: This method of portfolio assessment is insufficiently reliable as a sole method for high stakes, single-instance assessment, but has a place as part of a triangulation process. Repeated portfolio assessment by paired observers would increase reliability. Longer term studies are required to establish whether portfolio assessment positively influences learner behaviour.     

Assessment of patient management skills and clinical skills of practising doctors using computer-based case simulations and standardised patients

CONTEXT: Standardised assessments of practising doctors are receiving growing support, but theoretical and logistical issues pose serious obstacles. OBJECTIVES: To obtain reference performance levels from experienced doctors on computer-based case simulation (CCS) and standardised patient-based (SP) methods, and to evaluate the utility of these methods in diagnostic assessment. SETTING AND PARTICIPANTS: The study was carried out at a military tertiary care facility and involved 54 residents and credentialed staff from the emergency medicine, general surgery and internal medicine departments. MAIN OUTCOME MEASURES: Doctors completed 8 CCS and 8 SP cases targeted at doctors entering the profession. Standardised patient performances were compared to archived Year 4 medical student data. RESULTS: While staff doctors and residents performed well on both CCS and SP cases, a wide range of scores was exhibited on all cases. There were no significant differences between the scores of participants from differing specialties or of varying experience. Among participants who completed both CCS and SP testing (n = 44), a moderate positive correlation between CCS and SP checklist scores was observed. There was a negative correlation between doctor experience and SP checklist scores. Whereas the time students spent with SPs varied little with clinical task, doctors appeared to spend more time on communication/counselling cases than on cases involving acute/chronic medical problems. CONCLUSION: Computer-based case simulations and standardised patient-based assessments may be useful as part of a multimodal programme to evaluate practising doctors. Additional study is needed on SP standard setting and scoring methods. Establishing empirical likelihoods for a range of performances on assessments of this character should receive priority.     

Towards evidence-based practice in medical training: making evaluations more meaningful

CONTEXT: The evaluation of training is problematic and the evidence base inconclusive. This situation may arise for 2 main reasons: training is not understood as a complex intervention and, related to this, the evaluation methods applied are often overly simplistic. METHOD: This paper makes the case for construing training, especially in the field of specialist medical education, as a complex intervention. It also selectively reviews the available literature in order to match evaluative techniques with the demonstrated complexity. CONCLUSIONS: Construing training as a complex intervention can provide a framework for selecting the most appropriate methodology to evaluate a given training intervention and to appraise the evidence base for training fairly, choosing from among both quantitative and qualitative approaches and applying measurement at multiple levels of training impact.     

Comparison of academic, application form and social factors in predicting early performance on the medical course

OBJECTIVES: To compare the relative importance of social, academic and application form factors at admission in predicting performance in the first 3 years of a medicine course. DESIGN: Retrospective cohort study. SETTING: A single UK medical school. PARTICIPANTS: A total of 738 students who entered medical school between 1994 and 1997. MAIN OUTCOME MEASURE: Performance in Year 3 objective structured clinical examination (OSCE). RESULTS: School-leaving grades were significant predictors of success in the OSCE. Non-academic activities as assessed from the application form were associated with poorer performance. Mature students performed extremely well, and male and ethnic minority students performed less well. Socioeconomic status and type of school attended were not found to affect performance on the course. CONCLUSIONS: The relatively poor performance of male and ethnic minority students urgently needs further investigation. Our results carry no suggestion that, other things being equal, widening access to medical school for mature students and those from less affluent backgrounds would result in poorer performance.     

Competence in the musculoskeletal system: assessing the progression of knowledge through an undergraduate medical course

BACKGROUND: Professional bodies have expressed concerns that medical students lack appropriate knowledge in musculoskeletal medicine despite its high prevalence of use within the community. Changes in curriculum and teaching strategies may be contributing factors to this. There is little evidence to evaluate the degree to which these concerns are justified. OBJECTIVES: To design and evaluate an assessment procedure that tests the progress of medical students in achieving a core level of knowledge in musculoskeletal medicine during the course. PARTICIPANTS AND SETTING: A stratified sample of 136 volunteer students from all 5 years of the medical course at Sheffield University. METHODS: The progress test concept was adapted to provide a cross-sectional view of student knowledge gain during each year of the course. A test was devised which aimed to provide an assessment of competence set at the standard required of the newly qualified doctor in understanding basic and clinical sciences relevant to musculoskeletal medicine. The test was blueprinted against internal and external guidelines. It comprised 40 multiple-choice and extended matching questions administered by computer. Six musculoskeletal practitioners set the standard using a modified Angoff procedure. RESULTS: Test reliability was 0.6 (Cronbach's alpha). Mean scores of students increased from 41% in Year 1 to 84% by the final year. Data suggest that, from a baseline score in Year 1, there is a disparate experience of learning in Year 2 that evens out in Year 3, with knowledge progression becoming more consistent thereafter. All final year participants scored above the standard predicted by the Angoff procedure. CONCLUSIONS: This short computer-based test was a feasible method of estimating student knowledge acquisition in musculoskeletal medicine across the undergraduate curriculum. Tested students appear to have acquired a satisfactory knowledge base by the end of the course. Knowledge gain seemed relatively independent of specialty-specific clinical training. Proposals from specialty bodies to include long periods of disciplinary teaching may be unnecessary.     

Measuring errors and adverse events in health care

In this paper, we identify 8 methods used to measure errors and adverse events in health care and discuss their strengths and weaknesses. We focus on the reliability and validity of each, as well as the ability to detect latent errors (or system errors) versus active errors and adverse events. We propose a general framework to help health care providers, researchers, and administrators choose the most appropriate methods to meet their patient safety measurement goals.     

Improvements to the enzyme-developed radial immunodiffusion technique

An enzyme-developed radial immunodiffusion technique, previously known as the diffusion-in-gel enzyme-linked immunosorbent assay (DIG-ELISA), has been improved in two ways: (a) antibody-containing spots have been made larger and more distinct by revealing them with a mixture of hydrogen peroxide, 3,3′-diaminobenzidine and nickel, and further intensification of the ensuing spots with silver; (b) the reliability of the method has been enhanced by chemically coupling the antigen to a layer of a polyamino acid (poly(lysine, phenylalanine)) adsorbed to the bottom of the polystyrene petri dish. The usefulness of the improved technique is illustrated by reference to the measurement of serum concentrations of IgM and IgG, and in the assessment of antibody levels against a particulate antigen (erythrocytes).