Workplace‐based assessments of junior doctors: do scores predict training difficulties?

Medical Education 2011: 45 : 1190–1198 Objectives Workplace‐based assessment (WPBA) is an increasingly important part of postgraduate medical training and its results may be used as evidence of professional competence. This study evaluates the ability of WPBA to distinguish UK Foundation Programme (FP) doctors with training difficulties and its effectiveness as a surrogate marker for deficiencies in professional competence. Methods We conducted a retrospective observational study using anonymised records for 1646 trainees in a single UK postgraduate deanery. Data for WPBAs conducted from August 2005 to April 2009 were extracted from the e‐portfolio database. These data included all scores submitted by trainees in FP years 1 and 2 on mini‐clinical evaluation exercise (mini‐CEX), case‐based discussion (CbD), direct observation of procedural skills (DOPS) and mini‐peer assessment tool (mini‐PAT) assessments. Records of trainees in difficulty, as identified by their educational supervisors, were tagged as index cases. Main outcome measures were odds ratios (ORs) for associations between mean WPBA scores and training difficulties. Further analyses by the reported aetiology of the training difficulty (health‐, conduct‐ or performance‐related) were performed. Results Of the 1646 trainees, 92 had been identified as being in difficulty. Mean CbD and mini‐CEX scores were lower for trainees in difficulty and an association was found between identified training difficulties and average scores on the mini‐CEX (OR = 0.54; p = 0.034) and CbD (OR = 0.39; p = 0.002). A receiver operator characteristic curve analysis of mean WPBA scores for diagnosing ‘in difficulty’ status yielded an area under the curve of 0.64, indicating weak predictive value. There was no statistical evidence that mean scores on DOPS and mini‐PAT assessments differed between the two groups. Conclusions Analysis of a large dataset of WPBA scores revealed significant associations between training difficulties and lower mean scores on both the mini‐CEX and CbD. Models show that using WPBA scores is, however, not a valid way of screening for trainees in difficulty. Workplace‐based assessments have value as formative assessments that prompt supervision, feedback and reflection. They should not be relied upon to certify competence and their use for such ends may reduce their effectiveness in training. Their results should be interpreted in the context of multiple other methods of assessment, with the aim of achieving a genuinely holistic and representative assessment of professional competence.     

Surgeons' participation in continuing medical education: is it evidence‐based?

BacKGROUND: Interactive forms of continuing medical education (CME) are more likely to improve clinical practice than traditional, passive approaches. This study investigated CME participation and preferences among surgeons. METHOD: Questionnaire survey of surgeons in New South Wales, Australia. RESULTS: On average, respondents (n = 418, 77% response rate) committed 364 hours (interquartile range 228-512 hours) to CME per year. Surgeons working at tertiary referral teaching hospitals were twice as likely as those working in other types of hospital to report spending more than 12 hours per month on CME (OR 2.1, 95% CI: 1.4-3.1). Overall, reading accounted for 17% of CME time and attending conferences a further 12%. Clinical audit accounted for significantly less CME time (3.5%) (both P < 0.001). Conferences were considered the single most useful form of CME by 28% (95% CI: 24-33%). Over half (55%, 95% CI: 50-59%) ranked reading as 1 of the 3 most useful types of CME, whereas significantly fewer so ranked clinical audit (6%, 95% CI: 4-9%) (chi2 = 230.8, 1 d.f., P < 0.001). CONCLUSION: Australian surgeons commit a considerable amount of time to CME, but much of this time is spent in passive educational activities. Development of acceptable and effective CME programmes will benefit both surgeons and their patients.     

Web‐ or paper‐based portfolios: is there a difference?

OBJECTIVE: To determine the differential effects of a paper-based versus a web-based portfolio in terms of portfolio quality, user-friendliness and student motivation. METHODS: An experimental design was used to compare Year 1 medical students' reflective portfolios. The portfolios differed in presentation medium only (i.e. web-based versus paper-based). Content analysis, a student questionnaire and mentor interviews were used to evaluate portfolio quality, user-friendliness and student motivation. A total of 92 portfolios were scored independently by 2 raters using a portfolio quality-rating instrument. RESULTS: Portfolio structure, quality of reflection and quality of evidence showed no significant effects of presentation medium. Multi-level analysis showed a significant effect for student motivation: web-based portfolios scored 0.39 more than paper-based portfolios (P < 0.05; effect size 0.76). The mentors reported no differences in portfolio quality, except that there were more visuals in web-based portfolios. Students spent significantly more time preparing the web-based than the paper-based portfolios (15.4 hours versus 12.2 hours; t = 2.1, P < 0.05; effect size 0.46). The 2 student groups did not differ significantly in terms of their satisfaction with the portfolio. The mentors perceived the web-based portfolios as more user-friendly. CONCLUSIONS: The web-based portfolios were found to enhance students' motivation, were more user-friendly for mentors, and delivered the same content quality compared with paper-based portfolios. This suggests that web-based presentation may promote acceptance of portfolios by students and teachers alike.     

The ‘expert’ in problem‐based and case‐based learning: necessary or not?

BACKGROUND: Problem-based learning (PBL) and other small-group teaching methods which incorporate principles of adult learning, are exciting innovations in medical education. In the application of these methods to medical curricula many schools have introduced non-expert tutors. However, research evaluating the effectiveness of non-expert lead teaching has been inconclusive. AIM: The present study aim was to compare the outcome of teaching in small groups facilitated by either an 'expert' or a 'non-expert' tutor, in a single topic area. METHOD: Fourth-year medical students were allocated randomly to teaching of eating disorders either by a non-expert or an expert tutor. Outcomes were evaluated by (i) a knowledge test, and (ii) self-report ratings by tutors and students of their learning methods and other qualities. RESULTS: The study found that while the non-expert tutor was rated more highly for her group management skills, and she also rated her students more highly in the area of oral communication, students who were taught by the expert scored higher in the end-of-course test in the topic area. CONCLUSION: The findings suggest that caution should be exercised, and the need for more research before widespread adoption of teaching by non-expert tutors.     

Scientist or science‐stuffed? Discourses of science in North American medical education

Context: The dominance of biomedical science in medical education has been contested throughout the past century, with recurring calls for more social science and humanities content. The centrality of biomedicine is frequently traced back to Abraham Flexner’s 1910 report, ‘Medical Education in the United States and Canada’. However, Flexner advocated for a scientist‐doctor, rather than a curriculum filled with science content. Examination of the discourses of science since Flexner allows us to explore the place of various knowledge forms in medical education. Methods: A Foucauldian critical discourse analysis was performed, examining the discourses of scientific medicine in Flexner’s works and North American medical education articles in subsequent decades. Foucault’s methodological principles were used to identify statements, keywords and metaphors that emerged in the development of the discourses of scientific medicine, with particular attention to recurring arguments and shifts in the meaning and use of terms. Results: Flexner’s scientist‐doctor was an incisive thinker who drew upon multiple forms of knowledge. In the post‐Flexner medical education reforms, the perception of science as a discursive object embedded in the curriculum became predominant over that of the scientist as the discursive subject who uses science. Science was then considered core curricular content and was discursively framed as impossibly vast. A parallel discourse, one of the insufficiency of biomedical science for the proper training of doctors, has existed over the past century, even as the humanities and social sciences have remained on the margins in medical school curricula. Conclusions: That discourses of scientific medicine have reinforced the centrality of biomedicine in medical education helps to explain the persistent marginalisation of other important knowledge domains. Medical educators need to be aware of the effects of these discourses on understandings of medical knowledge, particularly when contemplating curricular reform.     

Cost‐effectiveness analysis based on the number‐needed‐to‐treat: common sense or non‐sense?

This paper explores and critically discusses some of the methodological limitations of using the number-needed-to-treat (NNT) in economic evaluation. We argue that NNT may be a straightforward measure of benefit when the effect of an intervention is immediate, but that serious problems arise when the effect is delay rather than avoidance of an adverse event. In this case, NNT is not a robust or accurate measure of effect, but will vary considerably and inconsistently over time. This weakness will naturally spill over onto any CEA based on NNT. A literature review demonstrated that CEAs based on NNT were all published within the last five years, and that all studies suffered from important limitations. A major weakness of using NNT is the imposed restrictions on the outcome measure, which can only be strictly uni-dimensional and non-generic. Using NNT in economic evaluations is obtained at a cost in terms of both methodological shortcomings, and a reduced ability for such evaluations to serve as a useful tool in decision making processes. The use of NNT in economic evaluations might be better avoided.To every complicated question, there is a simple, straightforward, easy--and probably wrong answer (Occam's Sledgehammer).     

Workplace‐based assessments in Foundation Programme training: do trainees in difficulty use them differently?

Context Trainee‐led workplace‐based assessment (WPBA) is increasingly used in postgraduate medical training. Trainees in difficulty are known to behave differently from their peers; these differences may be reflected in their use of WPBAs and may give new insights into the behaviour and assessment of struggling trainees. Methods Data were extracted for 76 115 assessments, completed by 1900 UK Foundation Programme (FP) trainees. Of these 1900 trainees, 95 (5%) were FP trainees in difficulty (FTiDs). We analysed aspects of the use of WPBAs, using multiple logistic regressions, to compare the behaviours of FTiDs with those of their peers. Results Of 48 possible comparisons, only two (i.e. the rate expected to occur by chance) showed statistically significant differences: relative to their peers, FTiDs were more likely to choose nurse assessors in direct observations of procedural skills (odds ratio [OR] 7.05, 95% confidence interval [CI] 1.23–40.43) and more likely to choose non‐clinical assessors for assessments using the mini‐peer assessment tool (OR 30.44, 95% CI 1.34–689.29). Conclusions Key features of assessor choice for FTiDs are familiarity and likelihood of receiving a positive assessment. This analysis has not demonstrated that FTiDs use WPBAs any differently from their peers who are not in difficulty, although it does suggest associations and trends that require further exploration. These null results are interesting and raise hypotheses for prospective confirmation or disproof, and for further qualitative work investigating how struggling trainees use WPBAs in order to guide the future implementation of WPBAs in postgraduate training.     

Evidence‐based practice?

In 1948, the world of health and medical research changed forever with the publication of the Medical Research Council's randomised controlled trial (RCT) on the use of streptomycin in the treatment of tuberculosis (TB).1,2 In that year, there were 341 papers on Streptomycin and TB published in the medical literature, but only one RCT, and this was the paper that defined a generation of TB treatment. The process of undertaking RCTs was quickly developed to include health promotion.     

Champions of science or blacklisting bureaucrats?

Unconvinced that science could police itself against research misconduct, the federal government in 1989 created an Office of Scientific Integrity (OSI) within the National Institutes of Health. Since then, OSI has launched high-profile investigations into big-name scientists using a method that researchers say violates their due process of law and destroys reputations. Changes are in the offing, but are they enough to protect the integrity of government-funded research and scientists' right to due process?     

'Detox': science or sales pitch?

There is no question that the world is becoming increasingly toxic, with worldwide dissemination of industrial chemicals, pesticides, heavy metals and radioactive elements. Many of these toxins have demonstrated harmful effects including cancer, reproductive, metabolic, and mental health effects. It is also known that many toxins undergo bioaccumulation through the food chain and that synergistic effects can occur whereby combinations of toxins can be more potent than the sum of individual toxins.     

Meta‐analysis of Gaussian individual patient data: Two‐stage or not two‐stage?

Quantitative evidence synthesis through meta‐analysis is central to evidence‐based medicine. For well‐documented reasons, the meta‐analysis of individual patient data is held in higher regard than aggregate data. With access to individual patient data, the analysis is not restricted to a “two‐stage” approach (combining estimates and standard errors) but can estimate parameters of interest by fitting a single model to all of the data, a so‐called “one‐stage” analysis. There has been debate about the merits of one‐ and two‐stage analysis. Arguments for one‐stage analysis have typically noted that a wider range of models can be fitted and overall estimates may be more precise. The two‐stage side has emphasised that the models that can be fitted in two stages are sufficient to answer the relevant questions, with less scope for mistakes because there are fewer modelling choices to be made in the two‐stage approach. For Gaussian data, we consider the statistical arguments for flexibility and precision in small‐sample settings. Regarding flexibility, several of the models that can be fitted only in one stage may not be of serious interest to most meta‐analysis practitioners. Regarding precision, we consider fixed‐ and random‐effects meta‐analysis and see that, for a model making certain assumptions, the number of stages used to fit this model is irrelevant; the precision will be approximately equal. Meta‐analysts should choose modelling assumptions carefully. Sometimes relevant models can only be fitted in one stage. Otherwise, meta‐analysts are free to use whichever procedure is most convenient to fit the identified model.     

Pharmacovigilance: a science or fielding emergencies?

Pharmacovigilance involves the assessment of risks and benefits of medicines. There are legal and regulatory aspects of this process, and the licensing of a new medicine is always provisional. The systems, in the past, have had limited statistical involvement, either in terms of personnel who are statistically trained, or in terms of statistical methods. This is changing. The high profile activities of pharmacovigilance have often been emergencies, though most is routine activity. Application of statistical thinking and of techniques is being done to help detect adverse effects of medicines rather earlier so that some emergencies may be avoided.     

Do clinical pathways enhance access to evidence‐based acute myocardial infarction treatment in rural emergency departments?

Objective: The objective of this study is to measure the impact of a five‐step implementation process for an acute myocardial infarction (AMI) clinical pathway (CPW) on thrombolytic administration in rural emergency departments. Design: Cluster randomised controlled trial. Setting: Six rural Victorian emergency departments participated. Intervention: The five‐step CPW implementation process comprised (i) engaging clinicians; (ii) CPW development; (iii) reminders; (iv) education; and (v) audit and feedback. Main outcome measures: The impact of the intervention was assessed by measuring the proportion of eligible AMI patients receiving a thrombolytic and time to thrombolysis and electrocardiogram. Results: Nine hundred and fifteen medical records were audited, producing a final sample of 108 patients eligible for thrombolysis. There was no significant difference between intervention and control groups for median door‐to‐needle time (29 mins versus 29 mins; P = 0.632), proportion of those eligible receiving a thrombolytic (78% versus 84%; P = 0.739), median time to electrocardiogram (7 mins versus 6 mins; P = 0.669) and other outcome measures. Results showed superior outcome measures than other published studies. Conclusions: The lack of impact of the implementation process for a chest pain CPW on thrombolytic delivery or time to electrocardiogram in these rural hospitals can be explained by a ceiling effect in outcome measures but was also compromised by the small sample. Results suggest that quality of AMI treatment in rural emergency departments (EDs) is high and does not contribute to the worse mortality rate reported for AMIs in rural areas.