An expert-performance perspective of research on medical expertise: the study of clinical performance

CONTEXT: Three decades ago Elstein et al. published their classic book on medical expertise, in which they described their failure to identify superior performance by peer-nominated diagnosticians using high- and low-fidelity simulations of the everyday practice of doctors. OBJECTIVE: This paper reviews the results of subsequent research, with a particular emphasis on the progress toward Elstein et al.'s goal of capturing the essence of superior clinical performance in standardised settings in order to improve clinical practice. RESULTS: Research following publication of Elstein et al.'s book was influenced by laboratory research in cognitive psychology, which resulted in a redirection of its original focus on capturing clinical performance in practice to studies of changes in cognitive processes as functions of extended clinical experience. There is currently renewed interest in linking laboratory research with studies of the acquisition of superior (expert) performance in the clinic. CONCLUSIONS: Research on medical expertise and simulation training in technical procedures and diagnosis provide exciting opportunities for establishing translational research on the acquisition of superior (expert) performance in the clinic by capturing it with representative tasks in the laboratory, reproducing it for experimental analysis, and developing training activities, such as deliberate practice, that can induce measurable improvements in performance in the clinic.     

Science is fundamental: the role of biomedical knowledge in clinical reasoning

CONTEXT: Although training in basic science is generally considered a critical aspect of medical education, there is little consensus regarding its precise role in clinical reasoning. Whereas some reports suggest that biomedical knowledge is rarely used in routine diagnosis, other research has found that biomedical knowledge can become an integral part of the expert knowledge base. OBJECTIVE: The purpose of the current paper is to present evidence in support of different views regarding the role of biomedical knowledge, including the two-world hypothesis, encapsulation theory and recent work on the role of biomedical knowledge in novice diagnosticians. The implications of these models for clinical teaching will be examined. DISCUSSION: Recent work suggests that biomedical knowledge can help novices develop a coherent and stable mental representation of disease categories. As a result, learners are able to retain clinical knowledge over time and maintain diagnostic accuracy when faced with clinical challenges. This suggests that clinical teachers should attempt to make explicit connections between biomedical knowledge and clinical facts during training.     

Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel

The Script Concordance Test (SCT) assesses reasoning in the context of uncertainty. Because there is no single correct answer, scoring is based on the comparison of answers provided by examinees with those provided by members of a reference panel made up of experienced practitioners. The study aimed to assess the discriminatory power of the SCT based on the variability of the reference panel's answers. Items from a bank covering different family medicine domains were classified into 3 groups according to the degree of variability of answers provided by a pool of experienced doctors. A variability index (mean squared error) was used to select items in the low, moderate and high variability categories. A 102-item test (Cronbach's alpha 0.70), made up of 3 subtests of each category, was administered to 3 contrasting groups in family medicine: 157 clerkship students, 30 residents and 30 practising doctors. anova and effect size (ES) were used to quantify and test the discrimination power of the 3 subtests. The high variability subtest showed high effect size for discrimination between extreme groups (ES = 1.5; F = 16.3, P < 0.001), whereas the moderate variability subtest showed less effect size (ES = 0.56; F = 57, P = 0.041). The low variability subtest did not discriminate significantly (ES = 0.31; F = 2.9, P = 0.06). Variability of answers within the reference panel is a key component of the discriminatory power of the SCT. In accordance with theory, the presence of variability ensures discrimination between levels of clinical experience. These results imply important considerations for the construction of efficient SCTs.     

Inducing expertise effects in clinical case recall

BACKGROUND: This study was directed at illuminating a well known phenomenon in the medical expertise literature, the 'intermediate effect' in clinical case recall. This robust phenomenon consists of the finding that medical students of intermediate levels of expertise outperform both experts and novices in clinical case recall after diagnosing cases. It deals in particular with the findings of OME researchers who have reported a monotonically increasing recall with level of expertise. PURPOSE: To address possible causes for this anomaly in medical expertise and to experimentally demonstrate how data elaboration can cause expertise effects in clinical case recall. METHOD: Expert nephrologists, intermediate level students and novices were presented with 6 medical cases under 3 different conditions: laboratory data cases without special instructions, laboratory data cases with instructions to elaborate, and cases with laboratory data and a relevant clinical context. RESULTS: Only when participants were required to elaborate on each of the information units presented to them did case recall show an expertise effect. If laboratory data are framed within the context of a patient's history and physical examination data, the 'intermediate effect' appears. CONCLUSIONS: The instructions used in the elaboration condition seem to have induced a deeper, more detailed, analysis of the patient case. It is therefore interesting to note that these instructions only affected the recall of the experts and had no effect on the novices' or intermediates' recall. We might conclude from this that expertise effects in clinical case recall are only produced when the normal processing of patient information is disrupted.     

Teaching from the clinical reasoning literature: combined reasoning strategies help novice diagnosticians overcome misleading information

OBJECTIVE: Previous research has revealed a pedagogical benefit of instructing novice diagnosticians to utilise a combined approach to clinical reasoning (familiarity-driven pattern recognition combined with a careful consideration of the presenting features) when diagnosing electrocardiograms (ECGs). This paper reports 2 studies demonstrating that the combined instructions are especially valuable in helping students overcome biasing influences. METHODS: Undergraduate psychology students were trained to diagnose 10 cardiac conditions via ECG presentation. Half of all participants were instructed to reason in a combined manner and half were given no explicit instruction regarding the diagnostic task. In Study 1 (n = 60), half of each group was biased towards an incorrect diagnosis through presentation of counter-indicative features. In Study 2 (n = 48), a third of the test ECGs were presented with a correct diagnostic suggestion, a third with an incorrect suggestion, and a third without a suggestion. RESULTS: Overall, the instruction to utilise a combined reasoning approach resulted in greater diagnostic accuracy relative to leaving students to their own intuitions regarding how best to approach new cases. The effect was particularly pronounced when cases were made challenging by biasing participants towards an incorrect diagnosis, either through mention of a specific feature or by making an inaccurate diagnostic suggestion. DISCUSSION: These studies advance a growing body of evidence suggesting that various diagnostic strategies identified in the literature on clinical reasoning are not mutually exclusive and that trainees can benefit from explicit guidance regarding the value of both analytic and non-analytic reasoning tendencies.     

The benefits of flexibility: the pedagogical value of instructions to adopt multifaceted diagnostic reasoning strategies

Objectives Building on the advice of previous research to avoid parsing diagnostic strategies too finely, recent studies have shown that teaching novices to utilise analytic and non‐analytic reasoning strategies yields higher diagnostic accuracy than teaching either in isolation. This study assesses the extent to which students spontaneously adopt a combined approach and compares its benefits with those experienced with a contrastive learning strategy known to enhance analogical transfer. Methods A sample of 48 naïve students were trained to identify features on electrocardiograms (ECGs) and assign diagnoses. Half the participants learned in a standard manner, encountering diagnoses (and their associated features) in sequence. The remaining participants were explicitly instructed to draw comparisons between the diagnostic category being learned and another confusable diagnostic category (contrastive learning). Half the participants in both groups were further instructed to carefully identify all features while trusting guidance provided by feelings of familiarity (a combined reasoning strategy). The remaining participants were given no instructions on how to approach the diagnostic task. Results Greater diagnostic accuracy was achieved following both contrastive learning and instructions to use a combined reasoning strategy relative to the control conditions. These variables did not interact with each other, nor did they interact with novelty of the test case. The effects were observed immediately after learning and following a 1‐week delay. Discussion The results emphasise the importance of explicitly empowering students to utilise multiple diagnostic strategies, including non‐analytic approaches. In addition, this study reveals the benefit that can be gained from contrastive learning in a medical domain.     

Making sense of clinical reasoning: judgement and the evidence of the senses

BACKGROUND: Close noticing, as keen discrimination and judgement between qualities, is a key capability for work in visual domains in medicine. This generic capability is normally assumed, and its specifics are left to develop through experience, as traditional apprenticeship in a specialty. Discrimination is an outcome of learning in the affective domain, and introduces a vital aesthetic dimension to clinical work that aligns with the interests of the medical humanities. An aesthetic approach to clinical reasoning, however, remains largely unexplored as an explicit focus for medical education. FRAMEWORK AND PRACTICE PARADOXES: We offer a framework for an explicit education of perceptual discrimination in the visual domain as a form of practice 'artistry', turning a surface 'looking' into a deeper 'seeing'. Such an education, however, raises certain paradoxes. While novices typically 'see' what they expect to see in visual images (sign and symptom), experts also make similar errors. Further, experts become familiar with the use of visual heuristics in diagnosis, such as vivid natural referents to aid in rapid pattern recognition in an encompassing diagnostic 'glance', yet this appears to defeat the first principle of describing what you see. Employing a model of imagination as a tacit form of knowing that 'prepares' and enhances perception, we suggest, however, that the judicious use of such heuristics can be positive. Moreover, the mechanics of the process of clinical judgement in visual domains can be detailed to inform educational agendas. A further paradox is that of experts using both idiosyncratic heuristics and protocol-driven practices, where these seem to offer contradictory approaches to gaining knowledge. We recognise this as a facet of medicine's inherent uncertainty, in the face of complex, ambiguous and unique material, that must be addressed through clinical education. CONCLUSION: We equate 'aesthetics' with 'sensibility' and describe clinical expertise as 'connoisseurship' of informational images. Such connoisseurship, a particular form of knowing, can, in turn, be defined as an aesthetic sensibility informing practice artistry. It can be articulated and analysed to provide a basis for educational enhancement. Connoisseurship is not a technical-rational procedure but is inherently paradoxical and such paradox may be valued as an educational resource, rather than seen as a hindrance.     

Inter-rater agreement in judging errors in diagnostic reasoning

CONTEXT AND OBJECTIVE: Inter-rater agreement is essential in rating clinical performance of doctors and other health professionals. The purpose of this study was to establish inter-rater agreement in categorising errors in the diagnostic process made by clinicians using computerised decision support systems. METHODS: Eight possible error categories were developed for coding errors in diagnostic hypotheses and plans for next steps in the work-up. Two independent doctor judges rated 54 work-ups (representing 2 cases, each worked-up by 27 doctors). Inter-rater agreement between the 2 raters was computed using the kappa coefficient. RESULTS: High inter-rater agreement was achieved in all categories except where the manual was not sufficiently specific and raters had to use their judgement. As is typical of the kappa coefficient, however, agreement corrected for chance fell markedly into the "poor" range when the percentages of expected and observed agreement were about the same. CONCLUSION: Raters can achieve good agreement in categorising errors provided they are given explicit scoring rules and do not rely solely upon clinical judgement. The kappa coefficient has limitations in cases where the expected agreement between judges is high and variability is low. The use of 2 indices to assess agreement, analogous to test sensitivity and specificity, is recommended.     

The clinical reasoning process

Medical school teachers must have an accurate idea of the doctor's clinical reasoning process (CRP) in order to provide students with learning experiences and evaluations that will ensure their acquisition of an effective and efficient CRP. It is difficult to derive this understanding from much that has been written on the subject. It is important to recognize that clinical problems are ill-structured and that the doctor's reasoning is built around a temporal unfolding of information. A model for the CRP is described along with a critique of other models that have been suggested. The results of research that examines components of the process must be seen in relation to the overall process.     

Trialling a new way to learn clinical skills: systematic clinical appraisal and learning

AIM: To describe and evaluate the effectiveness of a new method of teaching clinical skills designed to increase students' active and self-directed learning as well as tutor feedback. METHODS: A total of 22 consenting Year 4 medical students undertaking general practice and general surgery clinical experience were involved in a pre- and post-test research design. In the initial period of the study, students were taught clinical skills in a traditional manner. In the second phase a clinical teaching strategy called systematic clinical appraisal and learning (SCAL) was utilised. This learning strategy involved active and self-directed learning, holistic care and immediate feedback. Students independently saw a patient and were asked to make judgements about the patient's potential diagnosis, tests required, management, psychosocial needs, preventive health requirements, and any ethical problems. These judgements were then compared with those of the clinical supervisor, who saw the same patient independently. Students recorded details for each consultation. Comparisons were made of the two study periods to examine whether the use of SCAL increased the number of students' independent judgements, perceived student learning, tutor feedback and self-directed learning. RESULTS: During the SCAL learning period, students reported making a greater number of statistically significant independent judgements, and receiving significantly increased tutor feedback in both general practice and general surgery. The number of learning goals set by students was not found to differ between the two periods in surgery but significantly increased in general practice in the SCAL period. Students' perceptions of their learning significantly increased in the SCAL period in surgery but not in general practice. During the traditional learning period in both settings, there was limited student decision-making about most aspects of care, but particularly those relating to prevention, psychosocial issues and ethics. CONCLUSIONS: The SCAL approach appears to offer some advantages over traditional clinical skills teaching. It appears to encourage active and independent decision-making, and to increase tutor feedback. Further exploration of the approach appears warranted.     

Breaking down automaticity: case ambiguity and the shift to reflective approaches in clinical reasoning

CONTEXT: Two modes of case processing have been shown to underlie diagnostic judgements: analytical and non-analytical reasoning. An optimal form of clinical reasoning is suggested to combine both modes. Conditions leading doctors to shift from the usual mode of non-analytical reasoning to reflective reasoning have not been identified. This paper reports a study aimed at exploring these conditions by investigating the effects of ambiguity of clinical cases on clinical reasoning. METHODS: Participants were 16 internal medicine residents in the Brazilian state of Ceará. They were asked to diagnose 20 clinical cases and recall case information. The independent variable was the degree of ambiguity of clinical cases, with 2 levels: straightforward (i.e. non-ambiguous) and ambiguous. Dependent variables were processing time, diagnostic accuracy and proposition per category recalled. Data were analysed using a repeated measures design. RESULTS: Participants processed straightforward cases faster and more accurately than ambiguous ones. The proportion of text propositions recalled was significantly lower (t[15] = 2.29, P = 0.037) in ambiguous cases, and an interaction effect between case version and proposition category was also found (F[5, 75] = 4.52, P = 0.001, d = 0.232, observed power = 0.962). Furthermore, participants recalled significantly more literal propositions from the ambiguous cases than from the straightforward cases (t[15] = 2.28, P = 0.037). CONCLUSIONS: Ambiguity of clinical cases was shown to lead residents to switch from automatic to reflective reasoning, as indicated by longer processing time, and more literal propositions recalled in ambiguous cases.     

Diagnostic ability in relation to clinical seminars and extended-matching questions examinations

CONTEXT: At the Katholieke Universiteit Leuven, Belgium, final year medical students participate in 70 problem-solving clinical seminars over a period of 2 months, concentrating on clinical reasoning for formulating differential diagnoses. The clinical seminars end in an examination consisting of 200 extended-matching questions (EMQs). OBJECTIVES: This study asks whether problem-solving clinical seminars improve clinical reasoning and whether the EMQ examination measures clinical decision making. In order to detect an increase in diagnostic ability, the Diagnostic Thinking Inventory (DTI) was applied. The research hypotheses were: (i) DTI scores will be higher after the seminars than before, and (ii) the correlations between DTI scores and EMQ examination scores will be significant. METHODS: In the academic year 2003-04, 3 series of problem-solving clinical seminars were held. At the beginning and end of each series the students filled in the DTI. This questionnaire measures 2 aspects of diagnostic thinking: the degree of flexibility in thinking, and how knowledge is structured in the memory. RESULTS: For all data together, the DTI scores after the clinical seminars were significantly higher than before. Pearson correlations between DTI scores and EMQ examination scores were low but significant, with the exception of post-test Structure (not significant). CONCLUSIONS: Two months of intensive problem-solving clinical training was accompanied by an improvement in diagnostic thinking, as measured by DTI scores. Correlation between DTI scores and examination scores indicates that the EMQs measure an aspect of student achievement that is related to clinical reasoning.     

A practical guide to assessing clinical decision-making skills using the key features approach

AIM: This paper in the series on professional assessment provides a practical guide to writing key features problems (KFPs). Key features problems test clinical decision-making skills in written or computer-based formats. They are based on the concept of critical steps or 'key features' in decision making and represent an advance on the older, less reliable patient management problem (PMP) formats. METHOD: The practical steps in writing these problems are discussed and illustrated by examples. Steps include assembling problem-writing groups, selecting a suitable clinical scenario or problem and defining its key features, writing the questions, selecting question response formats, preparing scoring keys, reviewing item quality and item banking. CONCLUSION: The KFP format provides educators with a flexible approach to testing clinical decision-making skills with demonstrated validity and reliability when constructed according to the guidelines provided.     

Initial diagnostic hypotheses bias analytic information processing in non-visual domains

Context  Previous studies have shown that an initial diagnostic hypothesis biases automatic information processing. It is unclear if an initial hypothesis has a similar effect on analytic information processing. Our first objective was to study the effect of an initial diagnostic hypothesis on analytic processing. Our second objective was to assess the effect of clinical experience on analytic processing by evaluating the effect of clinical frequency and urgency of an alternative diagnosis on diagnosis selection.
Methods  During a 12-minute objective structured clinical examination station, 19 subspecialty medical residents diagnosed the cause of 3 clinical presentations: dyspnoea; headache, and chest pain. Subjects were randomly allocated cases for which the suggested initial hypothesis was either correct or incorrect. For cases with an incorrect initial hypothesis, the alternative diagnoses varied in the frequency with which they are encountered in clinical practice, and their clinical urgency, relative to the initial diagnostic hypothesis.
Results  All correct initial hypotheses were retained, compared with 10.9% of incorrect hypotheses. All cases with a correct initial hypothesis were diagnosed correctly, compared with 65.2% of cases with an incorrect hypothesis (risk ratio 1.5 [95% confidence interval 1.2–1.9], P  =   0.02). Clinical frequency and urgency were not associated with alternative diagnosis selection.
Discussion  Our results suggest that an initial diagnostic hypothesis biases analytic processing. The data used to reject an initial hypothesis appear to drive selection of an alternative hypothesis. Further studies aimed at finding strategies for increasing the likelihood of generating a correct initial hypothesis or debiasing an initial hypothesis are needed.     

Validation of core medical knowledge by postgraduates and specialists

BACKGROUND: Curriculum constructors and teachers must decide on the content and level of objectives and materials included in the medical curriculum. At University Medical Centre Utrecht it was decided to test relatively detailed knowledge at a regular level in study blocks and to design a progress test aimed at the medical core knowledge that every graduating doctor should possess. This study was conducted to validate the level of knowledge tested in this progress test. AIM: We designed a questionnaire to investigate whether postgraduate trainees and experienced specialists agree with item writers on what is required core knowledge. METHODS: Postgraduates and specialists received a questionnaire with 80 items designed to test core knowledge. Respondents were asked to indicate to what extent the items actually represented the core knowledge required of a recently graduated medical student. RESULTS: Of the clinical questions, 82.4% were judged to reflect core knowledge, whereas only 42.4% of the basic science questions were judged to reflect core knowledge. There was a strikingly high correlation on the mean judgements per item of postgraduate trainees versus medical specialists (r = 0.975). CONCLUSION: Many items, written to reflect core knowledge, appear to be judged by postgraduates and clinicians as pertaining to non-core knowledge. Postgraduate trainees appear to be as capable as experienced specialists of making judgements regarding core knowledge. Fewer basic science items are regarded as core knowledge than clinical items. This may suggest that, specifically, basic science teachers do not agree with physicians on what is to be considered medical core knowledge for graduating doctors.     

How residents learn: qualitative evidence for the pivotal role of clinical activities

OBJECTIVES: Medical councils worldwide have outlined new standards for postgraduate medical education. This means that residency programmes will have to integrate modern educational views into the clinical workplace. Postgraduate medical education is often characterised as a process of learning from experience. However, empirical evidence regarding the learning processes of residents in the clinical workplace is lacking. This qualitative study sought insight into the intricate process of how residents learn in the clinical workplace. METHODS: We carried out a qualitative study using focus groups. A grounded theory approach was used to analyse the transcribed tape recordings. A total of 51 obstetrics and gynaecology residents from teaching hospitals and affiliated general hospitals participated in 7 focus group discussions. Participants discussed how they learn and what factors influence their learning. RESULTS: An underlying theoretical framework emerged from the data, which clarified what happens when residents learn by doing in the clinical workplace. This framework shows that work-related activities are the starting point for learning. The subsequent processes of 'interpretation' and 'construction of meaning' lead to refinement and expansion of residents' knowledge and skills. Interaction plays an important role in the learning process. This is in line with both cognitivist and sociocultural views on learning. CONCLUSIONS: The presented theoretical framework of residents' learning provides much needed empirical evidence for the actual learning processes of residents in the clinical workplace. The insights it offers can be used to exploit the full educational potential of the clinical workplace.     

What every teacher needs to know about clinical reasoning

CONTEXT: One of the core tasks assigned to clinical teachers is to enable students to sort through a cluster of features presented by a patient and accurately assign a diagnostic label, with the development of an appropriate treatment strategy being the end goal. Over the last 30 years there has been considerable debate within the health sciences education literature regarding the model that best describes how expert clinicians generate diagnostic decisions. PURPOSE: The purpose of this essay is to provide a review of the research literature on clinical reasoning for frontline clinical teachers. The strengths and weaknesses of different approaches to clinical reasoning will be examined using one of the core divides between various models (that of analytic (i.e. conscious/controlled) versus non-analytic (i.e. unconscious/automatic) reasoning strategies) as an orienting framework. DISCUSSION: Recent work suggests that clinical teachers should stress the importance of both forms of reasoning, thereby enabling students to marshal reasoning processes in a flexible and context-specific manner. Specific implications are drawn from this overview for clinical teachers.     

'Talking the talk': school and workplace genre tension in clerkship case presentations

BACKGROUND: Socialisation into a community involves learning sanctioned ways of talking. This study investigates the case presentation genre as a site of socialisation into the clinical community of practice. METHODS: Sixteen oral case presentations and the teaching exchanges surrounding them (involving 11 students and 10 faculty members) were observed by paired researchers during inpatient paediatric medicine rounds. A total of 21 in-depth interviews were conducted with 11 students and 10 faculty. Both data sets were audio-recorded, transcribed and analysed for emergent themes and rhetorical strategies. RESULTS: Students emphasised case presentation as a school genre and described the ideal presentation as free of interruptions. As a consequence, students' presentation strategies were directed towards getting through the presentation without questions. In contrast, faculty responses suggested an understanding of the genre as a way of constructing shared professional knowledge. Faculty feedback was often explicit about critical issues in constructing shared knowledge, such as handling uncertainty. However, student presentations rarely reflected this feedback. CONCLUSIONS: The school genre described and enacted by students conflicts in key ways with the workplace genre evident in faculty feedback, suggesting that school and workplace iterations of case presentation may be at cross-purposes. Such cross-purposes have implications, because when students and teachers perceive a genre differently, a 'gap' is created in their interactions. Even rich and contextually situated feedback may get lost or distorted as it crosses this gap. Explicit acknowledgement of the multiple and flexible iterations of case presentation will improve the learning that novices experience through acquiring this central form of professional 'talk'.