The effect of an extended hospital induction on perceived confidence and assessed clinical skills of newly qualified pre-registration house officers

BACKGROUND: Recent studies raise concerns over the preparedness of newly qualified doctors for the role of the pre-registration house officer (PRHO). This study aimed to assess self-perception of preparedness, objective assessment of core clinical skills and the effect of an extended clinical induction programme prior to commencing full duties. METHODS: A group of 26 newly qualified doctors from 1 district general hospital underwent an extended 5-day, ward-based induction programme. The participants completed questionnaires on their own perceptions of their preparedness for PRHO duties and underwent an objective structured clinical examination (OSCE) of 4 core clinical skills prior to induction, on completion of induction and 1 month into working life. RESULTS: At the outset PRHOs had low perceptions of their own capabilities in all clinical scenarios and skills. Most perceptions improved after induction, although in 2 clinical areas they felt even less confident. One month into post there were significant improvements in all areas. Only 1 PRHO passed all 4 clinical skills assessments at the pre-induction assessment. Seven (26%) failed on 1 or more skills at the post-induction assessment. However, all participants were deemed competent in all skills at the 1-month assessment. CONCLUSION: Newly qualified doctors do not feel prepared for PRHO duties and objectively are not competent in basic clinical skills. An extended induction improves preparedness in some but not all clinical areas and improves performance of objectively assessed clinical skills.     

Traditional finals and OSCEs in predicting consultant and self-reported clinical skills of PRHOs: a pilot study

INTRODUCTION: As we move from standard 'long case' final examinations to new objective structured formats, we need to ensure the new is at least as good as the old. Furthermore, knowledge of which examination format best predicts medical student progression and clinical skills development would be of value. METHODS: A group of medical students sat both the standard long case examination and the new objective structured clinical examination (OSCE) to introduce this latter examination to our Medical School for final MB. At the end of their pre-registration year, the group and their supervising consultants submitted performance evaluation questionnaires. RESULTS: Thirty medical students sat both examinations and 20 returned evaluation questionnaires. Of the 72 consultants approached, 60 (83%) returned completed questionnaires. No correlation existed between self- and consultant reported performance. The traditional finals examination was inversely associated with consultant assessment. Better performing students were not rated as better doctors. The OSCE (and its components) was more consistent and showed positive associations with consultant ratings across the board. DISCUSSION: Major discrepancies exist between the 2 examination formats, in data interpretation and practical skills, which are explicitly tested in OSCEs but less so in traditional finals. Standardised marking schemes may reduce examiner variability and discretion and weaken correlations across the 2 examinations. This pilot provides empirical evidence that OSCEs assess different clinical domains than do traditional finals. Additionally, OSCEs improve prediction of clinical performance as assessed by independent consultants. CONCLUSION: Traditional finals and OSCEs correlate poorly with one another. Objective structured clinical examinations appear to correlate well with consultant assessment at the end of the pre-registration house officer year.     

Clinical skills in early postgraduate medical trainees: patterns of acquisition of confidence and experience among junior doctors in a university teaching hospital

BACKGROUND: Little is known about patterns of clinical skills acquisition among junior doctors undertaking clinical training in the early postgraduate period. A better understanding would assist in the design of effective educational interventions for this group. METHODS: We conducted a cross-sectional survey of the levels of confidence and experience with a broad raft of clinical skills among early PGY1 trainees (interns), PGY2s and PGY3s within a university teaching hospital network in Sydney at the beginning of the clinical year in 1999. The instrument was a new validated 69 item questionnaire. A total of 92 respondents took part, representing 100% of the PGY1 (n=36), PGY2 (n=31) and PGY3 (n=25) cohorts. RESULTS: Commencing interns reported high confidence levels with a small group of practical skills but less confidence with clinical management skills. Significant positive differences were found for confidence with all skill areas between early PGY1 and PGY3. We identified three patterns for skills acquisition. Pattern A was the most common, with a significant difference in levels of confidence and experience between PGY1 and PGY2 but not between PGY2 and PGY3 (e.g. suturing a simple laceration). In Pattern B, significant differences were found in levels of confidence and experience between PGY1 and PGY2 as well as between PGY2 and PGY3 (e.g. cardiopulmonary resuscitation). In Pattern C, significant differences were found in levels of experience between PGY2 and PGY3 but not between PGY1 and PGY2 (e.g. endotracheal intubation). There was a significant correlation between reported confidence and experience for all skill areas. CONCLUSIONS: Early postgraduate medical trainees in a Sydney teaching hospital acquire high levels of confidence and experience in most skill areas after two years of training. The first postgraduate year is particularly significant for the development of clinical skills.     

Item response theory: applications of modern test theory in medical education

CONTEXT: Item response theory (IRT) measurement models are discussed in the context of their potential usefulness in various medical education settings such as assessment of achievement and evaluation of clinical performance. PURPOSE: The purpose of this article is to compare and contrast IRT measurement with the more familiar classical measurement theory (CMT) and to explore the benefits of IRT applications in typical medical education settings. SUMMARY: CMT, the more common measurement model used in medical education, is straightforward and intuitive. Its limitation is that it is sample-dependent, in that all statistics are confounded with the particular sample of examinees who completed the assessment. Examinee scores from IRT are independent of the particular sample of test questions or assessment stimuli. Also, item characteristics, such as item difficulty, are independent of the particular sample of examinees. The IRT characteristic of invariance permits easy equating of examination scores, which places scores on a constant measurement scale and permits the legitimate comparison of student ability change over time. Three common IRT models and their statistical assumptions are discussed. IRT applications in computer-adaptive testing and as a method useful for adjusting rater error in clinical performance assessments are overviewed. CONCLUSIONS: IRT measurement is a powerful tool used to solve a major problem of CMT, that is, the confounding of examinee ability with item characteristics. IRT measurement addresses important issues in medical education, such as eliminating rater error from performance assessments.     

Clinical skills education: outcomes of relationships between junior medical students, senior peers and simulated patients

Objectives  Peer-assisted learning (PAL) has been reported to have educational benefits in cross-year, small-group teaching in other contexts. Accordingly, we explored whether senior medical students are effective tutors for their junior peers in clinical skills education, and how the participants in the learning triad (tutors, learners and simulated patients [SPs]) perceive the learning environment created in PAL.
Methods  Year 2 students were randomly allocated to one of two groups for skills training. Group 1 ( n  = 64) were tutored by volunteer Year 6 students, and Group 2 ( n  = 67) by paid doctors. The results of both groups in a clinical skills examination were compared using an independent samples t -test. Qualitative data, obtained from Year 2 students ( n  = 125) by written questionnaire and Year 6 students ( n  = 11) and SPs ( n  = 3) by focus group interviews, were analysed for themes.
Results  Students receiving PAL did at least as well in the clinical skills examination as students with qualified tutors (difference in mean total score: 0.7 marks out of 112; 95% confidence interval − 3.8 to 2.4). The PAL environment was perceived as 'comfortable' and fostered the development of confidence in all participants. Peer tutors created a more active learning environment than doctor tutors for both learners and SPs and reported personal benefits from teaching.
Conclusions  With appropriate support, volunteer Year 6 student tutors are as effective as graduate doctors for small-group structured tutorials in clinical skills. Educational relationships were forged between all participants in the learning triad.     

The rating of pre-clerkship examination questions by postgraduate medical students: an assessment of quality and relevancy to medical practice

OBJECTIVE: To determine whether postgraduate students are able to assess the quality of undergraduate medical examinations and to establish whether faculty can use their results to troubleshoot the curriculum in terms of its content and evaluation. SUBJECTS: First and second year family medicine postgraduate students. MATERIALS: A randomly generated sample of undergraduate medical examination questions. METHODS: Postgraduate students were given two undergraduate examinations which included questions with an item difficulty (ID) > 0.60. The students answered and then rated each question on a scale of 1-7. RESULTS: The percentage of postgraduate students answering each question correctly correlated significantly with the average perceived relevance (Examination 1: r=0.372; P < 0.05; Examination 2: r=0.458; P < 0.05). Questions plotted for average postgraduate/undergraduate performance ratio versus the average perceived relevance were significantly correlated (Examination 1: r=0.462; P < 0.01; Examination 2: r=0.458; P < 0.05). CONCLUSIONS: This study offers a method of validating question appropriateness prior to examination administration. The design has the potential to be used as a model for determining the relevancy of a medical curriculum.     

An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation

PURPOSE: This investigation aimed to explore the measurement properties of scores from a patient simulator exercise. METHODS: Analytic and holistic scores were obtained for groups of medical students and residents. Item analysis techniques were used to explore the nature of specific examinee actions. Interrater reliability was calculated. Scores were contrasted for third year medical students, fourth year medical students and emergency department residents. RESULTS: Interrater reliabilities for analytic and holistic scores were 0.92 and 0.81, respectively. Based on item analysis, proper timing and sequencing of actions discriminated between low- and high-ability examinees. In general, examinees with more advanced training obtained higher scores on the simulation exercise. CONCLUSION: Reliable and valid measures of clinical performance can be obtained from a trauma simulation provided that care is taken in the development and scoring of the scenario.     

Assessment of patient management skills and clinical skills of practising doctors using computer-based case simulations and standardised patients

CONTEXT: Standardised assessments of practising doctors are receiving growing support, but theoretical and logistical issues pose serious obstacles. OBJECTIVES: To obtain reference performance levels from experienced doctors on computer-based case simulation (CCS) and standardised patient-based (SP) methods, and to evaluate the utility of these methods in diagnostic assessment. SETTING AND PARTICIPANTS: The study was carried out at a military tertiary care facility and involved 54 residents and credentialed staff from the emergency medicine, general surgery and internal medicine departments. MAIN OUTCOME MEASURES: Doctors completed 8 CCS and 8 SP cases targeted at doctors entering the profession. Standardised patient performances were compared to archived Year 4 medical student data. RESULTS: While staff doctors and residents performed well on both CCS and SP cases, a wide range of scores was exhibited on all cases. There were no significant differences between the scores of participants from differing specialties or of varying experience. Among participants who completed both CCS and SP testing (n = 44), a moderate positive correlation between CCS and SP checklist scores was observed. There was a negative correlation between doctor experience and SP checklist scores. Whereas the time students spent with SPs varied little with clinical task, doctors appeared to spend more time on communication/counselling cases than on cases involving acute/chronic medical problems. CONCLUSION: Computer-based case simulations and standardised patient-based assessments may be useful as part of a multimodal programme to evaluate practising doctors. Additional study is needed on SP standard setting and scoring methods. Establishing empirical likelihoods for a range of performances on assessments of this character should receive priority.     

The effect of in-training assessment on clinical confidence in postgraduate education

INTRODUCTION: The literature on how in-training assessment (ITA) works in practice and what educational outcomes can actually be achieved is limited. One of the aims of introducing ITA is to increase trainees' clinical confidence; this relies on the assumption that assessment drives learning through its content, format and programming. The aim of this study was to investigate the effect of introducing a structured ITA programme on junior doctors' clinical confidence. The programme was aimed at first year trainees in anaesthesiology. METHODS: The study involved a nationwide survey of junior doctors' self-confidence in clinical performance before (in 2001) and 2 years after (in 2003) the introduction of an ITA programme. Respondents indicated confidence on a 155-item questionnaire related to performance of clinical skills and tasks reflecting broad aspects of competence. A total of 23 of these items related to the ITA programme. RESULTS: The response rate was 377/531 (71%) in 2001 and 344/521 (66%) in 2003. There were no statistically significant differences in mean levels of confidence before and 2 years after the introduction of the ITA programme - neither in aspects that were related to the programme nor in those that were unrelated to the programme. DISCUSSION: This study demonstrates that the introduction of a structured ITA programme did not have any significant effect on trainees' mean level of confidence on a broad range of aspects of clinical competence. The importance of timeliness and rigorousness in the application of ITA is discussed.     

Factors in faculty evaluation of medical students' performance

CONTEXT: Faculty members often use global rating scales as a method of assessing various characteristics of medical students' clinical performance. The purpose of this study was to determine if some performance characteristics are more highly associated with the overall faculty grade than others. METHODS: The clinical performance of 211 surgery clerkship students was evaluated by 2 or 3 faculty preceptors. Faculty rated students on 10 specific performance characteristics, using a 5-point scale. Faculty then assigned a numerical grade summarising the faculty's view of the student's performance. Reliability of the ratings was estimated by the intraclass correlation, and 1-way (analysis of variance) anova was used to test for differences among the students' mean ratings. Logistic regression was employed to determine the accuracy of each performance measure in predicting students' grades (A or B). Stepwise logistic regression was used to determine if there was a combination of performance characteristics that best predicted students' grades. RESULTS: The inter-rater reliabilities were low (相似文献   

Responding to the implications of the genetics revolution for the education and training of doctors: a medical humanities approach

BACKGROUND: Rapid advances in the field of genetics continue to present medical educators with significant challenges. Whilst there is undoubtedly a pressing need to educate doctors about genetic disorders, research and therapies, there is a parallel need to provide a context for all of these. CONTENT: An interdisciplinary, arts and humanities based approach, responding to this need, is described. This teaching has been successfully delivered both as optional and core undergraduate teaching, and as part of continuing professional development. THE HUMAN PERSPECTIVE: STORIES NOT HISTORIES: Understanding of the patient's perspective can be significantly improved by drawing on both written and oral stories of illness. THE HISTORICAL PERSPECTIVE: LEARNING FROM THE PATIENT: Experiential learning provides insights into the social history of developments in genetics, thereby placing the current concern and debate about the new genetics in context. THE ROLE OF THE MEDIA: THE POWER TO PERSUADE: Critical reading skills can be developed and the power of the popular press to influence the reader acknowledged by analysing and employing the skills of the journalist when reporting developments in biotechnology. LEARNER ASSESSMENT AND EVALUATION: Assessment, both formative and summative, demonstrates sophisticated insights and perspectives into the lived experience of genetic illness. Learner evaluation of the teaching is high. CONCLUSION: Medical humanities offers a powerful way to convey an understanding of how genetic disorders impact on the lives of patients and families, and to set this against the background of a history rich in the uses, and abuses, of knowledge of heredity.     

Setting up a clinical skills learning facility

Objective  This paper outlines the considerations to be made when establishing a clinical skills learning facility.
Considerations  Establishing a clinical skills learning facility is a complex project with many possible options to be considered. A number of professional groups, undergraduate or postgraduate, may be users. Their collaboration can have benefits for funding, uses and promotion of interprofessional education. Best evidence and educational theory should underpin teaching and learning. The physical environment should be flexible to allow a range of clinical settings to be simulated and to facilitate a range of teaching and learning methods, supported by computing and audio-visual resources. Facilities should be available to encourage self-directed learning. The skills programme should be designed to support the intended learning outcomes and be integrated within the overall curriculum, including within the assessment strategy. Teaching staff may be configured in a number of ways and may be drawn from a variety of backgrounds. Appropriate staff development will be required to ensure consistency and quality of teaching with monitoring and evaluation to assure appropriate standards. Patients can also play a role, not only as passive teaching material, but also as teachers and assessors. Clinical, diagnostic and therapeutic equipment will be required, as will models and manikins. The latter will vary from simple part task trainers to highly sophisticated human patient simulators. Care must be taken when choosing equipment to ensure it matches specified requirements for teaching and learning.
Conclusion  Detailed planning is required across a number of domains when setting up a clinical skills learning facility.     

Using standardised students in faculty development workshops to improve clinical teaching skills

PURPOSE: We describe the use of standardised students (SSs) in interdisciplinary faculty development programmes to improve clinical teaching skills. Standardised students are actual health professions students who are trained to portray a prototypical teaching challenge consistently across many encounters with different faculty participants. METHODS: The faculty development programmes described focused on the skills of providing feedback and brief clinical teaching. At the beginning of each session, each participant was videotaped in encounters with 2 different SSs. Using microteaching (an instructional method in which learners view short segments of their own videotaped performance and discuss the tapes with a facilitator, consultant or other workshop participants), each group of participants and instructors reviewed the tapes and reflected on the encounters, providing immediate feedback to participants and modelling different approaches to the same teaching problem. The same process was repeated with more complicated scenarios after 2 weeks and again after 6 months offering reinforcement, further practice and more sophisticated development of the strategies learned. Participants completed post-session evaluations and a follow-up telephone survey. RESULTS: A total of 36 faculty members from the colleges of medicine, dentistry, pharmacy and nursing participated in workshops in 2000-01. The workshops were rated as highly relevant to participants' teaching, and most participants reported that they had learned a great deal. Participants most appreciated reviewing the videotaped interactions, the feedback they received, the interactions with their colleagues, the interdisciplinary nature of the groups and the practical focus of the workshops. CONCLUSIONS: Standardised students provide a high fidelity, low risk, simulated environment in which faculty can reflect on and experiment with new teaching behaviours. Such encounters can enhance the effectiveness and impact of faculty development programmes to improve clinical teaching skills.     

Predictive validity of a written knowledge test of skills for an OSCE in postgraduate training for general practice

PURPOSE: To examine the validity of a written knowledge test of skills for performance on an OSCE in postgraduate training for general practice. METHODS: A randomly-selected sample of 47 trainees in general practice took a knowledge test of skills, a general knowledge test and an OSCE. The OSCE included technical stations and stations including complete patient encounters. Each station was checklist rated and global rated. RESULTS: The knowledge test of skills was better correlated to the OSCE than the general knowledge test. Technical stations were better correlated to the knowledge test of skills than stations including complete patient encounters. For the technical stations the rating system had no influence on the correlation. For the stations including complete patient encounters the checklist rating correlated better to the knowledge test of skills than the global rating. CONCLUSION: The results of this study support the predictive validity of the knowledge test of skills. In postgraduate training for general practice a written knowledge test of skills can be used as an instrument to estimate the level of clinical skills, especially for group evaluation, such as in studies examining the efficacy of a training programme or as a screening instrument for deciding about courses to be offered. This estimation is more accurate when the content of the test matches the skills under study. However, written testing of skills cannot replace direct observation of performance of skills.     

Attitudes towards transferable skills in medical undergraduates

CONTEXT: Changes to the style of medical teaching will place a greater responsibility on individual medical students to manage their own learning, highlighting the need for students to develop good so-called 'transferable' skills at an early stage in their undergraduate career. OBJECTIVES: To assess the attitudes of first year undergraduates towards transferable skills, and investigate the gender difference in these attitudes. To assess the contribution of their first year course to skills development. SUBJECTS: First year students, enrolled on a traditional-style course. METHOD: A questionnaire asking the students to consider: (a) the importance of named transferable skills for medicine; (b) their own ability in these areas; and (c) the influence of their first year course. RESULTS: All students, irrespective of gender, regarded transferable skills as very important to medicine, rating organizational skills and self-learning skills as most important. Overall, students have a high level of confidence in their own skills. Male students rated their overall level of skills more highly than women. In particular they rated their information handling, managing self-learning and technical skills more highly. Students feel that their first year course has enhanced their skills in most areas. CONCLUSIONS: Our results suggest that students will feel equipped to succeed in a learning system which places the onus on them to take responsibility for their own learning. They clearly believe that they have the necessary skills for independent learning. The study highlights the need to enhance students' self-evaluation skills.     

Educational impact of in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice

Objectives To investigate the experiences and opinions of programme directors, clinical supervisors and trainees on an in‐training assessment (ITA) programme on a broad spectrum of competence for first year training in anaesthesiology. How does the programme work in practice and what are the benefits and barriers? What are the users' experiences and thoughts about its effect on training, teaching and learning? What are their attitudes towards this concept of assessment? Methods Semistructured interviews were conducted with programme directors, supervisors and trainees from 3 departments. Interviews were audiotaped and transcribed. The content of the interviews was analysed in a consensus process among the authors. Results The programme was of benefit in making goals and objectives clear, in structuring training, teaching and learning, and in monitoring progress and managing problem trainees. There was a generally positive attitude towards assessment. Trainees especially appreciated the coupling of theory with practice and, in general, the programme inspired an academic dialogue. Issues of uncertainty regarding standards of performance and conflict with service declined over time and experience with the programme, and departments tended to resolve practical problems through structured planning. Discussion Three interrelated factors appeared to influence the perceived value of assessment in postgraduate education: (1) the link between patient safety and individual practice when assessment is used as a licence to practise without supervision rather than as an end‐of‐training examination; (2) its benefits to educators and learners as an educational process rather than as merely a method of documenting competence, and (3) the attitude and rigour of assessment practice.     

The development of medical teachers: an enquiry into the learning histories of 10 experienced medical teachers

AIM: The aim of the study was to explore the different ways in which doctors have learned to teach and train. INTRODUCTION: There is no coherent theory of medical teacher development. Doctors are experts in what they teach; most have had little or no training in how they teach. Research has mostly concentrated on the acquisition and improvement of pedagogical skills by attendance at formal, generally short courses. These may have limited impact. METHODS: We carried out semistructured interviews with 10 experienced medical teachers. A review of the literature had suggested areas to explore. Interviews were transcribed and coded and thematic analysis and grounded theory used as the framework for qualitative analysis. RESULTS: Four areas were identified as important in teacher development: acquisition of educational knowledge and skills; modelling and practice of teaching skills; encouragement and motivation of teachers, and constraints on teaching and learning. DISCUSSION: The results suggest a model for teacher development that begins with doctors as learners, learning to learn and watching teachers teach. They then start to teach, acquiring and practising skills, and subsequently move on to reflect on their teaching. They can be encouraged to teach but may also be prevented from teaching. CONCLUSIONS: This inductive study proposes a model for medical teacher development that attempts to explain how doctors learn to teach and train. More research is needed to clarify the findings. There are implications for faculty development.