The clinical and educational experiences of pre-registration house officers in general practice

AIMS: To describe aspects of the clinical experience and educational supervision gained by pre-registration house officers (PRHOs) in general practice, and to relate these to the current General Medical Council (GMC) aims for general clinical training in general practice. DESIGN: Qualitative evaluation, part of which involved semistructured interviews with 12 PRHOs who were experiencing a general practice rotation. Interviews were conducted at the beginning and the end of the pre-registration year, and following return to hospital work after completion of the general practice placement. SETTINGS: Three teaching hospitals, two district general hospitals and six general practices in south-east England. PARTICIPANTS: 12 PRHOs who were involved in rotations incorporating a general practice placement. RESULTS: To varying degrees, the GMC aims for training in general practice were met for all the participants. All PRHOs recognized the value of the clinical experience and educational supervision they received in general practice. They particularly valued aspects such as having an individual training programme based on their own needs, and the interlinking of theory and practice, which aided learning. Most felt that having responsibility for their own patients acted as an important incentive for learning, and in general, PRHOs appreciated having the time to learn which general practice allowed. CONCLUSIONS: For the majority of PRHOs, the time spent in general practice was seen as a positive clinical and educational experience. In a variety of ways, the general practice placement encouraged PRHOs to develop the self-directed learning skills seen as essential to the lifelong learning advocated by the GMC. A number of recommendations are made to help improve the integration of the hospital and general practice components of these rotations.     

Defining characteristics of educational competencies

Context  Doctor competencies have become an increasing focus of medical education at all levels. However, confusion exists regarding what constitutes a competency versus a goal, objective or outcome.
Objectives  This article attempts to identify the characteristics that define a competency and proposes criteria that can be applied to distinguish between competencies, goals, objectives and outcomes.
Methods  We provide a brief overview of the history of competencies and compare competencies identified by international medical education organisations (CanMEDS 2005, Institute for International Medical Education, Dundee Outcome Model, Accreditation Council for Graduate Medical Education/American Board of Medical Specialties). Based upon this review and comparisons, as well as on definitions of competencies from the literature and theoretical and conceptual analyses of the underpinnings of competencies, the authors develop criteria that can serve to distinguish competencies from goals, objectives and outcomes.
Results  We propose 5 criteria which can be used to define a competency: it focuses on the performance of the end-product or goal-state of instruction; it reflects expectations that are external to the immediate instructional programme; it is expressible in terms of measurable behaviour; it uses a standard for judging competence that is not dependent upon the performance of other learners, and it informs learners, as well as other stakeholders, about what is expected of them.
Conclusions  Competency-based medical education is likely to be here for the foreseeable future. Whether or not these 5 criteria, or some variation of them, become the ultimate defining criteria for what constitutes a competency, they represent an essential step towards clearing the confusion that reigns.     

CanMEDS evaluation in Canadian postgraduate training programmes: tools used and programme director satisfaction

Context The Royal College of Physicians and Surgeons of Canada (RCPSC) CanMEDS framework is being incorporated into specialty education worldwide. However, the literature on how to evaluate trainees in the CanMEDS competencies remains sparse. Objectives The goals of this study were to examine the assessment tools used and programme directors’ perceptions of how well they evaluate performance of the CanMEDS roles in Canadian postgraduate training programmes. Methods We conducted a web‐based survey of programme directors of RCPSC‐accredited training programmes. The survey consisted of two questions. Question 1 was designed to establish which assessment tools were used to assess each of the CanMEDS roles. Question 2 was intended to assess programme directors’ perceived satisfaction with CanMEDS evaluation in their programmes. Results A total of 149 of the eligible 280 programme directors participated in the survey. Programme directors used a variety of assessment tools to evaluate trainees in CanMEDS competencies. Programmes used more tools to evaluate the Medical Expert (mean = 4.03, standard deviation [SD] = 1.59) and Communicator (mean = 2.36, SD = 1.02) roles. Programme directors used the fewest tools for the Collaborator (mean = 1.75, SD = 1.10) and Manager (mean = 1.75, SD = 1.18) roles. More than 92% of the programmes used in‐training evaluation reports to evaluate all the CanMEDS roles. Programme directors were satisfied with their evaluation of the Medical Expert role, but less so with assessment of the other CanMEDS competencies. Conclusions This study demonstrates that Canadian postgraduate training programmes use a variety of assessment tools to evaluate the CanMEDS competencies. Programme directors are neutral or concerned about how the CanMEDS roles other than that of Medical Expert are evaluated in their programmes. Further efforts are required to establish best practice in CanMEDS evaluation.     

A study of pre-registration house officers' clinical skills

BACKGROUND: Little is known about the ability of pre-registration house officers (PRHOs) to perform basic clinical skills just prior to entering the medical register. OBJECTIVES: To find out whether PRHOs have deficiencies in basic clinical skills and to determine if the PRHOs themselves or their consultants are aware of them. METHOD: All 40 PRHOs at the Chelsea and Westminster and Whittington Hospitals were invited to undertake a 17 station OSCE of basic clinical skills. Each station was marked by one examiner completing an overall global score after completing an itemised checklist. An adequate station performance was the acquisition of a pass/borderline pass grade. Prior to the OSCE, a questionnaire was given to each PRHO asking them to rate their own abilities (on a 5-point scale) in the skills tested. A similar questionnaire was sent to the educational supervisors of each PRHO asking them to rate their house officer's ability in each of the same skills. RESULTS: Twenty-two PRHOs participated. Each PRHO failed to perform adequately a mean of 2.4 OSCE stations (SD 1.8, range 1-8). There were no significant correlations between OSCE performance and either self- or educational supervisor ratings. The supervisor felt unable to give an opinion on PRHO abilities in 18% of the skills assessed. DISCUSSION: This study suggests that PRHOs may have deficiencies in basic clinical skills at the time they enter the medical register. Neither the PRHOs themselves nor their consultants identified these deficiencies. A large regional study with sufficient power is required to explore the generalizability of these concerns in more detail.     

The effect of a Structured Question Grid on the validity and perceived fairness of a medical long case assessment

PROBLEM: A perception that the reliability of our oral assessments of clinical competence was vitiated by lack of consistency in questioning. DESIGN: Parallel group controlled trial of a Structured Question Grid for use in clinical assessments. The Structured Question Grid required assessors to see the patient personally in advance of the student and to write down for each case the points they wished to examine. The Structured Question Grid limited assessors to two questions on each point, one designated a pass question and one at a higher level. Three basic science and three clinical reasoning issues were required, so that a total of 12 questions was allowed. SETTING: Small (70 students/year) undergraduate medical school with an integrated, problem-based curriculum. SUBJECTS: Sixty-seven students in the fourth year of a 5-year course were assessed, each seeing one patient and being examined by a pair of assessors. Assessor pairs were allocated to use the Structured Question Grid or to assess according to their usual practice. RESULTS: After the assessment but before being informed of the result the students completed a questionnaire on their experience and gave their performance a score between 0 and 100. The questions asked were based on focus group discussions with a previous student cohort, and concerned principally the perceived fairness and subjective validity of the assessment. The assessors independently completed a similar questionnaire, gave the student's performance a score between 0 and 100, and assigned an overall pass/fail grade. CONCLUSIONS: No difference was detected between students' or assessors' views of the fairness of the assessment for assessors who had used the Structured Question Grid compared to those who had not. Students whose assessors used the Structured Question Grid considered the assessment less representative of their ability. No difference was detected in the chance of students being assessed as failing or on the likelihood of a discrepancy between students' and assessors' ratings of students as passing or failing.     

The effect of the general practice registrar year on perceived skills in palliative care in the West Midlands

OBJECTIVES: To ascertain the effect of 12 months spent as a GP registrar on perceived skills in palliative care. DESIGN: A previously validated questionnaire for use with medical undergraduates is modified and used to survey perceived skills in five aspects of providing palliative care in five different scenarios at two points during the 12-month period of general practice vocational training where no specific teaching intervention is conducted. SETTING: The West Midlands. PARTICIPANTS: 210 GP registrars. RESULTS: Perceived skill ratings were seen to significantly increase during the 12-month period, but anxiety in caring for the dying did not significantly decrease. Ratings of skills were lowest when caring for a child dying with leukaemia or a young adult dying with AIDS. In addition, other important variables which had a statistically significant influence were gender and age, but interestingly not the number of previous senior house officer (SHO) posts undertaken or whether the respondent had had formal teaching on the subject in the past. CONCLUSION: It might therefore be postulated that training as a GP registrar has an important impact on the development of perceived skills in palliative care.     

Procedures for establishing defensible programmes for assessing practice performance

The assessment of the performance of doctors in practice is becoming more widely accepted. While there are many potential purposes for such assessments, sometimes the consequences of the assessments will be 'high stakes'. In these circumstances, any of the many elements of the assessment programme may potentially be challenged. These assessment programmes therefore need to be robust, fair and defensible, taken from the perspectives of consumer, assessee and assessor. In order to inform the design of defensible programmes for assessing practice performance, a group of education researchers at the 10th Cambridge Conference adopted a project management approach to designing practice performance assessment programmes. This paper describes issues to consider in the articulation of the purposes and outcomes of the assessment, planning the programme, the administrative processes involved, including communication and preparation of assessees. Examples of key questions to be answered are provided, but further work is needed to test validity.     

The impact of formal instruction in clinical examination skills on medical student performance -- the example of peripheral nervous system examination

BACKGROUND: Peripheral nervous system examination is an essential part of the full medical clerking of a patient. We have investigated the effectiveness of formal instruction in peripheral nervous system examination compared to the traditional bedside ward teaching that our students usually receive. METHOD: We instructed an unselected group of 22 medical students in peripheral nervous system examination in a clinical skills centre and evaluated them with a 12 item marking schedule before and after instruction. The performance of this group was then compared to the rest of their year (220 students) in an end of year OSCE, which included a neurology station assessing sensory examination of the lower limbs. RESULTS: Students formally instructed in neurology significantly improved their scores after instruction and scored 15% higher marks (90% vs. 75%) than the rest of their year in the end of year neurology OSCE station 2 months later (P < 0.01, Mann Whitney U-test). They did not perform significantly better in the OSCE overall. CONCLUSIONS: Formal instruction in neurological examination resulted in a significant increase in the end of year neurology OSCE station score compared to traditional heterogeneous teaching methods.     

Evaluating primary care as a base for medical education: the report of the Cambridge Community-based Clinical Course

BACKGROUND: Medical students receive increasing amounts of their clinical education in a primary care setting. The educational possibilities of such attachments are still being explored. AIM: To report the evaluation of a small, radical innovation to provide students with a long-term community attachment which integrated with hospital-based education. METHOD: Between 1993 and 1998, 13 medical students completed 15-month attachments with a single general practice in England. The course offered them experience in the major clinical specialties throughout this period. Students were exposed to specialist as well as generalist education in the context of patients with whom they could establish a continuing relationship. The innovation was evaluated by its feasibility, by students' examination results, by analysis of clinical experience, through formal student feedback and by cost. RESULTS: The course was practicable in a particular setting with academic leadership. The students all passed their exams. They had wide, appropriate clinical experience even though the attachment was to a single practice. When they returned to the hospital environment, students did not feel themselves at a disadvantage compared with traditional students. The costs of the course are controversial: placement costs were higher than in the hospital, but those for facilities were lower. CONCLUSION: It is possible to run a course like this successfully. It remains the most radical attempt to share clinical education in the UK between primary and secondary/tertiary care. Further research is required into providing long-term clinical attachments in NHS primary care settings.     

Monitoring standards of training

In postgraduate medical education there will always be a tension between delivering the service and ensuring time for reflection and learning. The balance requires monitoring to ensure national implementation of standards and to provide an external review for hospitals. However, the current system in the United Kingdom of visiting by many different agencies is disruptive to hospitals and wasteful of resources. The North Western Deanery has worked with medical royal colleges to develop a co-ordinated system of visiting and reporting which is designed to share information and expertise while reducing the pressure on hospitals. SETTING AND MAINTAINING STANDARDS: Clear standards are published and hospitals are encouraged to measure their own performance against these so that they are able to address identified problems. The effectiveness of the visits depends on collection and collation of data, especially face-to-face interviews with trainees and consultants. At the end of the visit a structured verbal feedback is given to senior managers and clinicians in the hospital and is followed by a written report. IMPROVING THE QUALITY OF TRAINING: Systematic review of training placements by the postgraduate dean's team has resulted in a steady increase in the quality of training placements and a structured approach has given trusts a framework to review the quality of training between visits. The future must lie in greater investment of responsibility in postgraduate deans for routine monitoring of training placements to consistent standards, which would allow the newly emerging Medical Education Standards Board (MESB) to review deaneries rather than individual placements.     

Clarifying the concepts of confidence and competence to produce appropriate self-evaluation measurement scales

INTRODUCTION: This paper reviews the literature on self-evaluation and discusses the findings of a small-scale qualitative study which explored the terms 'confidence' and 'competence' as useful measures in a self-evaluation scale. Four pre-registration house officers took part in interviews and completed a provisional instrument to assess their perceived competence. FINDINGS: Competence and confidence are useful terms for house officers expressing beliefs about their ability to perform their job but the terms should not be used synonymously. In our study, 'competent' represented what individuals knew about their ability and was based on the individual's previous experience of the task. 'Confident' described a judgement which influenced whether an individual was willing or not to undertake an activity. Confidence was not necessarily based on known levels of competence and therefore performance of tasks which were unfamiliar to the house officer also involved the assessment of risk. The authors give examples of task and skill scales which may be useful in the process of self-evaluation by pre-registration house officers. CONCLUSIONS: The authors suggest that the process of assessing oneself is complicated, and by its very nature can never be objective or free from the beliefs and values individuals hold about themselves. Therefore self-evaluation instruments are best used to help individuals analyse their work practices and to promote reflection on performance. They should not be used to judge the 'accuracy' of the individual's evaluation.     

Development of a three-centre simultaneous objective structured clinical examination

OBJECTIVE: To describe the development, organization, implementation and evaluation of a yearly multicentre, identical and simultaneous objective structured clinical examination (OSCE). SUBJECTS: All fifth-year medical students in a 6-year undergraduate medical programme. SETTING: The Christchurch, Dunedin and Wellington Schools of Medicine of the University of Otago, New Zealand. METHOD: One practice and two full 18-station OSCEs have been completed over 2 years, for up to 72 students per centre, in three centres. The process of development and logistics is described. Data are presented on validity, reliability and fairness. RESULTS: Face and content validity were established. Internal consistency was 0.83-0. 86 and interexaminer reliability, as assessed by the coefficient of correlation, averaged 0.78. Students rated the OSCE highly on relevance. Of the total variance in total OSCE marks, the schools contributed 6.9%, and the students 93.1%, in the first year. In the second year the schools contributed 6.2% and the students 93.8%. CONCLUSION: Implementation of a psychometrically sound, multicentre, simultaneous and identical OSCE is possible with a low level of interschool variation.     

Medical students proposing questions for their own written final examination: evaluation of an educational project

BACKGROUND: In order to emphasise learning more than control, from autumn 2000 we have invited medical students to propose questions for their own written examination in family medicine. One out of three student's proposals was guaranteed to be a part of their coming written examination, possibly somewhat modified. AIM: To evaluate how sixth year medical students experienced the project, and to what extent their performance in the examination was influenced. PARTICIPANTS: Sixth year medical students. MAIN OUTCOME MEASURES: The project was evaluated using (i) marks in examination; (ii) scores on self-administered questionnaires; and (iii) students' free text evaluation. RESULTS: Fifty-seven of 64 (89%) students taking their examination in autumn 2000, and 56 of 59 (95%) students taking the exam in spring 2001, responded. In autumn 2000, 34 (60%) students reported that the project had changed their learning strategies. During spring 2001, 46 of 56 students participated in producing questions, using a mean of 2.6 hours on the work. Students got 5-7% higher marks on their own questions on a scale ranging from 1 to 12. The students' free text evaluation showed that they had prepared especially thoroughly for the topics proposed by the students. They found it comforting to know at least one of the questions in the examination, and the students' questions were found relevant for general practice. CONCLUSION: Encouraging students to write questions for their own examination makes them feel more confident during the examination period, and may increase their reflective learning, without seriously limiting topics studied or violating the control function of the examination.     

A system for maintaining the educational and training standards of junior doctors

The development of junior doctors' competence is complex because the hospital environment in which doctors work places many demands on them. The need for quality education and training and personal development may be in direct conflict with the service commitments required from hospitals. This paper describes the methods by which the Postgraduate Medical Council of New South Wales, Australia, addresses the needs of junior doctors in the state in order to improve the quality of their education. Key elements of the Council's function include the provision of hospital clinical supervisors who oversee junior doctor education and training, and central involvement in supplying the junior doctor workforce to all state hospitals who must meet defined accreditation standards. This paper also provides data on evaluation of those methods and some educational outcomes.     

Learning in primary care--a report

A symposium on Learning in Primary Care was held in Cape Town, South Africa, as a pre-conference workshop to the 9th International Ottawa Conference on Medical Education. The aim of this report is to inform medical educationalists of important issues in learning in primary care and to stimulate further debate. Four international speakers gave presentations on their experiences in teaching and learning in primary care. Objective positive outcome measures include acquiring clinical skills equally well in general practice as in hospital, and improved history taking, physical examination and communication skills learning. Students regard the course as an essential requirement for learning and are appreciative of the wider aspect to learning provided by the community, giving a more holistic view of health. A SWOT analysis (strengths, weaknesses, opportunities and threats) of teaching and learning in primary care identified that learning in primary care is of a generalist nature and reality based, but is hampered by a lack of resources. The increased professionalization of teaching in primary care results in better training, cost containment, and improved quality of health care at community level. It is important to focus on turning threats into opportunities. Academic credibility needs to be established by conducting research on learning in primary care and developing the conceptual basis of primary care.     

Promoting effective teaching and learning: hospital consultants identify their needs

OBJECTIVES: The aim of this study was to help hospital consultants identify their needs in relation to teaching skills, leading to the development of a teacher training programme. DESIGN: The study was directed at all 869 consultants in the region and initially involved a postal questionnaire which had a 60.5% response rate. SETTING: Hospitals throughout Northern Ireland. SUBJECTS: Hospital consultants. RESULTS: Results from this questionnaire indicated that while the majority of respondents were interested teachers, only 34% had received any teacher training. The questionnaire was followed by a focus group study involving three groups of consultants drawn randomly from those who had responded to the questionnaire. Participants in these groups identified the following key areas of hospital education: qualities of hospital teachers; selection procedures; problems of teaching in hospitals; the need for teacher training and how it should be provided. CONCLUSION: The study highlighted that hospital teachers need to acquire and update their teaching skills through attending courses that should include basic teaching and assessment/appraisal skills. These courses should last 1 or 2 days and be provided at a regional or subregional level. As a result of this study, teacher training courses have been developed in this region.     

Standardized or real patients to test clinical competence? The long case revisited

BACKGROUND: In undergraduate clinical examinations, the use of real patients as long cases is being replaced by objective structured clinical examinations (OSCEs) which use simulated scenarios, although we lack published psychometric data on long cases to support the move from real to simulated patients. AIM: To assess candidate performance across two history-taking long cases to estimate the number of cases required for a reliable assessment. Results are compared with psychometric data from an OSCE. SETTING: A final-year qualifying undergraduate clinical examination. METHOD: Two observed history-taking long cases were included, alongside an OSCE. Candidates interviewed two unstandardized real patients. The history-taking part (14 minutes) was observed, uninterrupted, by examiner(s) who assessed data gathering, interviewing, and diagnostic and management skills. The presentation (7 minutes) was unstructured; the examiner(s) intervened as appropriate. Marks were expressed as a percentage of the total possible score and analysed using generalizability theory to estimate intercase reliability. RESULTS: Two examiner pairs independently rated both long cases for 79 (36.7%) of the 214 candidates. Projections based on generalizability theory showed that 10 20-minute cases would give reliabilities of 0.84 for single-marked and 0.88 for double-marked candidates, compared with a projected reliability of 0.73 for the same 214 candidates taking the OSCE. CONCLUSION: If history-taking long cases are observed, three-and-a-half hours of testing time using 10 unstandardized patients would produce a reliable test. Long cases therefore are, in terms of reliability, no worse and no better than OSCEs in assessing clinical competence.     

Reliability of the MRCP(UK) Part I Examination, 1984-2001

OBJECTIVES: To assess the reliability of the MRCP(UK) Part I Examination over the period 1984-2001, and to assess how the reliability is related to the difficulty of the examination (mean mark) and to the spread of the candidates' marks (standard deviation). METHODS: Retrospective analysis of the reliability (KR20) of the MRCP(UK) examination recorded in examination records for the 54 diets between 1984 and 2001. RESULTS: The reliability of the examination showed a mean value of 0.865 (SD 0.018, range 0.83-0.89). There were fluctuations in the reliability over time, and multiple regression showed that reliability was higher when the mean mark was relatively high, and when the standard deviation of the marks was high. CONCLUSIONS: The reliability of the MRCP(UK) Examination was maintained over the period 1984-2001. As theory predicted, the reliability was related to the average mark and to the spread of marks.     

Medical students' errors in pharmacotherapeutics

OBJECTIVES: This study analysed the errors made by 16 final-year medical students in a classroom prescribing exercise. The aim was to gain greater understanding of the reasons for non-optimal prescribing and of how to improve basic training in pharmacotherapeutics. METHODS: The task was to adjust a patient's phenytoin sodium dosage to achieve better control of seizures. It was based on a real-life case, and was presented as a written exercise. Process-tracing and think-aloud techniques were used to study the students' performance. RESULTS: The results suggest that the root cause of the errors was lack of a knowledge base which integrated scientific knowledge with clinical know-how. Three different clinical reasoning strategies were observed. Students who followed an incremental strategy demonstrated superior scientific knowledge and this resulted in less hazardous errors. Those who followed gambling or backward-reasoning strategies appeared to possess inferior scientific knowledge and this resulted in more hazardous errors. CONCLUSIONS: The results support current trends towards integrating basic medical science into a foundation of clinical know-how, as in the problem-based curriculum. They also emphasize the importance of a thorough grounding in medical science as a means of minimizing error.