The accountability of clinical education: its definition and assessment

BACKGROUND: Medical education is not exempt from increasing societal expectations of accountability. Competition for financial resources requires medical educators to demonstrate cost-effective educational practice; health care practitioners, the products of medical education programmes, must meet increasing standards of professionalism; the culture of evidence-based medicine demands an evaluation of the effect educational programmes have on health care and service delivery. Educators cannot demonstrate that graduates possess the required attributes, or that their programmes have the desired impact on health care without appropriate assessment tools and measures of outcome. OBJECTIVE: To determine to what extent currently available assessment approaches can measure potentially relevant medical education outcomes addressing practitioner performance, health care delivery and population health, in order to highlight areas in need of research and development. METHODS: Illustrative publications about desirable professional behaviour were synthesized to obtain examples of required competencies and health outcomes. A MEDLINE search for available assessment tools and measures of health outcome was performed. RESULTS: There are extensive tools for assessing clinical skills and knowledge. Some work has been done on the use of professional judgement for assessing professional behaviours; scholarship; and multiprofessional team working; but much more is needed. Very little literature exists on assessing group attributes of professionals, such as clinical governance, evidence-based practice and workforce allocation, and even less on examining individual patient or population health indices. CONCLUSIONS: The challenge facing medical educators is to develop new tools, many of which will rely on professional judgement, for assessing these broader competencies and outcomes.     

Clinical teaching: maintaining an educational role for doctors in the new health care environment

CONTEXT AND OBJECTIVES: Good clinical teaching is central to medical education but there is concern about maintaining this in contemporary, pressured health care environments. This paper aims to demonstrate that good clinical practice is at the heart of good clinical teaching. METHODS: Seven roles are used as a framework for analysing good clinical teaching. The roles are medical expert, communicator, collaborator, manager, advocate, scholar and professional. RESULTS: The analysis of clinical teaching and clinical practice demonstrates that they are closely linked. As experts, clinical teachers are involved in research, information retrieval and sharing of knowledge or teaching. Good communication with trainees, patients and colleagues defines teaching excellence. Clinicians can 'teach' collaboration by acting as role models and by encouraging learners to understand the responsibilities of other health professionals. As managers, clinicians can apply their skills to the effective management of learning resources. Similarly skills as advocates at the individual, community and population level can be passed on in educational encounters. The clinicians' responsibilities as scholars are most readily applied to teaching activities. Clinicians have clear roles in taking scholarly approaches to their practice and demonstrating them to others. CONCLUSION: Good clinical teaching is concerned with providing role models for good practice, making good practice visible and explaining it to trainees. This is the very basis of clinicians as professionals, the seventh role, and should be the foundation for the further development of clinicians as excellent clinical teachers.     

A review of the evaluation of clinical teaching: new perspectives and challenges

PURPOSE: This article discusses the importance of the process of evaluation of clinical teaching for the individual teacher and for the programme. Measurement principles, including validity, reliability, efficiency and feasibility, and methods to evaluate clinical teaching are reviewed. CONTEXT: Evaluation is usually carried out from the perspective of the learner. This article broadens the evaluation to include the perspectives of the teacher, the patient and the institutional administrators and payers in the health care system and recommends evaluation strategies. RESULTS: Each perspective provides specific feedback on factors or attributes of the clinical teacher's performance in the domains of medical expert, professional, scholar, communicator, collaborator, patient advocate and manager. Teachers should be evaluated in all domains relevant to their teaching objectives; these include knowledge, clinical competence, teaching effectiveness and professional attributes. CONCLUSIONS AND IMPLICATIONS: Using this model of evaluation, a connection can be made between teaching and learning about all the expected roles of a physician. This can form the basis for systematic investigation into the relationship between the quality of teaching and the desired outcomes, the improvement of student learning and the achievement of better health care practice. It is suggested that the extent of effort and resources devoted to evaluation should be commensurate with the value assigned to the evaluation process and its outcomes.     

Linking assessment to learning: a new route to quality assurance in medical practice

BACKGROUND: If continuing professional development is to work and be sensible, an understanding of clinical practice is needed, based on the daily experiences of doctors within the multiple factors that determine the nature and quality of practice. Moreover, there must be a way to link performance and assessment to ensure that ongoing learning and continuing competence are, in reality, connected. Current understanding of learning no longer holds that a doctor enters practice thoroughly trained with a lifetime's storehouse of knowledge. Rather a doctor's ongoing learning is a 'journey' across a practice lifetime, which involves the doctor as a person, interacting with their patients, other health professionals and the larger societal and community issues. OBJECTIVES: In this paper, we describe a model of learning and practice that proposes how change occurs, and how assessment links practice performance and learning. We describe how doctors define desired performance, compare actual with desired performance, define educational need and initiate educational action. METHOD: To illustrate the model, we describe how doctor performance varies over time for any one condition, and across conditions. We discuss how doctors perceive and respond to these variations in their performance. The model is also used to illustrate different formative and summative approaches to assessment, and to highlight the aspects of performance these can assess. CONCLUSIONS: We conclude by exploring the implications of this model for integrated medical services, highlighting the actions and directions that would be required of doctors, medical and professional organisations, universities and other continuing education providers, credentialling bodies and governments.     

Quality assessment in general practice trainers

INTRODUCTION: General practice trainers hold a key position in general practice training, especially through their provision of a role model. Their own competence in general practice care is important in this regard. The purpose of the study was to evaluate whether a quality assessment programme could identify the strengths and weaknesses of GP trainers in four main domains of general practice care. METHODS: The quality assessment programme comprised validated tests on four domains of general practice: general medical knowledge, knowledge of medical-technical skills, consultation skills and practice management. The criterion for the identification of relative strengths and weaknesses of GP trainers was a variation in the scores of trainers indicating higher and lower scores (strengths and weaknesses) within each domain. RESULTS: GP trainers (n=105) were invited to participate in the study and 90% (n=94) did so. The variation in scores allowed the indication of strengths and weaknesses. Main strengths were: general medical knowledge of the digestive system; knowledge of medical skills relating to the skin; consultation skills concerning empathy; practice management with regard to accessibility. Main weaknesses were: general medical knowledge of the neurological system; knowledge of the medical/technical skills relating to the endocrine metabolic and nutritional system; consultation skills regarding shared decision making; practice management involving cooperation with staff and other care providers. DISCUSSION: This first systematic evaluation of GP trainers identified their strengths and weaknesses. The weaknesses identified will be used in the improvement process as topics for collective improvement in the GP trainers' general curriculum and in individual learning plans.     

Concept mapping assessment in medical education: a comparison of two scoring systems

BACKGROUND: Concept mapping has the potential to measure important aspects of a student's evolving knowledge framework in a way that conventional examinations cannot. This is important because development of an elaborate and well-structured knowledge framework is a critical step toward becoming an expert in a particular field. Little is known about the best way to score concept maps in the setting of medical education. Therefore, as a preliminary step in addressing this question, we compared two different scoring systems for validity: a structural method based on the organization of a map's hierarchical structure and a relational method based, not on structure, but on the quality of each individual map component. METHODS: A total of 21 paediatric resident doctors completed concept map training, drew a preinstruction concept map about "seizures", completed a seizure education course, and then drew a postinstruction seizure map. Two raters using both structural and relational methods scored each map. RESULTS: Structural scores increased significantly after instruction and were higher in more experienced residents, but relational scores were not significantly different. Interrater scoring reliability for both methods ranged from moderate to strong, but was greater using the relational scoring method. CONCLUSIONS: These data suggest that scoring systems for evaluating concept maps in postgraduate medical education may need to account for structural features of maps, if scores are to reflect changes in the developing knowledge frameworks of resident doctors. More research to further evaluate reliability and validity is critical prior to any future use of concept mapping assessment in medical education.     

Video and qualitative research: analysing medical practice and interaction

CONTEXT: Video has long been recognised as providing an important resource within medical education, particularly, perhaps, for training in primary health care. As a resource for research, and more specifically within qualitative social science studies of medical practice, video has proved less pervasive, despite its obvious advantages. METHODS: In this paper, we sketch an approach to using video to inform the analysis of medical practice and the ways in which health care is accomplished through social interaction and collaboration. Drawing on our own research, we discuss two brief examples: the first involves the use of computing technology in primary health care and the second concerns informal instruction during surgery. The examples illustrate the multimodal character of medical work, how activities are accomplished through the interplay of talk, the visual and the use of material artefacts. They also illustrate the ways in which video provides access to the complex forms of social interaction and collaboration that underpin health care. DISCUSSION: We reflect upon the research opportunities afforded by video and the ways in which video-based studies of interaction can contribute to the practice and practicalities of medicine.     

Assessing student reflection in medical practice. The development of an observer-rated instrument: reliability, validity and initial experiences

INTRODUCTION: This study describes the development of an instrument to measure the ability of medical students to reflect on their performance in medical practice. METHODS: A total of 195 Year 4 medical students attending a 9-hour clinical ethics course filled in a semi-structured questionnaire consisting of reflection-evoking case vignettes. Two independent raters scored their answers. Respondents were scored on a 10-point scale for overall reflection score and on a scale of 0-2 for the extent to which they mentioned a series of perspectives in their reflections. We analysed the distribution of scores, the internal validity and the effect of being pre-tested with an alternate form of the test on the scores. The relationships between overall reflection score and perspective score, and between overall reflection score and gender, career preference and work experience were also calculated. RESULTS: The interrater reliability was sufficient. The range of scores on overall reflection was large (1-10), with a mean reflection score of 4.5-4.7 for each case vignette. This means that only 1 or 2 perspectives were mentioned, and hardly any weighing of perspectives took place. The values over the 2 measurements were comparable and were strongly related. Women had slightly higher scores than men, as had students with work experience in health care, and students considering general practice as a career. CONCLUSIONS: Reflection in medical practice can be measured using this semistructured questionnaire built on case vignettes. The mean score allows for the measurement of improvement by future educational efforts. The wide range of individual differences allows for comparisons between groups. The differences found between groups of students were as expected and support the validity of the instrument.     

" 'Just think of TB and Asians', that's all I ever hear": medical learners' views about training to work in an ethnically diverse society

The case for being able to respond effectively to cultural and ethnic diversity in health care is attracting increasing debate in medical education. However research exploring the perspectives of learners is lacking. AIMS: We sought medical learners' perceptions and their perceived training needs in relation to cultural and ethnic diversity in health care. METHODS: A series of nine focus group interviews was conducted with 55 medical learners, including undergraduate students in a UK medical school and a group of postgraduate general practitioners in training. Interview data were analysed using qualitative methods. RESULTS: Participants had a broad but superficial awareness of multicultural issues. This focused upon "difference" with students emphasizing their need to acquire knowledge of different beliefs and practices. Current teaching was perceived as inadequate and limited largely to ethnic patterns of disease. Most felt a need for greater training. They regarded development of particular communication skills, such as working with interpreters as helpful. Beyond avoiding stereotyping, learners rarely identified reflecting upon their attitudes or the issue of racism as important. Students anticipated a range of potential problems for further training, but sought learning that was relevant, practically oriented and stimulating. CONCLUSIONS: The study points to learners' experience of inadequate training but suggests a willingness to learn more. The possible predominance of a "difference" perspective might drive a narrow focus upon learning cultural knowledge at the expense of promoting a balance with self-reflection upon attitudes and developing generic skills. Educators might heed learners' views about how they should be taught successfully.     

Defining competency - the role of standard setting

CONTEXT: The responsibility to determine just who is competent to practice medicine, and at what standard, is great. Whilst there is still a period available for potential remediation, examinations at the completion of year three of the four-year Graduate Entry Medical Programme (GEMP) at Flinders University of South Australia (FUSA) are high stakes and contain the majority of final summative assessment for the certification of student to doctor. Therefore, the medical school has recently examined its methods for certification, the clinical practice standards sought in its programme and how to determine these standards. DESIGN: For all assessments a standard was documented and methods employed to set these standards using specific measures of performance. A modification of the Angoff method was applied to the written examination and the Rothman method, using two criteria, was used to determine competency in the objective structured clinical examination (OSCE). These methods were used for the first time in 1998. Both methods used trained 'experts' as standard setters and both methods used the notion of the 'borderline candidate' to determine the passing standard. This paper describes these two criterion-referenced standard-setting procedures as used in this school and related examination performance. CONCLUSIONS: Whilst the use of standard-setting procedures goes part way to defining and measuring competence, it is time consuming and requires significant examiner training and acceptance. Using 50% to determine who is and isn't competent is simpler but not transparent, fair nor defensible.     

Selecting performance assessment methods for experienced physicians

BACKGROUND: While much is now known about how to assess the competence of medical practitioners in a controlled environment, less is known about how to measure the performance in practice of experienced doctors working in their own environments. The performance of doctors depends increasingly on how well they function in teams and how well the health care system around them functions. METHODS: This paper reflects the combined experiences of a group of experienced education researchers and the results of literature searches on performance assessment methods. CONCLUSION: Measurement of competence is different to measurement of performance. Components of performance could be re-conceptualised within a different domain structure. Assessment methods may be of a different utility to that in competence assessment and, indeed, of different utility according to the purpose of the assessment. An exploration of the utility of potential performance assessment methods suggests significant gaps that indicate priority areas for research and development.     

Exploring UK medical and social work students’ legal literacy: comparisons,contrasts and implications

To ensure acceptable practice standards both doctors and social workers should draw on relevant legal rules when reaching professional judgements concerning, for instance, children requiring protection, people with severe mental distress and adults at risk, information sharing, consent to intervention and service user involvement in their care and treatment. Many practitioners use the law to maintain high standards of professionalism. However, research has uncovered limited awareness of legal rules and poor standards of health and social care. Academic benchmarks and practice requirements for health and social care professions centrally position legal knowledge for secure decision‐making. Model curricula exist. However, the outcomes of the taught curriculum on students’ confidence in their legal knowledge and skills have been relatively overlooked. This article introduces the concept of legal literacy, a distillation of knowledge, understanding, skills and values that enables practitioners to connect relevant legal rules with their professional practice, to appreciate the roles and duties of other practitioners and to communicate effectively across organisational boundaries. It presents the outcomes for a 2006–2009 study of 1154 UK medical and 638 social work students of their law learning for practice, response rates of 46% and 68%. Significant differences were found between medical and social work students’ attitudes towards the law, and in their self‐ratings of legal knowledge and skills. Confidence levels were low and anxiety high, especially among medical students, although law teaching had some positive outcomes on knowledge and skill development. Social work and medical students associated different themes with the law, the latter especially foregrounding ethics, negligence and liability, which could affect inter‐professional working. Students are not fully prepared for legally literate practice, with a consequent need to review the time allocated for, and the content of law learning and the subsequent availability of continuing professional development.     

Assessment in general practice: the predictive value of written-knowledge tests and a multiple-station examination for actual medical performance in daily practice

This study compares the predictive values of written-knowledge tests and a standardized multiple-station examination for the actual medical performance of general practitioners (GPs) in order to select effective assessment methods to be used in quality-improvement activities. A comprehensive assessment was performed in four phases. First, 100 GPs from the southern part of the Netherlands were assessed by a general medical knowledge test and by a knowledge test on technical skills. Second, in order to check for time-order effects, participants were randomly divided into two groups of 50 each, comparable on scores of both knowledge tests and on professional characteristics. Finally, both groups went through a multiple station examination using standardized patients and a practice video assessment of real surgery, but in opposite orders. Consultations were videotaped and assessed by well-trained peer observers. The drop-out rate was 10%. In both groups the predictive value of medical knowledge tests, ranging from 0.43 to 0.56 (Pearson correlation disattenuated), proved to be comparable with the predictive value of the multiple-station examination for actual performance (0.33-0.59). The overall explained variance of scores of the practice video assessment, measured by multiple regression analysis with performance scores as dependent variables and scores on the knowledge tests and the multiple-station examination as independent variables was moderate (19%). A time-order effect showed in only one direction: from practice video assessment to the multiple-station examination. The GP's professional characteristics did not contribute to the explanation of variation in performance. Medical knowledge tests can predict actual clinical performance to the same extent as a multiple-station examination. Compared with a station examination, a knowledge test may be a good alternative method for assessment the procedures of a large number of practising GPs.     

The current status and future needs of education and training in family medicine and primary care in South Africa

South Africa is undergoing tremendous political and social change affecting every sphere of society, including medical education and the delivery of health services. The legacy of its history created a health system that in some respects can be compared to the best in the world, but one also characterized by inequity, discrimination and lack of access to even basic services for the rural and the poor. Its medical education system trails behind modern trends such as problem-based learning, community-based education and the utilizing of general/family practitioners as trainers. Vocational training in family practice is not compulsory for independent practice. The discipline of family practice has nevertheless developed the programmes and core infrastructure for such a future undertaking in the form of masters programmes in family medicine at all medical schools. The recently introduced system of compulsory recertification through continuous professional development provides a window of opportunity to develop locally relevant curricula and appropriate education and training methods for family practitioners. Challenges for family practice include the establishment of the role and value of the discipline in a developing country with a health system based on a nurse-driven primary care service and the re-orientation of family medicine teachers, trained in a biomedical paradigm, to the patient-centred approach. The aspirations of family practice are to define the core content of the discipline, establish and nurture a culture of research in primary care, and to develop and introduce appropriate under and postgraduate training programmes for the new generation of family doctors.     

Theory in Practice: Why “Good Medicine” and “Scientific Medicine” are not Necessarily the Same Thing

The term “scientific medicine”, ubiquitous in medical literature although poorly defined, can be traced to a number of assumptions, three of which are examined in this paper: that medicine is a form of knowledge-driven practice, where the established body of proven medical knowledge determines what doctors do; if what doctors do is either inadequate or ineffective, the chief reason is the absolute or relative lack of adequate knowledge for providing care for patients; evaluating medical practice boils down to comparing it to a set of standards which should be univocally applied to concrete situations. This paper intends to provide at least a tentative assessment of how does this set of assumptions fare in the real world of clinical care. The methodology was based on direct observation of medical consultations and independent evaluation by referees of data from of medical records. The review of the data shows a far more contingent relationship between handbook prescribed procedures and actual medical practice, even though the referees evaluated the reported data mostly with favorable scores. Furthermore, a few problems were observed relating to the inadequacy of the so-called biomedical model in dealing with some of the more prevalent health problems. The authors conclude that, more than any “technical inadequacies”, it would seem that this study has underlined the limitations of the biomedical model in responding to the tasks it attributes to itself, an issue that has to be addressed more effectively by medical education, be it in terms of undergraduate or graduate schooling.     

The effect of the general practice registrar year on perceived skills in palliative care in the West Midlands

OBJECTIVES: To ascertain the effect of 12 months spent as a GP registrar on perceived skills in palliative care. DESIGN: A previously validated questionnaire for use with medical undergraduates is modified and used to survey perceived skills in five aspects of providing palliative care in five different scenarios at two points during the 12-month period of general practice vocational training where no specific teaching intervention is conducted. SETTING: The West Midlands. PARTICIPANTS: 210 GP registrars. RESULTS: Perceived skill ratings were seen to significantly increase during the 12-month period, but anxiety in caring for the dying did not significantly decrease. Ratings of skills were lowest when caring for a child dying with leukaemia or a young adult dying with AIDS. In addition, other important variables which had a statistically significant influence were gender and age, but interestingly not the number of previous senior house officer (SHO) posts undertaken or whether the respondent had had formal teaching on the subject in the past. CONCLUSION: It might therefore be postulated that training as a GP registrar has an important impact on the development of perceived skills in palliative care.     

Surgical skills training: simulation and multimedia combined

CONTEXT: Basic surgical skills are needed throughout the medical profession, but current training is haphazard and unpredictable. There is increasing pressure to provide transparency about training and performance standards. There is a clear need for inexperienced learners to build a framework of basic skills before carrying out surgical procedures on patients. Effective learning of a skill requires sustained deliberate practice within a cognitive framework, and simulation offers an opportunity for safe preparation. OBJECTIVES: This paper presents a new approach to basic surgical skills training, where tuition using a specially designed computer program is combined with structured practice using simulated tissue models. This approach to teaching has evolved from practical experience with surgical skills training in workshops. METHODS: Pilot studies with 72 first-year medical students highlighted the need for separate programs for teaching and for self-directed learning. The authors developed a training approach in the light of this experience. Subsequent in-depth observational and interview studies examined (a) individual teaching sessions between surgical teachers and learners (five consultant surgeons and five senior house officers) and (b) group teaching sessions with general practitioners (14 participants in three group interviews). Further work has resulted in a self-directed learning program. CONCLUSIONS: Qualitative analysis of observational and interview data provides strong preliminary support for the effectiveness of this approach. The response of teachers and learners was extremely positive. The combination of information (presented by computer) and practice of psychomotor skill (using simulated tissue models) could be extended to other surgical and practical skills.     

Evaluation of undergraduate medical education in Finnish community-oriented and traditional medical faculties: a 10-year follow-up

OBJECTIVES: This study focused on Finnish physicians' views of their undergraduate medical education. Differences between traditional and community-oriented medical faculties were examined and changes which had taken place during a 10-year follow-up period were also assessed. METHODS: The study was based on data retrieved from a postal survey made among Finnish physicians in 1998. The study population consisted of all doctors who graduated between 1987 and 1996 (n=4926); those born on odd-numbered days were selected for this study (n=2492). A postal questionnaire and two reminders were sent to those selected, and 1822 questionnaires were returned, giving a response rate of 73.1%. RESULTS: Physicians who graduated from the community-oriented faculties were more satisfied with their undergraduate medical education when compared with their colleagues graduating from traditional faculties. There were some differences between the universities with respect to education for hospital work. The teaching of primary health care, however, was clearly more effective in community-oriented faculties. The proportion of graduates who were satisfied with their primary care education was over 70% in community-oriented faculties, whereas in the traditional faculties it was only 35-45%. CONCLUSIONS: According to graduates, the community-oriented medical school curriculum better meets the needs of practising physicians than that in traditional faculties. In curriculum reforms, more emphasis should be placed on comprehensive medical education, which includes both primary and secondary health care.     

Simulation in surgical training: educational issues and practical implications

BACKGROUND: Surgical skills are required by a wide range of health care professionals. Tasks range from simple wound closure to highly complex diagnostic and therapeutic procedures. Technical expertise, although essential, is only one component of a complex picture. By emphasising the importance of knowledge and attitudes, this article aims to locate the acquisition of surgical skills within a wider educational framework. SIMULATORS: Simulators can provide safe, realistic learning environments for repeated practice, underpinned by feedback and objective metrics of performance. Using a simple classification of simulators into model-based, computer-based or hybrid, this paper summarises the current state of the art and describes recent technological developments. Advances in computing have led to the establishment of precision placement and simple manipulation simulators within health care education, while complex manipulation and integrated procedure simulators are still in the development phase. EVALUATION: Tension often exists between the design and evaluation of surgical simulations. A lack of high quality published data is compounded by the difficulties of conducting longitudinal studies in such a fast-moving field. The implications of this tension are discussed. THE WIDER CONTEXT: The emphasis is now shifting from the technology of simulation towards partnership with education and clinical practice. This highlights the need for an integrated learning framework, where knowledge can be acquired alongside technical skills and not in isolation from them. Recent work on situated learning underlines the potential for simulation to feed into and enrich everyday clinical practice.     

What anatomy shall we teach medical and dental students in a primary health care curriculum?

The opinions of 142 doctors on the relevance of anatomy to the diagnosis and management of common clinical problems in their current medical and dental practice were analysed. This was in a bid to determine the relevant anatomy course content for the new primary health care oriented medical and dental curriculum of the College of Medicine, University of Lagos. The respondents gave high scores to the relevance of anatomy knowledge to the management of acute abdomen (mean = 3.5), dislocated shoulder (3.3), Colles' fracture (3.2), palmar space abscess (3.2), obstructed labour (3.2), carcinoma of the breast (3.2), ectopic pregnancy (3.1), flail chest (3.1) and upper respiratory obstruction (3.0). They gave minimal scores to helminthiasis (mean = 1.5) common cold and anaemia (1.6), sickle cell disease (1.7), gastroenteritis (1.8), dental abscess (2.0), hypertension (2.2) and asthma (2.2). A basis for selecting relevant anatomy course content is deduced for an undergraduate curriculum in which the responsibilities and competence of the graduates is known. A nationwide extension of the study, especially amongst general practitioners and first-line doctors in rural areas, would be useful for identification of health problems that require little or no knowledge of anatomy and which can be safely managed by lower cadres of health personnel, traditional practitioners and members of the lay community.