Portfolio-based learning: continuing medical education for general practitioners — a mid-point evaluation

In October 1994 a project was initiated by the General Practice Continuing Medical Education Tutors in the Department of General Practice at Sheffield University. The project sought to evaluate the efficiency (effort expended) and effectiveness (distance travelled) of a model of continuing professional development for general practitioners through individual portfolio-based learning in co-mentoring groups. Learning demonstrated through the portfolio was accredited for the postgraduate education allowance of participants. This paper addresses the process of portfolio development at the mid-point of a year-long trial to ascertain the strengths, weaknesses and possible future development of such a process within the context of continuing medical education.     

Performance-based assessment in continuing medical education for general practitioners: construct validity

The use of performance-based assessment has been extended to postgraduate education and practising doctors, despite criticism of validity. While differences in expertise at this level are easily reflected in scores on a written test, these differences are relatively small on performance-based tests. However, scores on written tests and performance-based tests of clinical competence generally show moderate correlations. A study was designed to evaluate construct validity of a performance-based test for technical clinical skills in continuing medical education for general practitioners, and to explore the correlation between performance and knowledge of specific skills. A 1-day skills training was given to 71 general practitioners, covering four different technical clinical skills. The effect of the training on performance was measured with a performance-based test using a randomized controlled trial design, while the effect on knowledge was measured with a written test administered 1 month before and directly after the training. A training effect could be shown by the performance-based test for all four clinical skills. The written test also demonstrated a training effect for all but one skill. However, correlations between scores on the written test and on the performance based test were low for all skills. It is concluded that construct validity of a performance-based test for technical clinical skills of general practitioners was demonstrated, while the knowledge test score was shown to be a poor predictor of competence for specific technical skills.     

Evaluation of minor surgery courses for general practitioners

This paper describes the evaluation of a two-day minor surgery training course for general practitioners and vocational trainees. The course was developed by a team from a wide range of specialties. A pilot run of six of these courses was held in various locations in the United Kingdom during 1994, organized by the Royal College of General Practitioners and supported by the Department of Health. The courses were attended by 144 participants in total: 52 general practitioners and 92 vocational trainees. The evaluation addressed (a) the process of training which resulted in substantial changes being made to the original design and content of the courses; and (b) the outcome of the training in terms of knowledge, confidence and competence. A central feature of the course was the use of sophisticated simulated tissue. Participant feedback indicated that although the simulated tissue was not considered to be very realistic it substantially increased levels of confidence and competence to carry out minor surgery. Knowledge on various aspects of minor surgery increased significantly. The feedback from the evaluation was forwarded to a minor surgery working party at the Department of Health whose remit was to prepare guidelines for teaching, authorising and carrying out minor surgery in general practice.     

The market and educational principles in continuing medical education for general practice

In an attempt to increase participation by established general practitioners (GPs) in continuing medical education (CME) the Department of Health in 1990 created an allowance (the Postgraduate Education Allowance, PGEA) to be spent by GPs on the educational provision of their choice. Although the PGEA has increased attendance by established GPs at educational activities, the market created in CME provision has favoured low cost activity of questionable value. This paper examines criteria for the validation of general practice CME, based on the principles of adult learning, which could be used to manage this market. Failure to adopt an appropriate validating process could lead to the PGEA becoming discredited and the imposition of periodic reaccreditation as a means of enforcing participation by GPs in CME.     

A review of evaluation outcomes of web-based continuing medical education

INTRODUCTION: The Internet and worldwide web have expanded opportunities for the provision of a flexible, convenient and interactive form of continuing medical education (CME). Larger numbers of doctors are accessing and using the Internet to locate and seek medical information. It has been suggested that a significant proportion of this usage is directly related to questions that arise from patient care. A variety of Internet technologies are being used to provide both asynchronous and synchronous forms of web-based CME. Various models for designing and facilitating web-based CME learning have also been reported. The purpose of this study was to examine the nature and characteristics of the web-based CME evaluative outcomes reported in the peer-reviewed literature. METHODS: A search of Medline was undertaken and the level of evaluative outcomes reported was categorised using Kirkpatrick's model for levels of summative evaluation. RESULTS: The results of this analysis revealed that the majority of evaluative research on web-based CME is based on participant satisfaction data. There was limited research demonstrating performance change in clinical practices and there were no studies reported in the literature that demonstrated that web-based CME was effective in influencing patient or health outcomes. DISCUSSION: The findings suggest an important need to examine in greater detail the nature and characteristics of those web-based learning technologies, environments and systems which are most effective in enhancing practice change and ultimately impacting patient and health outcomes. This is particularly important as the Internet grows in popularity as a medium for knowledge transfer.     

Training of general practitioners in interpreting chest radiographs

Easily accessible chest and skeleton roentgen examinations are needed in primary health care. In Finland the health centres perform about a third of all roentgen examinations. Many of the films are interpreted by health centre doctors at least in the primary phase. About 40% of them are chest radiographs. In 2640 chest radiographs interpreted by general practitioners the final roentgen diagnosis was pathological in 54%. Pathological conditions were totally overlooked in 13%. Of normal cases 19% were interpreted falsely positive. The greatest need for general practitioners' training is in the common findings in the chest, for example cardiac insufficiency and inflammatory as well as atelectatic changes in the lungs. Pulmonary findings in children proved difficult to interpret and were proposed as a subject for training. The role of poor image quality in interpretation errors must be emphasized when less experienced doctors interpret chest films. If regular radiological consultation is not available, postgraduate training in diagnostics of chest radiographs should be considered in the training of specialists in general practice as well as in their supplementary training.     

The development of a participant questionnaire to assess continuing medical education presentations

OBJECTIVE: Feedback on presentation skills is important for developing skilled educators, but often this feedback is based on evaluation tools that have been developed with little concern for psychometric issues or for how the information will be used for feedback. The purpose of this study was to develop a reliable participant questionnaire to assess the quality of continuing medical education (CME) presentations and to provide presenters with feedback. DESIGN: The questionnaire was developed using an iterative approach, with doctors as raters, and tested during a variety of CME presentations. The resulting questionnaire consists of 9 items rated on a 7-point Likert scale. The psychometric analysis reported in this paper was completed using data from grand rounds presentations. RESULTS: Psychometric analysis, based on 319 evaluations from 17 presentations (average of 19 evaluations/presentation), revealed a high level of reliability (0.91), indicating that the items met a reasonable standard and that the raters were discriminating between the quality of the presentations adequately. CONCLUSION: This 9-item, participant questionnaire provides a reliable measure of the quality of CME presentations, while also providing presenters with useful feedback. Further studies will investigate if this instrument can be used to assess other CME formats and how best to provide feedback to presenters.     

The effects of continuing medical education on family doctor performance in office practice: a randomized control study

A randomized controlled study was conducted to determine if specifically designed continuing medical education in the fields of cardiovascular and cancer medicine could change doctor office behaviour significantly. Thirty-one volunteer family doctors from 25 offices participated. Six (three cardiovascular and three cancer) learning objectives were defined. Two educational formats were selected as the independent variables: (1) group interaction opportunities (face-to-face and teleconference); and (2) concisely written newsletters. Chart measures of doctor performance prior to and 6 and 12 months following education served as the dependent variables. The family doctors receiving education were found to perform the recommended behaviours significantly more than those who did not receive the education (P less than 0.05) at 6 months post-education. This difference was maintained at the 12-month post-educational period for one of the educational programmes offered. A carefully planned programme of continuing medical education will result in favourable changes in the office practice of volunteer doctors. These changes can persist for as long as 12 months. Adherence to several essential learning principles is required.     

An evaluation of a national scheme for continuing professional development (CPD) for career grade doctors: the Royal College of Paediatrics and Child Health's programme for paediatricians evaluated by focus group methodology

OBJECTIVES: To evaluate a national continuing professional development (CPD) scheme through the views and experiences of its participants. METHODS: A qualitative focus group methodology was used. Forty-nine career grade paediatricians were allocated to nine focus groups according to their work settings, ages and posts. All groups discussed the following: (a) ease or difficulty of achieving CPD requirements; (b) whether a personal professional development plan (PDP) enabled the planning process for CPD; (c) whether CPD participation facilitated changes in practice, and (d) their views on the CPD system in operation at the time of the study. RESULTS: Taking part in CPD had a positive effect on participants' clinical practice and broadened their professional and academic base. The main difficulties in achieving CPD were variable financial support (especially for non-consultant staff) and pressure due to extra work accumulating during study leave. The quality of internal CPD meetings was inconsistent. A professional development plan was useful for planning CPD needs, but there was a conflict between the educational and training needs of the individual and the employing NHS Trusts. The scheme's guidelines were clear but there were reservations about its layout and the system of recording points. CONCLUSION: This study enabled discovery of the views of paediatricians participating in a national CPD scheme operated by the Royal College of Paediatrics and Child Health. Consequently, changes were made to the scheme's presentation (but not content) and process of recording points. Barriers to achieving CPD requirements were identified and are being monitored. Efforts to involve non-consultant career grade paediatricians have been increased.     

Predictive validity of a written knowledge test of skills for an OSCE in postgraduate training for general practice

PURPOSE: To examine the validity of a written knowledge test of skills for performance on an OSCE in postgraduate training for general practice. METHODS: A randomly-selected sample of 47 trainees in general practice took a knowledge test of skills, a general knowledge test and an OSCE. The OSCE included technical stations and stations including complete patient encounters. Each station was checklist rated and global rated. RESULTS: The knowledge test of skills was better correlated to the OSCE than the general knowledge test. Technical stations were better correlated to the knowledge test of skills than stations including complete patient encounters. For the technical stations the rating system had no influence on the correlation. For the stations including complete patient encounters the checklist rating correlated better to the knowledge test of skills than the global rating. CONCLUSION: The results of this study support the predictive validity of the knowledge test of skills. In postgraduate training for general practice a written knowledge test of skills can be used as an instrument to estimate the level of clinical skills, especially for group evaluation, such as in studies examining the efficacy of a training programme or as a screening instrument for deciding about courses to be offered. This estimation is more accurate when the content of the test matches the skills under study. However, written testing of skills cannot replace direct observation of performance of skills.     

Continuing medical education and the education allowance: variation in credits obtained by general practitioners

The educational credits obtained by general practitioners for the postgraduate education allowance were examined. With over 3 months of the qualifying period remaining, 68.4% had reached the requirement. The educational day attainments show almost a normal distribution, with 4.2% of doctors completing more than double the requirement.     

Patient-management problems as a learning tool for the continuing medical education of general practitioners

The value of Patient-Management Problems (PMPs as a learning tool for continuing medical education was studied by using two frequently seen cardiovascular problems (angina and high blood pressure) and a province-wide sample of full-time general practitioners. The results indicate that PMPs can be a motivating and effective means of CME for the general practitioner; that knowledge was gained through the successive resolution of three pmps; that corrective feedback enhanced learning; and, that most of the knowledge gained on the paper cases was transferred to practice as reported by the participants on a questionnaire. Furthermore, while cueing may be a confounding factor when PMPs are used for evaluation purposes, it was shown to facilitate learning in the present learning context.     

Effect of a short skills training course on competence and performance in general practice

OBJECTIVES: Evaluation of the efficacy of a short course of technical clinical skills to change performance in general practice. DESIGN: Subjects were self-selected general practitioners (n=59) who were unaware of the study design. They were assigned to the intervention group (n=31) or control group (n=28) according to their preference for course date. The course covered four different technical clinical skills (shoulder injection technique, PAP-smear, laboratory examination of vaginal discharge, ophthalmoscopic control in diabetes mellitus). Main outcome measures used were pre- and post-training scores on a knowledge test of skills (60 multiple choice items), and pre- and post-training performance of procedures in practice using a log-diary covering 20 days. SETTING: University of Maastricht, The Netherlands. SUBJECTS: Self-selected general practitioners. RESULTS: Competence, as measured by the knowledge test of skills, improved significantly as a result of the training and skills test scores were satisfactory after training. A significant effect on performance in practice was found for PAP-smear and shoulder injection technique, whereas no effect could be demonstrated for examination of vaginal discharge and ophthalmoscopic control in diabetes mellitus. CONCLUSIONS: A good degree of competence is a necessary but not always sufficient condition for a physician to change his performance in practice. While some skills training seems adequate to bring about desired changes, for other skills more complex interventions are probably needed.     

Workplace-based assessment for general practitioners: using stakeholder perception to aid blueprinting of an assessment battery

CONTEXT: The implementation of an assessment system may be facilitated by stakeholder agreement that appropriate qualities are being tested. This study investigated the extent to which stakeholders perceived 8 assessment formats (multiple-choice questions, objective structured clinical examination, video, significant event analysis, criterion audit, multi-source feedback, case analysis and patient satisfaction questionnaire) as able to assess varying qualities of doctors training in UK general practice. METHODS: Educationalists, general practice trainers and registrars completed a blueprinting style of exercise to rate the extent to which each evaluation format was perceived to assess each of 8 competencies derived primarily from the General Medical Council document 'Good Medical Practice'. RESULTS: There were high levels of agreement among stakeholders regarding the perceived qualities tested by the proposed formats (G = 0.82-0.93). Differences were found in participants' perceptions of how well qualities were able to be assessed and in the ability of the respective formats to test each quality. Multi-source feedback (MSF) was expected to assess a wide range of qualities, whereas Probity, Health and Ability to work with colleagues were limited in terms of how well they could be tested by the proposed formats. DISCUSSION: Awareness of the perceptions of stakeholders should facilitate the development and implementation of workplace-based assessment (WPBA) systems. These data shed light on the acceptability of various formats in a way that will inform further investigation of WPBA formats' validity and feasibility, while also providing evidence on which to base educational efforts regarding the value of each format.     

Management of somatic presentations of psychiatric illness in general medical settings: evaluation of a new training course for general practitioners

An improved teaching package is described which aims to help general practice trainees manage somatized presentations of psychological distress. The package comprises a training videotape in which a reattribution model is demonstrated, with material for role-play of new skills and small-group video feedback of consultations. Eighteen general practice trainees attending an 8-week course in psychiatry participated fully in the somatization management teaching programme. The teaching package was evaluated by blind rating of general interview skills and model specific skills demonstrated by trainees during 10 to 15-minute clinical interviews with professional role-players. Ratings were made on pre-training and post-training videotaped interviews. A significant improvement was demonstrated in general interview skills. Improvements were also noted in specific reattribution skills post-training. The evaluation revealed that skills in the model can be effectively learned, and that improvements in the package have resulted in its improved efficacy.     

Professional development for general practitioners through fellowship by assessment

BACKGROUND AND STUDY AIMS: The Royal College of General Practitioners has designed its Fellowship by Assessment (FBA) programme with the twin goals of promoting higher standards of care and offering GPs a professional challenge. By November 2000, 223 eligible doctors (2%) had opted to take FBA. This number is increasing annually. There is, however, little research to account for why GPs undertake it, or what the barriers might be. The aim of our study was to investigate GPs' experience and thinking on this issue. METHODS: A total of 13 GPs who had attained Fellowship by Assessment in the Tamar and Severn Faculties were invited to participate in a qualitative study. They were identified from the list of RCGP Fellows in the Institute of General Practice, University of Exeter, and purposefully selected to include trainers and non-trainers, men and women, rural and urban practitioners, and single-handed and large practices. Analysis of in-depth interviews was informed by grounded theory. FINDINGS: FBA served to acknowledge high standards of care. There was some variation in views about FBA, however, ranging from a perspective which prized the professional award, to the view that endorsed it as a pragmatic and useful way of structuring and monitoring improvements. Fellowship by Assessment improves the care of patients and empowers the doctors by improving their confidence. It is, however, hugely time-consuming and while our findings suggest it should be encouraged, there is a need for ways to be found whereby doctors may be supported in this form of professional development.     

Helping tomorrow's doctors to gain a population health perspective - good news for community stakeholders

INTRODUCTION: The modernising agenda of the NHS and recommendations of professional bodies demand that all doctors achieve a basic understanding of a population health perspective. The principle of integrated learning and the logistics of provision make it inappropriate for such learning to be delivered solely by public health specialists, and community-based learning has been promoted as the best setting in which to assist this objective. However, there is little evidence from practice as to whether non-specialist staff are willing or able to play a role in orienting tomorrow's doctors to the needs of communities. METHODS: Semi-structured interviews, questionnaires and focus groups iterated the opinion of key stakeholders on their preferred contributions to community-oriented undergraduate medical education. Framework analysis was used to elicit key outcomes and process factors. RESULTS: There was consensus that community-based learning should be a core element of the medical curriculum as it can demonstrate the socioenvironmental context of care and the doctor's role in interagency working and preventive care. Effective academic/NHS partnerships were called for, with higher education leading the agenda on aims and objectives, creating collaborative structures, and reallocating resources to support new learning. Community-based stakeholders would offer the translation of theory into practice by demonstrating clinical and social diversity, models of teamworking, and a context for the application of prior learning. They also promoted recurrent contact with communities, and suggested an enhanced role in mentoring students through longer term relationships. CONCLUSION: Community-based personnel and NHS users are an under-used resource for medical education. Their proposed contributions strongly accord with the agenda for a population health perspective in basic training. Curriculum planners need to make long-term partnerships with community-based agencies, rather than using them as an intermittent provider of limited learning sessions with narrowly defined objectives.     

Problem-based learning in distance education: a first exploration in continuing medical education

The Wellcome Tropical Institute has assisted countries in the tropics to establish viable systems of continuing medical education, particularly for young doctors practising in rural areas. As part of this strategy the Institute has developed material for use in distance learning. The first attempt to apply the problem-based learning approach to written material for use by an individual learner in the absence of a tutor led to a trial in Ghana, Kenya and Pakistan to compare a conventionally designed module with a problem-based learning module on the same topic for their respective acceptability, effectiveness and efficiency. The design, implementation and results of these three comparative trials are presented.     

Assessment in general practice: the predictive value of written-knowledge tests and a multiple-station examination for actual medical performance in daily practice

This study compares the predictive values of written-knowledge tests and a standardized multiple-station examination for the actual medical performance of general practitioners (GPs) in order to select effective assessment methods to be used in quality-improvement activities. A comprehensive assessment was performed in four phases. First, 100 GPs from the southern part of the Netherlands were assessed by a general medical knowledge test and by a knowledge test on technical skills. Second, in order to check for time-order effects, participants were randomly divided into two groups of 50 each, comparable on scores of both knowledge tests and on professional characteristics. Finally, both groups went through a multiple station examination using standardized patients and a practice video assessment of real surgery, but in opposite orders. Consultations were videotaped and assessed by well-trained peer observers. The drop-out rate was 10%. In both groups the predictive value of medical knowledge tests, ranging from 0.43 to 0.56 (Pearson correlation disattenuated), proved to be comparable with the predictive value of the multiple-station examination for actual performance (0.33-0.59). The overall explained variance of scores of the practice video assessment, measured by multiple regression analysis with performance scores as dependent variables and scores on the knowledge tests and the multiple-station examination as independent variables was moderate (19%). A time-order effect showed in only one direction: from practice video assessment to the multiple-station examination. The GP's professional characteristics did not contribute to the explanation of variation in performance. Medical knowledge tests can predict actual clinical performance to the same extent as a multiple-station examination. Compared with a station examination, a knowledge test may be a good alternative method for assessment the procedures of a large number of practising GPs.