Teaching and evaluating first and second year medical students' practice of evidence-based medicine

PURPOSE: To implement an evidence-based medicine (EBM) curriculum for Year 1 and 2 medical students, and to develop a method to evaluate their practice of EBM in discrete and relevant worksteps. METHODS: For the 100 students entering Year 1 of their medical education in 2000, we implemented a curriculum with 25-30 student contact hours of EBM instruction which used a variety of teaching formats and spanned the first and second years of their training. We developed an evaluation module that assessed the following 5 steps in the practice of EBM: generating well built questions; searching for evidence; critical appraisal; applying the evidence, and self-evaluation. We tested 2 different versions of the test module 3-months apart with the same cohort of second year students, and correlated their scores on the second module with examination components of a comprehensive assessment. We obtained feedback from the students regarding the EBM curriculum and evaluation method. RESULTS: Each test module took 2-4 hours to complete and 5-8 minutes to grade. There was moderate test-retest reliability for the total test scores (r = 0.35, P < 0.001). Step 1 scores correlated with the mock board examination scores (r = 0.23, P = 0.05). Step 2 scores correlated with the peer assessment factor "work habits" (r = 0.24, P = 0.02), and Step 3 scores correlated with clinical reasoning exercises (r = 0.31, P = 0.002). Step 4 scores lacked test-retest reliability and did not correlate with components of the comprehensive assessment. The majority of students felt there was too much focus on EBM during the first 2 years of the curriculum and they rated the EBM test module the lowest rated component of the comprehensive assessment. CONCLUSIONS: Although we have demonstrated preliminary reliability and validity of a new evaluation instrument that assess the domains of scientific knowledge, work habits and reasoning skills required in the practice of EBM, many of the correlations were weak, and we remain in the very early stages of determining if, when and how EBM instruction should occur in medical education.     

Effectiveness of a regional self-study perinatal education programme: a successful adaptation in Yucatan, Mexico

OBJECTIVE: To evaluate the effectiveness of the Perinatal Continuing Education Programme (PCEP) in a Latin American country. METHODS: We carried out a study within secondary and tertiary care, and rural Mexican Institute of Social Security (IMSS) hospitals on the Yucatan Peninsula. Participants were doctors, nurses and nursing assistants working with pregnant women and newborns at each hospital. The PCEP was translated into Spanish and then implemented between January 1998 and December 2001. Two nurses at each hospital were trained to co-ordinate the programme and the personnel were invited to participate. Participation involved purchasing the self-teaching books, study outside work hours and participation in skills demonstration and practice sessions. Evaluation included the percentage of personnel who participated in and those who completed the programme, an opinion survey of the programme, level of pre- and post-intervention knowledge, and the quality of neonatal care according to expert-recommended routines. Results were analysed with chi-square and Student's t-tests. RESULTS: A total of 65.3% of the 1421 people in the study population began the programme and 72% of those completed it. Improvement was observed in 14 of 23 (P<0.05) evaluated neonatal care practices. Participants rated the written material as very clear and useful in daily practice. CONCLUSIONS: The PCEP is an effective strategy for improving the level of knowledge and perinatal care in all regional hospitals on the Yucatan Peninsula, Mexico. This initial application of the PCEP in a Spanish-speaking country was successful.     

The current state of medical education in Japan: a system under reform

CONTEXT: Not since just after World War II has there been as dramatic a change in the system of medical education in Japan as in the last several years. Medical school curricula are including more education that mimics clinical practice through problem-based learning, organ-based curricula and implementation of the objective structured clinical examination (OSCE). In response to criticism and concerns, the Japanese government has also implemented 2 major changes in the system of postgraduate medical education. First, a 2-year structured internship has been required of all medical school graduates; the first cohort to undertake this completed it in April 2006. Second, an internship matching system was adopted and first implemented in 2003. DISCUSSION: These reforms are leading to significant shifts in clinical education in Japan. Increasing numbers of medical graduates are entering residency programmes outside specialised university hospitals and core rotations place an increased emphasis on primary care. CONCLUSIONS: These changes in the training of young doctors suggest that the general clinical competency of doctors in Japan will improve in the coming years.     

Towards evidence-based practice in medical training: making evaluations more meaningful

CONTEXT: The evaluation of training is problematic and the evidence base inconclusive. This situation may arise for 2 main reasons: training is not understood as a complex intervention and, related to this, the evaluation methods applied are often overly simplistic. METHOD: This paper makes the case for construing training, especially in the field of specialist medical education, as a complex intervention. It also selectively reviews the available literature in order to match evaluative techniques with the demonstrated complexity. CONCLUSIONS: Construing training as a complex intervention can provide a framework for selecting the most appropriate methodology to evaluate a given training intervention and to appraise the evidence base for training fairly, choosing from among both quantitative and qualitative approaches and applying measurement at multiple levels of training impact.     

Sustainability of change with quality general practitioner education in adolescent health: a 5-year follow-up

OBJECTIVE: To determine whether improvements gained in general practitioners' (GPs') self-perceived competency, attitudes and knowledge after an intervention in adolescent health care designed with evidence-based strategies in continuing medical education, are maintained longterm, 5 years post intervention. The intervention was designed with evidence-based strategies in continuing medical education. DESIGN: We carried out a follow-up postal survey of the cohort of metropolitan Australian GPs trained in the intervention 5 years previously. Measures Subsets of the original measures, used in the randomised controlled trial of the intervention, were selected to re-assess the GPs by postal survey. Self-perceived competency, attitude and knowledge were measured. Doctors were also asked about further training in adolescent health over the 5 years since the intervention and about self-reported practice. RESULTS: A total of 46 of 54 (85%) of the original intervention group returned a questionnaire. Scores at 5 years were all higher than at baseline (P < 0.01) and improvements were sustained in all measures from 12 months to 5 years after the intervention. In all, 25/46 (54%) doctors had received further training in related areas over the 5 years, but this did not improve sustainability. A total of 45/46 (98%) reported maintaining their clinical approach to youth and 22/46 (46%) reported maintaining practices to address systemic barriers to adolescent health care access. CONCLUSIONS: Quality education designed according to evidence-based strategies of effectiveness has advantages for longterm sustainability.     

The impact of a modern medical curriculum on students' proposed behaviour on meeting ethical dilemmas

OBJECTIVE: To evaluate the impact of a modern medical curriculum on students' proposed behaviour on encountering ethical dilemmas. DESIGN: Cohort design. SETTING: University of Glasgow Medical School. SUBJECTS: The first intake of students into Glasgow's new curriculum (n = 238). Main outcome measure Student answers consistent with consensus professional judgement on the ethical dilemmas posed by the vignettes of the Ethics and Health Care Survey Instrument. RESULTS: The probability of giving a consensus answer was lowest pre-Year 1 and highest post-Year 1. It reduced slightly post-Years 3 and 5, but remained significantly higher than at pre-Year 1. The performance of students undertaking a 1-year intercalated BSc, however, appeared to regress on testing post-Year 4. CONCLUSIONS: While the first year of the curriculum had a positive impact on students, the remainder of the curriculum did not impact to the same extent. These findings support the recommendation that small group teaching, the predominant teaching method in Year 1, should be preferred to lecture and large group teaching, the predominant method of the remaining curricular years. Full integration of ethics and law teaching within the rest of the curriculum is recommended, particularly during the clinical years. This has training implications for all medical teachers involved in the curriculum. The assessment of ethics should be incorporated into all formal examinations. It is recommended that ethics be addressed as part of a wider approach to professionalism in order to promote integration.     

The effectiveness of problem-based learning compared to traditional teaching in undergraduate psychiatry

OBJECTIVES: A change from traditional to problem-based learning (PBL) methods in a psychiatry attachment was evaluated by comparing the learning styles, attitudes to psychiatry and examination performance of 2 cohorts of students. It was hypothesised that the PBL curriculum would result in increased deep learning, decreased surface learning, more favourable attitudes to psychiatry and improved examination performance. It was predicted that students' examination success would be related to the use of deep and strategic learning and favourable attitudes. METHODS: Consecutive cohorts of Year 2 clinical students taught using a traditional psychiatry curriculum (n = 188) and a PBL curriculum (n = 191) were compared. Students completed the Study Process Questionnaire to assess their learning styles and the Attitudes to Psychiatry Scale at the beginning and end of the attachment. Students completed 2 end-of-attachment examinations, a multiple-choice paper and a viva. RESULTS: The PBL curriculum resulted in significantly better examination performance than did the traditional teaching curriculum, both for multiple-choice questions and the viva. No differences in learning styles or attitudes to psychiatry were found between the curricula. Students were significantly more successful in the examinations if they had received the PBL curriculum, were female, and used strategic learning. CONCLUSIONS: Examination performance indicated that the PBL curriculum was more successful than the previous course, but that this improvement was not due to students using more effective learning styles or having more favourable attitudes towards psychiatry. It is possible that students learned more effectively during the teaching sessions in the PBL curriculum, but did not change their preferred learning styles.     

Current perspectives on medical education in China

CONTEXT: China has a long tradition of education and medicine. However, limited economic conditions and a huge population mean that further development of medical education in China must be tailored to meet the country's needs. OBJECTIVES: The aim of this paper is to describe current medical education practice in China with reference to the general and historical purposes of education in China and how they have affected and continue to affect student learning. Reference is also made to both Western medicine and traditional Chinese medicine. METHODS: It is argued that traditional educational practices in China have encouraged rote learning and that creativity is not cultivated. This affects the way many Chinese students learn medicine. Since 1949, the Chinese medical education system has developed according to its own needs. The current system for training medical students is complex, with medical school curricula lasting 3-8 years. However, medical education reform is taking place and new teaching methods are being introduced in some schools. DISCUSSION: Medical education is important to China's large population. The undergraduate medical education system is being streamlined and national standards are being established. Innovations in medical education have recently been encouraged and supported, including the adoption of problem-based learning. It is important that the momentum is kept up so that the health care of a fifth of the world's population is assured.     

Evaluation of a handheld clinical decision support tool for evidence-based learning and practice in medical undergraduates

INTRODUCTION: Handheld computers (PDAs) uploaded with clinical decision support software (CDSS) have the potential to facilitate the adoption of evidence-based medicine (EBM) at the point-of-care among undergraduate medical students. Further evaluation of the usefulness and acceptability of these tools is required. METHODS: All 169 Year 4 undergraduate medical students at the University of Hong Kong completed a post-randomised controlled trial survey. Primary outcome measures were CDSS/PDA usefulness, satisfaction, functionality and utilisation. Focus groups were also conducted to derive complementary qualitative data on the students' attitudes towards using such new technology. RESULTS: Overall, the students found the CDSS/PDA useful (mean score = 3.90 out of 6, 95% confidence interval (CI) = 3.78, 4.03). They were less satisfied with the functional features of the CDSS (mean score = 3.45, 95% CI = 3.32, 3.59) and the PDA (mean score = 3.51 95% CI = 3.40, 3.62). Utilisation was low, with the average frequency of use less than once per week. Although students reported a need for information in patient care at least once daily, they infrequently used the CDSS in a clinical setting (20.4 +/- 10.4% of the time), with an average information retrieval success rate of 37.6 +/- 22.1% requiring 63.7 +/- 86.1 seconds. Multivariable regression shows that higher perceived CDSS/PDA usefulness was associated with more supportive faculty attitudes, greater knowledge of EBM, better computer literacy skills and increased use in a clinical setting. Greater satisfaction with the CDSS/PDA was associated with increased use in a clinical setting and higher successful search rates. Qualitative results were consistent with these quantitative findings and yielded additional information on students' underlying feelings that may explain the observations. CONCLUSIONS: While PDAs uploaded with the CDSS are able to provide students with better access to high quality information, improvements in faculty attitudes, students' knowledge of EBM and computer literacy skills, and having the CDSS specially designed for undergraduate use are essential to increasing student adoption of such point-of-care tools.     

The development of a participant questionnaire to assess continuing medical education presentations

OBJECTIVE: Feedback on presentation skills is important for developing skilled educators, but often this feedback is based on evaluation tools that have been developed with little concern for psychometric issues or for how the information will be used for feedback. The purpose of this study was to develop a reliable participant questionnaire to assess the quality of continuing medical education (CME) presentations and to provide presenters with feedback. DESIGN: The questionnaire was developed using an iterative approach, with doctors as raters, and tested during a variety of CME presentations. The resulting questionnaire consists of 9 items rated on a 7-point Likert scale. The psychometric analysis reported in this paper was completed using data from grand rounds presentations. RESULTS: Psychometric analysis, based on 319 evaluations from 17 presentations (average of 19 evaluations/presentation), revealed a high level of reliability (0.91), indicating that the items met a reasonable standard and that the raters were discriminating between the quality of the presentations adequately. CONCLUSION: This 9-item, participant questionnaire provides a reliable measure of the quality of CME presentations, while also providing presenters with useful feedback. Further studies will investigate if this instrument can be used to assess other CME formats and how best to provide feedback to presenters.     

Identifying barriers to the adoption of evidence-based medicine practice in clinical clerks: a longitudinal focus group study

OBJECTIVE: To identify and explore common barriers to the adoption of evidence-based medicine (EBM) practice in the undergraduate setting. DESIGN: Nested longitudinal, focus group-based, qualitative study. Setting The University of Hong Kong Medical School, Hong Kong, China. PARTICIPANTS: A group of 39 Year 4 medical undergraduates who participated in an EBM intervention cluster randomised crossover trial. MAIN OUTCOME MEASURES: Students' attitudes, opinions and perceptions of barriers to EBM use. RESULTS: General attitudes towards EBM and the teaching intervention were positive. Four sets of barriers to greater EBM use were identified as follows. (1) Learning environment including prevailing norms for student learning involving examination-oriented, textbook learning, prior availability of clinical practice guidelines, lack of encouragement from teachers and economy of time by utilising teacher expertise. (2) Limitations of evidence consisting of poor point-of-care access to medical literature, difficulty in locating evidence and the perceived low relevance of overseas evidence to Chinese patients. (3) Lack of opportunity to practise EBM due to lack of continuity of care and anxieties about negative teacher attitudes towards EBM use at the point-of-care. (4) Time constraints such as competing study demands and long evidence search time. CONCLUSIONS: Significant barriers to the successful implementation of EBM learning in the clinical clerkship setting were identified. These can be specifically targeted to ameliorate any inhibition of clinical learning they may impose.     

Comparison of directed and self-directed learning in evidence-based medicine: a randomised controlled trial

OBJECTIVES: To compare 2 educational programmes for teaching evidence-based medicine (EBM). DESIGN: Prospective randomised controlled trial accompanied by a qualitative evaluation. SETTING: University of Oslo, Norway, 2002-03. PARTICIPANTS: A total of 175 students entered the study. All tenth semester medical students from 3 semesters were eligible for inclusion if they completed baseline assessment and consent forms and either attended teaching on the first day of the semester or gave reasons for their absence on the first day in advance. Interventions One intervention was based on computer-assisted, self-directed learning (self-directed intervention), whilst the other was organised as workshops based on social learning theory (directed intervention). Both educational interventions consisted of 5 half-day sessions. MAIN OUTCOME MEASURES: The primary outcomes were knowledge about EBM and skills in critical appraisal. A secondary outcome measured attitudes to EBM. Outcomes were compared on an intention-to-treat basis using a stratified Wilcoxon rank-sum test. RESULTS: There were no differences in outcomes for the 2 study groups in terms of EBM knowledge (mean deviation 0.0 [95% confidence interval - 1.0, 1.0], P = 0.8), critical appraisal skills (MD 0.1 [95% CI - 0.9, 1.1], P = 0.5), or attitudes to EBM (MD - 0.3 [95% CI - 1.4, 0.8], P = 0.5). Follow-up rates were 96%, 97% and 63%, respectively. CONCLUSIONS: This trial and its accompanying qualitative evaluation suggest that self-directed, computer-assisted learning may be an alternative format for teaching EBM. However, further research is needed to confirm this and investigate alternative educational models.     

Electronic delivery of research summaries for academic generalist doctors: a randomised trial of an educational intervention

Objective  To determine whether weekly e-mails of structured, critically appraised summaries of new articles, within the format of a Weekly Browsing Journal Club (WBJC), would encourage use of evidence in practice.
Design  Randomised controlled trial.
Method  A sample of 107 academic general internists were randomly assigned to receive either the WBJC ( n  = 54) or a health news website ( n  = 53) by e-mail on a weekly basis for 3 months. Participants completed a pre- and post-intervention questionnaire that measured their use of evidence in practice, attitudes about the role of evidence in practice, critical appraisal ability and reading habits.
Results  Of 107 participants, 90% completed the trial. At baseline, when faced with questions, they found answers on which to base their clinical decisions 67% of the time. Participants incorporated evidence into patient care for 60% of patients and read evidence to help make decisions for 25%. After 3 months, the groups did not differ in the use of evidence in practice. Changes in attitudes toward the role of evidence in practice and in self-perceived critical appraisal ability were also similar. The subjects spent 2.2 hours reading 4.3 journals per week. Weekly reading time decreased by 10 minutes in the WBJC group, but increased by 26 minutes in the control group ( P =  0.02). At study completion, 82% of the WBJC group were satisfied with the intervention and 90% asked to continue.
Conclusion  While doctors appreciated these summaries, which improved their reading efficiency, the intervention had little impact on their use of research evidence in practice.     

The theory of planned behaviour in medical education: a model for integrating professionalism training

Context  Teaching and evaluating professionalism remain important issues in medical education. However, two factors hinder attempts to integrate curricular elements addressing professionalism into medical school training: there is no common definition of medical professionalism used across medical education, and there is no commonly accepted theoretical model upon which to integrate professionalism into the curriculum.
Objectives  This paper proposes a definition of professionalism, examines this definition in the context of some of the previous definitions of professionalism and connects this definition to the attitudinal roots of professionalism. The problems described above bring uncertainty about the best content and methods with which to teach professionalism in medical education. Although various aspects of professionalism have been incorporated into medical school curricula, content, teaching and evaluation remain controversial. We suggest that intervening variables, which may augment or interfere with medical students' implementation of professionalism knowledge, skills and, therefore, attitudes, may go unaddressed.
Discussion  We offer a model based on the theory of planned behaviour (TPB), which describes the relationships of attitudes, social norms and perceived behavioural control with behaviour. It has been used to predict a wide range of behaviours, including doctor professional behaviours. Therefore, we propose an educational model that expands the TPB as an organisational framework that can integrate professionalism training into medical education. We conclude with a discussion about the implications of using this model to transform medical school curricula to develop positive professionalism attitudes, alter the professionalism social norms of the medical school and increase students' perceived control over their behaviours.     

Empirical evidence for symbiotic medical education: a comparative analysis of community and tertiary-based programmes

BACKGROUND: Flinders University has developed the Parallel Rural Community Curriculum (PRCC), a full year clinical curriculum based in rural general practice in South Australia. The examination performance of students on this course has been shown to be higher than that of their tertiary hospital-based peers. AIM: To compare the learning experiences of students in the community-based programme with those of students in the tertiary hospital in order to explain these improved academic outcomes. METHOD: A case study was undertaken, using an interpretivist perspective, with 3 structured interviews carried out over 2 academic years with each of 6 students from the community-based programme and 16 students from the tertiary hospital. The taped interviews were transcribed and analysed thematically using NUD*IST software. RESULTS: The community-based programme was successful in immersing the students in the clinical environment in a meaningful way. Four key themes were found in the data. These represented clear differences between the experiences of the community-based and hospital-based students. These differences involved: the value that students perceived they were given by supervising doctors and their patients; the extent to which the student's presence realised a synergy between the work of the university and the health service; opportunities for students to meet the aspirations of both the community and government policy, and opportunities for students to learn how professional expectations can mesh with their own personal values. CONCLUSION: This study has provided empirical evidence for the importance of the concept of symbiosis in understanding quality in medical education.     

Change in knowledge and skills of Year 3 undergraduates in evidence-based medicine seminars

PURPOSE: To evaluate the effect of a compulsory evidence-based medicine (EBM) seminar in critical appraisal skills and the overall acceptance of compulsory EBM seminars for Year 3 medical undergraduate students. METHODS: Small group seminars by peer teaching were conducted for up to 23 undergraduates. Knowledge and skills in EBM before and after the compulsory seminars were evaluated by 2 different sets of 20 questions. To apply knowledge, each undergraduate had to analyse an individual paper case using the principles of EBM. Undergraduates gave anonymous feedback using separate evaluation sheets at the end of the seminar. Main outcome variables were changes in knowledge and skills. RESULTS: A total of 132 Year 3 undergraduates at the University of Frankfurt participated in a compulsory EBM seminar during the academic half-year 2003/04 as part of their regular curriculum. Complete datasets were available for evaluation from 124 undergraduates (94%). The seminars led to an overall increase in knowledge (question paper score increase from 2.37 to 7.48, 99% CI 6.61-8.36, or 216%). Transfer of knowledge into a paper case scenario was generally good, with a mean score of 49.5 (SD 5.24) out of 55 points. Feedback indicated good overall acceptance of the seminars, with a median of 2 (score range from 1 = excellent to 6 = failed). CONCLUSION: Trained medical students are effective and well accepted EBM trainers in compulsory undergraduate seminars.     

Humanising medical education through faculty development: linking self-awareness and teaching skills

BACKGROUND: We conducted a longitudinal faculty development programme for medical school faculty, focused on enhancing learner-centred teaching skills, by integrating traditional elements of education, focusing on knowledge, skills and attitudes, with the non-traditional process elements of community building, self-awareness and relationship formation. METHODS: This year-long programme enrolled faculty from a range of clinical departments at a single institution. The participants gathered for day-long sessions in each of 9 months and also met at lunchtime once a month for "booster" meetings. Sessions were organised according to content areas identified as relevant to enhancing teaching skills. In addition to each content area, a variety of experiential and process learning methods were incorporated and modelled in each session. The impact of the programme was assessed using self-report data after each session, and together with paired interviews at the completion of the programme. EVALUATION: Programme attendance was 97%. Qualitative thematic and content analysis showed that faculty reported renewed energy and enthusiasm for teaching, as well as skills mastery. Faculty also reported significant improvements in self-awareness and habits of lifelong learning. Many faculty members felt that a singular benefit of the programme was developing and nurturing personal as opposed to instrumental relationships. CONCLUSIONS: It is feasible to successfully convene clinical faculty from different departments in the same faculty development programme with little or no concern for competition and conflict. The key to success is integrating content and process dimensions into a framework of community building and collective engagement. From the participants' perspective, skills and confidence, which by self-report increased for all participants, was less consequential than the opportunity to learn about themselves and their relationships to others in a safe environment. We have conducted this programme for 5 successive years.     

Comparison of outcomes of a community-based education programme executed with and without active community involvement

OBJECTIVE: The aim of this study was to evaluate the applicability of a previously designed set of generic objectives for community-based education (CBE) emphasising community involvement. METHODS: The study was designed as a non-blinded, randomised trial. Experimental and conventional groups of students following CBE programmes either closely or weakly matching the set of generic objectives were compared. Student groups were subjected to passive participatory observation. Students evaluated their programmes through questionnaires. The impact of student interventions was assessed by community compliance. Community perception of the programmes was evaluated through structured interviews with community representatives. RESULTS: Students in experimental groups appreciated their programme more than students in conventional groups. High compliance and appreciation were recorded in communities hosting the modified programme. Most students in conventional groups judged their posting negatively, largely because of the high number of households to be visited. Health interventions performed by conventional groups lacked co-operation between students and the community. Communities hosting conventional groups felt their health needs were scarcely discussed and addressed. CONCLUSIONS: The modification of an existing CBE programme to better match a set of generic CBE objectives emphasising community involvement had a positive effect on programme outcomes and levels of appreciation by both students and hosting communities. However, some confounding variables could not be controlled. Colleagues planning comparable studies may take advantage of the lessons we learned while performing this study.     

Development of continuing medical education in Europe: a review

This short review sets out some general facts and features concerning the development of continuing medical education (CME) in Europe. The review is intended as a basis for proposing plans and action for CME in the future. Europe, as it happens, was the birthplace and cradle of both scientifically based medical education in general and CME in particular. Three stages of development are distinguished: (1) Development of CME from the 19th century up to the beginning of the Second World War; (2) After 1945; (3) CME in the present.