Predicting doctor performance outcomes of curriculum interventions: problem-based learning and continuing competence

Context  Problem-based learning (PBL) is an educational strategy designed to enhance self-assessment, self-directed learning and lifelong learning. The present study examines a peer review programme to determine whether the impact of PBL on continuing competence can be detected in practice.
Objectives  This study aimed to establish whether McMaster graduates who graduated between 1972 and 1991 were any less likely to be identified as having issues of competence by a systematic peer review programme than graduates of other Ontario medical schools.
Methods  We identified a total of 1166 doctors who had graduated after 1972 and had completed a mandated peer review programme. Of these, 108 had graduated from McMaster and 857 from other Canadian schools. School of graduation was cross-tabulated against peer rating. A secondary analysis examined predictors of ratings using multiple regression.
Results  We found that 4% of McMaster graduates and 5% of other graduates were deemed to demonstrate cause for concern or serious concern, and that 24% of McMaster doctors and 28% of other doctors were rated as excellent. These differences were not significant. Multiple regression indicated that certification by family medicine or a specialty, female gender and younger age were all predictors of practice outcomes, but school of graduation was not.
Conclusions  There is no evidence from this study that PBL graduates are better able to maintain competence than graduates of conventional schools. The study highlights potential problems in attempting to link undergraduate educational interventions to doctor performance outcomes.     

The ‘expert’ in problem‐based and case‐based learning: necessary or not?

BACKGROUND: Problem-based learning (PBL) and other small-group teaching methods which incorporate principles of adult learning, are exciting innovations in medical education. In the application of these methods to medical curricula many schools have introduced non-expert tutors. However, research evaluating the effectiveness of non-expert lead teaching has been inconclusive. AIM: The present study aim was to compare the outcome of teaching in small groups facilitated by either an 'expert' or a 'non-expert' tutor, in a single topic area. METHOD: Fourth-year medical students were allocated randomly to teaching of eating disorders either by a non-expert or an expert tutor. Outcomes were evaluated by (i) a knowledge test, and (ii) self-report ratings by tutors and students of their learning methods and other qualities. RESULTS: The study found that while the non-expert tutor was rated more highly for her group management skills, and she also rated her students more highly in the area of oral communication, students who were taught by the expert scored higher in the end-of-course test in the topic area. CONCLUSION: The findings suggest that caution should be exercised, and the need for more research before widespread adoption of teaching by non-expert tutors.     

Techniques used by "expert" and "non-expert" tutors to facilitate problem-based learning tutorials in an undergraduate medical curriculum

BACKGROUND: There is inconclusive debate within the literature as to whether the best problem-based learning (PBL) tutors are subject experts or not. The debate hinges on whether knowledgeable tutors are tempted to intervene too often in PBL discussions compared to non-expert tutors, and whether the latter may not be able to sufficiently challenge the students' level of understanding. PURPOSE: To describe approaches used by tutors in PBL tutorials and to identify differences between tutors from medical and non-medical backgrounds. METHODS: The research reported in this paper was undertaken during the academic session 1999-2000 at the University of Liverpool Faculty of Medicine. A qualitative exploratory case study method was used and two PBL groups were observed. One of these groups had a medically qualified tutor and the other had a tutor from a humanities background. The focus of the observation was the discourse between tutor and students, which was analysed using a framework drawn from linguistics. Results were fed back to both the tutors and the students to check their perceptions of the interactions. RESULTS: Analysis of the tutorial group interaction revealed that tutors from both backgrounds used similar techniques to raise students' awareness, facilitate the group process and direct students' learning. Differences were noted between the two tutors: the medical tutor set out to raise students' awareness by using questioning techniques herself, whereas the non-medical tutor expected students to question each other. The non-medical tutor was observed to facilitate the group process more often than the medical tutor. CONCLUSIONS: Qualitative analysis of spoken discourse in PBL tutorials provides valuable insights into the processes involved in PBL, thereby generating material which is useful for both training of and giving feedback to PBL tutors.     

Standardized patients in the early acquisition of clinical skills

OBJECTIVES: Facing dramatic reductions of inpatient beds and fewer faculty tutors, the Queen's University medical school has had to consider alternatives to the traditional inpatient encounter for the early acquisition of clinical skills. The purpose of this study was to investigate the feasibility and impact on students of a model for a first-year clinical skills course, using exclusively standardized patients in place of inpatients, and using a smaller ratio of faculty to students. DESIGN: Twenty volunteers were selected from the first-year class of 75 students to participate as the experimental group. The remaining 55 students formed the control group. SETTING: Queen's University medical school, Ontario, Canada. SUBJECTS: First-year medical students. RESULTS: In measures of student satisfaction with the amount of performance feedback received during the course, there was a non-significant trend towards greater satisfaction among the experimental group compared to the control group. This occurred despite a tutor to student ratio of 1.5:10 in the experimental group compared to 3:10 in the control group. In the student evaluation of their tutors, mean scores for the two groups were 4.28 for the experimental group and 4. 06 for the control group (P = 0.10). The mean OSCE scores for the two groups were 76.4 for the experimental group and 76.5 for the control group (P = 0.93). There is no practical or statistical difference in either of these two scores. CONCLUSIONS: We conclude that this new model is feasible, was well received by the students and that the reduced tutor to student ratio and lack of inpatient encounters did not appear to adversely affect their learning.     

Implementation of problem-based learning in Asian medical schools and students' perceptions of their experience

BACKGROUND: Since the introduction of problem-based learning (PBL) at McMaster University in 1969, many medical schools in the USA, Canada and Europe have included PBL in their curricula. In the past decade, many medical schools in Asia have also done so. However, so far no one has questioned whether the outcomes expected of the learner in a PBL setting are applicable to students from different cultural upbringings. AIM: The aim of this study was to investigate the implementation of PBL in Asian medical schools, their students' perceptions of this new mode of teaching/learning and how the problems that have arisen may be overcome. METHOD: Published reports and conference presentations were gathered on the implementation of PBL in some Asian medical schools and comparisons of the experiences in PBL of Asian and students of other ethnic backgrounds. RESULTS: Most Asian medical schools and their students appear to be positive about adapting to PBL in their curriculum. The positive and negative observations appear to be similar to those experienced in non-Asian medical schools. The problems that arose for students in Asian medical schools in the early stages of implementing PBL appear to have been overcome after a period of adjustment with the help of supportive and open-minded tutors. The reports also suggested that trigger problems should be carefully designed to make them relevant and interesting for the students. CONCLUSION: Strong support from the academic administrators (dean and other staff responsible for implementation of the curriculum) in the introduction of PBL into the curriculum and careful training of both faculty and students appear to be key factors to ensure the successful implementation of PBL in Asian medical schools.     

Cross-year peer tutoring experience in a medical school: conditions and outcomes for student tutors

AIMS: To examine the features of cross-year peer tutoring and to explore their relationships to learners' characteristics and educational outcomes from the student-tutor perspective. METHOD: The records of 447 final year medical students were examined to provide data on the starting terms, frequency and course targets of peer tutoring activity of student tutors. The relationships of these features with their learners' characteristics, academic achievements and selective clerkship pathways were analysed. SETTING: The medical education programme at the University of Brasilia, Brazil. RESULTS: Analysis showed that about 96% of all graduates had acted as student tutors at some time during the programme, with great variation in starting terms, numbers and types of courses tutored. The average number of tutored courses per tutor was four. Frequency and variety of tutored courses were significantly related to achievement, learning style and gender. Higher achievers acted as student tutors for many terms and explored different subjects, and there is evidence that the experience expanded their academic expertise. Specific tutoring in a clinical course also related to strength of early career preference. Furthermore, there was a significant correlation between the number of terms of tutoring undertaken in a clinical course and the proportion of students choosing selective clerkship training in the same area by the end of programme. CONCLUSIONS: The findings suggest that acting as a peer tutor can be an appealing and constructive educational opportunity to further students' academic development. Enhanced expertise seems to relate to the accumulation and breadth of tutoring experience. Moreover, clinical tutoring may help students in making decisions regarding choice of career.     

Does feedback matter? Practice‐based learning for medical students after a multi‐institutional clinical performance examination

CONTEXT: Achieving competence in 'practice-based learning' implies that doctors can accurately self- assess their clinical skills to identify behaviours that need improvement. This study examines the impact of receiving feedback via performance benchmarks on medical students' self-assessment after a clinical performance examination (CPX). METHODS: The authors developed a practice-based learning exercise at 3 institutions following a required 8-station CPX for medical students at the end of Year 3. Standardised patients (SPs) scored students after each station using checklists developed by experts. Students assessed their own performance immediately after the CPX (Phase 1). One month later, students watched their videotaped performance and reassessed (Phase 2). Some students received performance benchmarks (their scores, plus normative class data) before the video review. Pearson's correlations between self-ratings and SP ratings were calculated for overall performance and specific skill areas (history taking, physical examination, doctor-patient communication) for Phase 1 and Phase 2. The 2 correlations were then compared for each student group (i.e. those who received and those who did not receive feedback). RESULTS: A total of 280 students completed both study phases. Mean CPX scores ranged from 51% to 71% of items correct overall and for each skill area. Phase 1 self-assessment correlated weakly with SP ratings of student performance (r = 0.01-0.16). Without feedback, Phase 2 correlations remained weak (r = 0.13-0.18; n = 109). With feedback, Phase 2 correlations improved significantly (r = 0.26-0.47; n = 171). Low-performing students showed the greatest improvement after receiving feedback. CONCLUSIONS: The accuracy of student self-assessment was poor after a CPX, but improved significantly with performance feedback (scores and benchmarks). Videotape review alone (without feedback) did not improve self-assessment accuracy. Practice-based learning exercises that incorporate feedback to medical students hold promise to improve self-assessment skills.     

Medical faculty as humanistic physicians and teachers: the perceptions of students at innovative and traditional medical schools

BACKGROUND AND OBJECTIVES: The training of caring physicians represents an important goal of medical education. Little is known however, on whether medical faculty constitute good role models for teaching humanistic skills to medical students. In this study, we examined to what extent medical students at innovative and traditional schools perceived their teachers as humanistic physicians and teachers. We also explored whether pre-clinical and clinical students shared the same perceptions. METHODS: A mail survey was conducted in Canada of all second-year students and senior clerks at one innovative medical school (problem-based learning (PBL), patient-centred, community-oriented) and three traditional medical schools. Students were asked to what extent they agreed or disagreed that the majority of their teachers behaved as humanistic physicians and teachers; 10 statements were used. Overall, 65% of the 1039 students returned the questionnaire. RESULTS: Over 25% of second-year students and 40% of senior clerks did not agree that their teachers behaved as humanistic caregivers with patients or were good role models in teaching the doctor-patient relationship. More than half of second-year students and senior clerks did not agree that their teachers valued human contact with them or were supportive of students who had difficulties. There were few differences in the way medical students at innovative and traditional schools perceived their teachers' humanistic qualities. At the pre-clinical level however, there were more students from the innovative school than from the traditional schools (around 60% vs. 40%, P < 0.005) who agreed that their teachers valued human contact with them and were supportive of students. CONCLUSION: Our results indicate that the PBL curriculum fosters better teacher-student relationships during the pre-clinical years. They also suggest that an unacceptably large number of medical students are taught by physicians who seem to lack compassion and caring in their interactions with patients. This study questions the adequacy of medical faculty as role models for the acquisition of caring competence by medical students.     

The relationship between professional behaviour grades and tutor performance ratings in problem-based learning

PURPOSE: We investigated the influence of harsh grading by tutors on tutor performance rating by students. METHODS: A total of 187 tutors assessed students' professional behaviour in tutorial groups. Students rated tutor performance after receiving their grades for professional behaviour. In addition, students were asked to indicate whether they perceived their professional behaviour grades as too positive, adequate or too negative. This was considered to reflect tutors' harshness of grading. Students also rated the quality of the feedback they received from tutors with respect to their grades. RESULTS: Professional behaviour grades that students perceived as too negative, adequate or too positive were associated with tutor performance ratings of 7.4 (SD = 0.9, scale 1-10, n = 33), 7.7 (SD = 0.9, scale 1-10, n = 95) and 7.5 (SD = 0.8, scale 1-10, n = 59), respectively. Harshness of grading did not influence tutor performance ratings significantly. Tutor ratings were predicted more effectively by the quality of the feedback tutors provided on grades than by the harshness of grading. CONCLUSIONS: Tutor performance ratings were not related significantly to harshness of grading. Two explanations can be given: (1) tutor performance ratings were based on rating by groups of students and (2) the percentage of tutors who rated students' professional behaviour as unsatisfactory was low. The strong relationship between tutor performance ratings and the adequacy of the feedback given by tutors suggests that the tutor performance ratings collected in this study are a valid measure of the quality of their teaching, although, for a full picture of teaching quality, more measures will be needed.     

Effectiveness of medical school admissions criteria in predicting residency ranking four years later

BACKGROUND: Medical schools across Canada expend great effort in selecting students from a large pool of qualified applicants. Non-cognitive assessments are conducted by most schools in an effort to ensure that medical students have the personal characteristics of importance in the practice of Medicine. We reviewed the ability of University of Toronto academic and non-academic admission assessments to predict ranking by Internal Medicine and Family Medicine residency programmes. METHODS: The study sample consisted of students who had entered the University of Toronto between 1994 and 1998 inclusive, and had then applied through the Canadian resident matching programme to positions in Family or Internal Medicine at the University of Toronto in their graduating year. The value of admissions variables in predicting medical school performance and residency ranking was assessed. RESULTS: Ranking in Internal Medicine correlated significantly with undergraduate grade point average (GPA) and the admissions non-cognitive assessment. It also correlated with 2-year objective structured clinical examination (OSCE) score, clerkship grade in Internal Medicine, and final grade in medical school. Ranking in Family Medicine correlated with the admissions interview score. It also correlated with 2nd-year OSCE score, clerkship grade in Family Medicine, clerkship ward evaluation in Internal Medicine and final grade in medical school. DISCUSSION: The results of this study suggest that cognitive as well as non-cognitive factors evaluated during medical school admission are important in predicting future success in Medicine. The non-cognitive assessment provides additional value to standard academic criteria in predicting ranking by 2 residency programmes, and justifies its use as part of the admissions process.     

Student perceptions of tutor skills in problem-based learning tutorials

OBJECTIVE: The problem-based learning (PBL) tutor plays a role that is different from the role of a teacher in a conventional teaching format. In the Faculty of Medicine and Health Sciences, United Arab Emirates, all students are Arab nationals and tutors are expatriates with different sociocultural backgrounds from the students. This study was designed to investigate how students evaluate tutors in PBL tutorials and whether student evaluations of tutors change with the progress of students in PBL tutorials. METHODS: Differences in tutor performance evaluation by male and female students were also analysed. The students evaluated 12 tutor skills in a scale of 1-3, 1 being 'below average' and 3, 'outstanding'. Student responses from a total of 314 (98.1%) completed forms collected over 2 academic years were analysed statistically. A total of 14 tutors participated in the PBL programme. RESULTS: The analysis revealed that tutors as a group were rated as having average to outstanding tutor skills in 10 items of the evaluation form. Students and faculty perceptions were different for the tutor skills of guiding students for information management. The students expected more support from tutors, whereas the tutors tried to emphasize self-learning in the PBL curriculum. Lower scores to the tutors in the 'problem' bringing sociocultural and religious issues for discussion showed that a gap in sociocultural/religious understanding between students and tutors might influence tutor skills. CONCLUSIONS: Differences in tutor evaluation by male and female students indicate necessity of adopting different strategies by tutors in a different sociocultural background. The results of the study have direct implications for faculty development.     

The process of defining a profile of student competencies at the University of Vienna Medical School

CONTEXT: A new law enabling curriculum reform led a small circle of interested faculty members at the University of Vienna medical school to launch the Medizin Curriculum Wien project (MCW) to plan a new curriculum under the supervision of the Committee on Educational Affairs (Studienkommission). OBJECTIVE: The first stage of the project dealt with the definition of a profile of competencies (Qualifikationsprofil), which describes all the competencies we want our students to possess upon graduation. METHOD: This paper presents the approach to and process of defining this profile at one of Europe's largest medical schools, currently enrolling over 1500 new students per year. The procedure involved: preparing materials; communication with stakeholders; information dissemination; feedback gathering; information structuring, and formulating the profile of competencies. RESULTS: Application of the outlined steps in two rounds proved successful for the development of a profile of student competencies for Vienna. General acceptance and awareness of the new profile as well as commitment for a further reform process was engendered. The distinct characteristics of the profile are discussed. CONCLUSION AND UTILIZATION: A profile of student competencies must balance the demanding objectives of curriculum planners and the actual challenges presented by many local parameters; this can be done by employing the process outlined in this paper. The defined profile serves as an accepted reference point in the further development of the medical curriculum.     

Assessing the relationship between peer and facilitator evaluations in case-based learning

OBJECTIVES: Attempts to validate peer evaluation and to incorporate it into the curriculum have met with mixed results. The purpose of this study was to assess the use of peer evaluations in a Year 1 case-based learning course. METHODS: As part of the formal grading process for the course, all faculty facilitators (n = 69 over 3 years) completed a 12-item evaluation form for each student at the conclusion of each case. As part of a course assignment, students (n = 415 over 3 years) completed brief evaluations of their peers based on 2 criteria: the overall quality of written reports, and participation in group discussion. In addition, students provided anonymous feedback in the written end-of-course evaluation about the peer evaluation process, and faculty were asked to comment during the wrap-up luncheon for small-group facilitators. RESULTS: Response rates for the 3 Year 1 medical student classes ranged from 95% to 99%. The average number of peer evaluations completed for each student was 4.6. The G coefficients for the rater-nested-within-person generalisability study were 0.52 for written reports and 0.60 for group participation; both were based on an average of 4-5 ratings. Correlation coefficients between peer and faculty evaluations in each of the 3 consecutive years of the course ranged from 0.46 to 0.63; all were statistically significant at P < 0.001. A correction for attenuation suggests that the true score correlation between faculty and peer measures is near 1.0. DISCUSSION: This study provides strong evidence that facilitator and peer ratings measure similar constructs and shows that, even among Year 1 medical students, peer evaluation can be conducted in a valid manner.     

Barriers to breaking bad news among medical and surgical residents

Communicating "bad news" to patients and their families can be difficult for physicians. OBJECTIVE: This qualitative study aimed to examine residents' perceptions of barriers to delivering bad news to patients and their family members. DESIGN: Two focus groups consisting of first- and second-year medical and surgical residents were conducted to explore residents' perceptions of the bad news delivery process. The grounded theory approach was used to identify common themes and concepts, which included: (1) guidelines to delivering bad news, (2) obstacles to delivering bad news and (3) residents' needs. SETTING: McMaster University, Hamilton, Ontario, Canada. SUBJECTS: First- and second-year residents. RESULTS: Residents were able to identify several guidelines important to communicating the bad news to patients and their family members. However, residents also discussed the barriers that prevented these guidelines from being implemented in day-to-day practice. Specifically, lack of emotional support from health professionals, available time as well as their own personal fears about the delivery process prevented them from being effective in their roles. Residents relayed the need for increased focus on communication skills and frequent feedback with specific emphasis on the delivery of bad news. The residents in our study also stressed the importance of processing their own feelings regarding the delivery process with staff. CONCLUSIONS: Although most residents realize important guidelines in the delivery of bad news, their own fears, a general lack of supervisory support and time constraints form barriers to their effective interaction with patients.     

Assessment of student performance in problem-based learning tutorial sessions

OBJECTIVES: To assess student performance during tutorial sessions in problem-based learning (PBL). DESIGN: A 24-item rating scale was developed to assess student performance during tutorial sessions in problem-based learning (PBL) as conducted during the pre-clinical years of Medical School at the National Autonomous University of Mexico. Items were divided into three categories: Independent study, Group interaction and Reasoning skills. Fourteen tutors assessed 152 first and second-year students in 16 tutorial groups. An exploratory factor analysis with an Oblimin rotation was carried out to identify the underlying dimensions of the questionnaire. SETTING: Medical School at the National Autonomous University of Mexico. SUBJECTS: Medical students. RESULTS: Factor analysis yielded four factors (Independent study, Group interaction, Reasoning skills, and Active participation) which together accounted for 76.6% of the variance. Their Cronbach reliability coefficients were 0.95, 0.83, 0.94 and 0. 93, respectively, and 0.96 for the scale as a whole. CONCLUSIONS: It was concluded that the questionnaire provides a reliable identification of the fundamental components of the PBL method as observable in tutorial groups and could be a useful assessment instrument for tutors wishing to monitor students' progress in each of these components.     

Not knowing that they do not know: self-assessment accuracy of third-year medical students

PURPOSE: The development of self-regulated learning is a major focus of our problem-based learning (PBL) medical programme. Students who are unsuccessful in assessments often seem to lack insight into the standard of their own performance, yet the ability to self-assess accurately is essential for the effective self-management of learning. The aim of this project was to evaluate the accuracy of self- and peer-assessment according to academic performance. METHOD: In 2004, 175 3rd-year students undertook an integrated, case-based, short-essay, formative assessment. After the assessment they were provided with model answers and marking criteria. Students marked their own assessment paper and the paper of one of their peers. Assessment papers were subsequently marked by faculty members. The following data was available for each student: self-mark, faculty-mark, score awarded by a peer and the score that they awarded to their peer. Self-assessment and peer-assessment ability was compared to overall academic performance. RESULTS: Low-achieving students score themselves and their peers generously. High-achieving students score themselves more harshly than faculty. However, they score their peers accurately. CONCLUSION: In the 3rd year of the programme low-achieving students are unable to assess accurately the quality of their own work or the work of their peers in a formative written assessment. The PBL curriculum does not guarantee the appropriate development of self-assessment skills.     

The evaluation of undergraduate students' written English language skills

CONTEXT: Writing is an important skill for practitioners and students, yet this is a skill rarely taught in a formal capacity at medical school. At the University of Adelaide many students are from non-English speaking backgrounds and have varying proficiencies in English. We wished to devise a method and instrument which could identify students who may benefit from formative feedback and tuition in writing. OBJECTIVES AND METHOD: Students' written account of a short clinical interview with a standardized patient was assessed using a new instrument (the Written Language Rating Scale) designed especially for this study. The assessment of writing was made by one rater with qualifications in teaching English as a second language. SUBJECTS: 127 second-year medical students enrolled at the University of Adelaide, Australia. INSTRUMENTS AND RESULTS: The scale appeared to have good internal consistency, face and construct validity, and test security was not an issue. However, it had questionable concurrent validity with a standardized language test, although this may be partly due to the period of time which had elapsed between administration of the two tests. CONCLUSIONS: This study was useful in providing a means to objectively rate students' written English language skills and to target students in need of formative feedback and tuition. However, further research is necessary for both evaluation of medical writing and interventions for its improvement.     

Evaluation of an integrated curriculum using problem-based learning in a clinical environment: the Manchester experience

DESCRIPTION OF COURSE: In 1994, a new problem-based leaning (PBL) curriculum for year 1 medical students was introduced at the University of Manchester. The use of PBL has continued into the clinical clerkships. Year 3 of the curriculum is based entirely in a clinical environment with PBL groups meeting in three teaching hospitals. During this year, all students undertake two integrated 14-week modules with overarching themes. Each week, groups of eight students discuss a trigger problem connected to the relevant theme. The steps the groups use in the PBL process have been amended to encourage students to link their discussion with clinical experience. EVALUATION: At the end of each module, all 309 students were requested to complete an evaluation questionnaire. The response rates were 80% (n=247) and 89% (n=275) for the two core modules. RESULTS: The students have remained 'happy with the way the course is going' (83% at the end of module 2). They were also asked to rate a number of statements on a 5-point Likert scale (5=strongly agree). Concerning PBL, the students remained confident about working in a group (median 4), producing a set of learning objectives (median 4) and linking clinical experience with other knowledge (median 4). However, there were changes over the year. Fewer students agreed at the end of module 2 that 'the working problems were stimulating' (P=0.002) or 'motivated them to learn' (P < 0.001), but the clinical firms were seen as providing more appropriate experience (P=0.01) and being aware of the new curriculum and responding to it (P=0.018). We also surveyed the PBL tutors and had 65 returns from 78 people involved in the 38 PBL groups in year 3. The great majority of these are doctors employed by the National Health service. Virtually all the responders were happy to continue being a tutor (97%) and would recommend it to a colleague (93%). CONCLUSIONS: The evaluation has been positive with PBL having been successfully implemented in a clinical environment. We have identified significant changes over the year, which we need to address as we consolidate the curriculum.