Assessing the relationship between peer and facilitator evaluations in case-based learning

OBJECTIVES: Attempts to validate peer evaluation and to incorporate it into the curriculum have met with mixed results. The purpose of this study was to assess the use of peer evaluations in a Year 1 case-based learning course. METHODS: As part of the formal grading process for the course, all faculty facilitators (n = 69 over 3 years) completed a 12-item evaluation form for each student at the conclusion of each case. As part of a course assignment, students (n = 415 over 3 years) completed brief evaluations of their peers based on 2 criteria: the overall quality of written reports, and participation in group discussion. In addition, students provided anonymous feedback in the written end-of-course evaluation about the peer evaluation process, and faculty were asked to comment during the wrap-up luncheon for small-group facilitators. RESULTS: Response rates for the 3 Year 1 medical student classes ranged from 95% to 99%. The average number of peer evaluations completed for each student was 4.6. The G coefficients for the rater-nested-within-person generalisability study were 0.52 for written reports and 0.60 for group participation; both were based on an average of 4-5 ratings. Correlation coefficients between peer and faculty evaluations in each of the 3 consecutive years of the course ranged from 0.46 to 0.63; all were statistically significant at P < 0.001. A correction for attenuation suggests that the true score correlation between faculty and peer measures is near 1.0. DISCUSSION: This study provides strong evidence that facilitator and peer ratings measure similar constructs and shows that, even among Year 1 medical students, peer evaluation can be conducted in a valid manner.     

Tutoring in problem-based learning: perceptions of teachers and students

This study examined teacher and student perceptions during the first 2 years of a complete transition from a conventional to a problem-based learning (PBL) curriculum at Dalhousie University. Teaching staff who had tutored in the two pre-clinical years (   n = 88  ) completed a questionnaire at the end of the 1993–94 academic year, and student assessments of their tutors were collated for all nine units (   n = 597  ).
Seven research questions were addressed in the study which examined the faculty, student and administrative aspects of tutoring. The results showed that faculty tutors rated PBL more highly than traditional medical school methods on eight of the nine items. Teaching staff were very satisfied with their tutoring experience, but expressed a need for further training in group facilitation, questioning, handling 'difficult' situations and evaluating students. They reported that their workload outside tutorials was cut almost in half in their second year of tutoring. Students expected a tutor to be a skilled group facilitator who would guide them in their learning, while helping to maintain a positive group climate. They did not want the tutor to teach the content as they perceived the task of learning to be their responsibility.
Several major administrative factors affected tutors' and students' perceptions of tutorials, including: changing tutorial group composition and tutor every 8–10 weeks; team tutoring; end-of-unit exam; conflicting demands of basic science 'vertical' units and ongoing 'horizontal' units; departmental budgetary requirements for basic medical education; recognition of tutoring in promotion and tenure decisions; and recruitment of tutors.     

Self-Directed Learning and Student Pragmatism

OBJECTIVE: To explore the experience and practice of students entering a problem-based (PBL) medical undergraduate course and to identify contributory social, curricular and contextual factors. DESIGN: A multiple case study exploiting the natural experimental features of the setting. Qualitative data collection methods, including participant observation, interviews and focus groups were supplemented by a survey. SETTING: The medical school at the University of Liverpool between 1996 and 1998, which launched an integrated PBL curriculum in line with recommendations of the General Medical Council of the UK. PARTICIPANTS: Two successive cohorts of first year medical students entering the PBL course. RESULTS: The first case study findings demonstrated student insecurity and dependence on the faculty resources rather than conforming to PBL principles. Uncertainty gradually diminished but resource dependence persisted, endorsed by peer interaction. Survey results confirmed the qualitative findings. The hypotheses raised for the second case study predicted replications of resource dependent learning, social influences and initial insecurity. Only the first two were upheld. Second year students' advice on textbooks undercut initial uncertainty but insecurity and social comparison reappeared following the changeover of PBL groups and tutors in the second semester. Survey results were replicated. CONCLUSION: Student learning was socially agreed amongst the peer group and directed by faculty given resources. It was not self-directed. Claims made for PBL type curricula may not be justified and GMC recommendations may remain unfulfilled without rigorous attention to educational principles.     

Student perceptions of tutor skills in problem-based learning tutorials

OBJECTIVE: The problem-based learning (PBL) tutor plays a role that is different from the role of a teacher in a conventional teaching format. In the Faculty of Medicine and Health Sciences, United Arab Emirates, all students are Arab nationals and tutors are expatriates with different sociocultural backgrounds from the students. This study was designed to investigate how students evaluate tutors in PBL tutorials and whether student evaluations of tutors change with the progress of students in PBL tutorials. METHODS: Differences in tutor performance evaluation by male and female students were also analysed. The students evaluated 12 tutor skills in a scale of 1-3, 1 being 'below average' and 3, 'outstanding'. Student responses from a total of 314 (98.1%) completed forms collected over 2 academic years were analysed statistically. A total of 14 tutors participated in the PBL programme. RESULTS: The analysis revealed that tutors as a group were rated as having average to outstanding tutor skills in 10 items of the evaluation form. Students and faculty perceptions were different for the tutor skills of guiding students for information management. The students expected more support from tutors, whereas the tutors tried to emphasize self-learning in the PBL curriculum. Lower scores to the tutors in the 'problem' bringing sociocultural and religious issues for discussion showed that a gap in sociocultural/religious understanding between students and tutors might influence tutor skills. CONCLUSIONS: Differences in tutor evaluation by male and female students indicate necessity of adopting different strategies by tutors in a different sociocultural background. The results of the study have direct implications for faculty development.     

Peer Assessment in Problem-Based Learning: A Qualitative Study

Peer assessment provides a powerful avenue for students to receive feedback on their learning. Although student perceptions of peer assessment have been studied extensively in higher education, little qualitative research has been undertaken with medical students in problem-based learning (PBL) curricula. A qualitative study of students’ attitudes to, and perceptions of, peer assessment was undertaken within the framework of a larger study of metacognition with first-year medical students at the University of Queensland. A highly structured format for provision of feedback was utilised in the study design. Many recommendations from the higher education literature on optimal implementation of peer-assessment procedures were put into practice. Results indicated the existence of six main themes: (1) increased responsibility for others, (2) improved learning, (3) lack of relevancy, (4) challenges, (5) discomfort, and (6) effects on the PBL process. Five of these themes have previously been described in the literature. However, the final theme represents a unique, although not unexpected, finding. Students expressed serious concerns about the negative impact of peer assessment on the cooperative, non-judgmental atmosphere of PBL tutorial groups. The practical implications of these findings are considered.     

On the relationship between group functioning and study success in problem-based learning

INTRODUCTION: In problem-based learning (PBL), discussion in the tutorial group plays a central role in stimulating student learning. Problems are the principal input for stimulating discussion. The quality of discussion is assumed to influence student learning and, in the end, study success. AIMS: To investigate the relationships between aspects of group functioning and study success. METHODS: First-year medical students (n = 116), forming 12 PBL groups, completed a 21-item questionnaire on various aspects of a PBL session. At the end of the unit, a course examination was administered. Scales were constructed and reliability analyses conducted. RESULTS: Group functioning and case quality were strongly correlated with students' grades in a course examination. Further, students' perceptions of group functioning, case quality and the quality of their own contribution were linked strongly with each other. CONCLUSIONS: Group functioning, case quality and study success are associated with each other in PBL. The interaction between these aspects of PBL in promoting learning calls for further investigation.     

Techniques used by "expert" and "non-expert" tutors to facilitate problem-based learning tutorials in an undergraduate medical curriculum

BACKGROUND: There is inconclusive debate within the literature as to whether the best problem-based learning (PBL) tutors are subject experts or not. The debate hinges on whether knowledgeable tutors are tempted to intervene too often in PBL discussions compared to non-expert tutors, and whether the latter may not be able to sufficiently challenge the students' level of understanding. PURPOSE: To describe approaches used by tutors in PBL tutorials and to identify differences between tutors from medical and non-medical backgrounds. METHODS: The research reported in this paper was undertaken during the academic session 1999-2000 at the University of Liverpool Faculty of Medicine. A qualitative exploratory case study method was used and two PBL groups were observed. One of these groups had a medically qualified tutor and the other had a tutor from a humanities background. The focus of the observation was the discourse between tutor and students, which was analysed using a framework drawn from linguistics. Results were fed back to both the tutors and the students to check their perceptions of the interactions. RESULTS: Analysis of the tutorial group interaction revealed that tutors from both backgrounds used similar techniques to raise students' awareness, facilitate the group process and direct students' learning. Differences were noted between the two tutors: the medical tutor set out to raise students' awareness by using questioning techniques herself, whereas the non-medical tutor expected students to question each other. The non-medical tutor was observed to facilitate the group process more often than the medical tutor. CONCLUSIONS: Qualitative analysis of spoken discourse in PBL tutorials provides valuable insights into the processes involved in PBL, thereby generating material which is useful for both training of and giving feedback to PBL tutors.     

Measuring the effect of interprofessional problem-based learning on the attitudes of undergraduate health care students

OBJECTIVES: This study aimed to measure the improvement in attitudes towards interprofessional collaboration of undergraduate health care students who have a single module of interprofessional problem-based learning (PBL) using real patients as triggers integrated into their curricula. DESIGN: A dedicated module, consisting of 5 PBL seminars, was integrated into the undergraduate medical, nursing and physiotherapy curricula at the participating institutions. Seminar groups consisted of students from a single profession in the control group, and of evenly distributed students from the participating professions in the intervention group. The Interdisciplinary Education Perception Scale was used to measure improvements in attitudes towards interprofessional co-operation. Patients, faculty members and students were included in the evaluation of the interprofessional module and their comments examined for indications of adverse effects of the use of patients in this setting. RESULTS: A total of 177 students were recruited into the study and assigned to 1 of 16 seminar groups, all of which attended the complete module, 8 in control mode and 8 in intervention mode. Statistically significant improvements could be identified in the overall attitudes of male students in the intervention group, and in attitudes pertaining to the competence and autonomy of individuals in one's own profession in the intervention group as a whole. No significant improvements were detected in the control group. No adverse effects of the use of real patients came to light. CONCLUSIONS: The integration of an interprofessional educational module that requires limited student and faculty time in undergraduate health care curricula may be proven to have an effect. The Interdisciplinary Education Perception Scale may be suitable for measuring such effect. Real patients may continue to contribute to education in this setting.     

Student perceptions about the characteristics of an effective discussion during the reporting phase in problem-based learning

To explore student perceptions of factors contributing to the effectiveness of discussions in the reporting phase of the problem-based learning (PBL) process, where students report and synthesise the results of self-study. Forty-eight Year 1 and 2 medical students participated in 6 focus group interviews about the characteristics of effective group discussions and possible improvements. The data were analysed qualitatively in several stages. The analysis yielded 4 main characteristics of effective discussions: asking for, giving and receiving explanations; integrating and applying knowledge; discussing differences with regard to learning content, and guiding and monitoring the content and the group process of the discussion. Integrating and applying knowledge included structuring, relating and summarising information and providing examples from practice. Discussing different opinions included discussing a variety of literature resources and disagreements. The main learning effects mentioned by the students were retention, understanding, integration and application of knowledge. Students have clear ideas about what promotes effective discussions during the reporting phase. Their PBL experience has provided them with some insights that are in line with theory and research on collaborative learning. Future research should examine differences between student and tutor perceptions of the quality of discussions. Introductions to PBL for students and tutors should include training in asking open but focused questions, supporting explanations with arguments and dealing with conflicts about learning content. Tutors should be trained in giving effective and personal feedback. Collaborative creation of external knowledge representations (i.e. concept maps) should be advocated, as should variety of literature resources.     

Facility with the English language and problem-based learning group interaction: findings from an Arabic setting

The Faculty of Medicine and Health Sciences (FMHS), United Arab Emirates (UAE) University is in a unique position to explore issues related to English language proficiency and medical student performance. All students entering the FMHS have English as a second language. This study focused on the issues of students' proficiency in English as measured by the TOEFL test, student background factors and interaction in problem-based learning (PBL) groups. Using a modification of Bales Interaction Process Analysis, four problem-based learning groups were observed over four thematic units, to measure the degree of student interaction within PBL groups and to compare this to individual TOEFL scores and key background variables. The students' contributions correlated highly with TOEFL test results in the giving of information (range r = 0.67-0.74). The female students adhered to interacting in English during group sessions, whereas the male students were more likely to revert to using Arabic in elaborating unclear phenomena (p < 0.01). The educational level of the student's mother was highly predictive of TOEFL scores for the male students, but not for female students. Multivariate analysis was undertaken to analyse the relative contribution of the TOEFL, parental education and years of studying in English. The best predictor of students' contributions in PBL groups was identified as TOEFL scores. The study demonstrates the importance of facilitating a locally acceptable level of English proficiency prior to admission to the FMHS. However, it also highlights the importance of not focusing only on English proficiency but paying attention to additional factors in facilitating medical students in maximizing benefits from interactions in PBL settings.     

Implementation of problem-based learning in Asian medical schools and students' perceptions of their experience

BACKGROUND: Since the introduction of problem-based learning (PBL) at McMaster University in 1969, many medical schools in the USA, Canada and Europe have included PBL in their curricula. In the past decade, many medical schools in Asia have also done so. However, so far no one has questioned whether the outcomes expected of the learner in a PBL setting are applicable to students from different cultural upbringings. AIM: The aim of this study was to investigate the implementation of PBL in Asian medical schools, their students' perceptions of this new mode of teaching/learning and how the problems that have arisen may be overcome. METHOD: Published reports and conference presentations were gathered on the implementation of PBL in some Asian medical schools and comparisons of the experiences in PBL of Asian and students of other ethnic backgrounds. RESULTS: Most Asian medical schools and their students appear to be positive about adapting to PBL in their curriculum. The positive and negative observations appear to be similar to those experienced in non-Asian medical schools. The problems that arose for students in Asian medical schools in the early stages of implementing PBL appear to have been overcome after a period of adjustment with the help of supportive and open-minded tutors. The reports also suggested that trigger problems should be carefully designed to make them relevant and interesting for the students. CONCLUSION: Strong support from the academic administrators (dean and other staff responsible for implementation of the curriculum) in the introduction of PBL into the curriculum and careful training of both faculty and students appear to be key factors to ensure the successful implementation of PBL in Asian medical schools.     

Incorporating problem-based learning and video technology in teaching group process in an occupational therapy curriculum

This paper describes an innovative, cost-effective method for teaching group dynamics in an occupational therapy curriculum. The revised "Models of Group Dynamics" course incorporated problem-based learning (PBL) sessions and video technology. In this single-semester course, the class of 24 students was divided into small groups of five to seven students. Each group participated in six hours of PBL sessions and six hours of observation. PBL; effectively served as the instructional methodology to stimulate group dynamics because it demands that the students engage in the group process by creating an open structure for discussion, negotiating goals, and building team consensus in the group. The PBL experience facilitated the integration of the various content areas of the curriculum by expecting the student to apply previous learning to the construction of a therapeutic treatment plan. Through the use of a closed-circuit video monitor, students in the observation group analyzed group process skills in real time without interruption. This format provided the opportunity for students to generate feedback responses about group process with faculty guidance. The use of a closed-circuit video monitor was a low-cost, effective tool that facilitated the learning process. Measures of student learning indicated that the new course design was effective in meeting course objectives. Measures of the effectiveness of the new course design included focus groups conducted in two time periods: after completion of the course and after completion of fieldwork. The results demonstrated that the course had continuing impact on group skills carried into the fieldwork experience.     

PBL in Undergraduate Medical Education: A Qualitative Study of the Views of Canadian Residents

Background and Objectives: At McMaster University, the birthplace of problem-based learning (PBL), administrators and curriculum planners have begun the process of renewing the undergraduate MD curriculum. One step has been to conduct an environmental scan that includes input from medical residents. Methods: Individual interviews with 17 medical residents and fellows currently enrolled at McMaster University and are graduates of six Canadian medical schools. Results: PBL appears to be well known even by graduates of non-PBL Canadian medical schools. Tutors are key to a successful PBL program, should be knowledgeable about the content area under study and able to effectively facilitate groups. Tutorial problems should be realistic, up-to-date, and challenge students to investigate more than the medical aspects of the case in question. Students need to be prepared, willing to participate in peer teaching, and supportive of the group learning process. PBL programs can be improved if they incorporate elements of traditional medical programs (e.g., mini-lectures, clear learning objectives, and unbiased evaluation of student progress) while retaining the essence of student-generated learning. Conclusions: Medical residents are an underutilized source of information about undergraduate medical programs. According to our participants, more emphasis on faculty development and upgrading health care problems will improve PBL-based undergraduate medical education.     

Faculty opinions about a revised pre-clinical curriculum

INTRODUCTION: Medical schools having innovative curricula have been encouraged to ascertain the levels of satisfaction of faculty members with the curriculum. Faculty at schools that employ problem-based learning (PBL) have been shown to have positive perceptions, but not all schools are in a position to adopt PBL on a large scale. This study sought to determine faculty members' opinions about a new curriculum that is less ambitious than one utilizing true PBL. CONTEXT AND SETTING: Since 1997, the University of Otago Medical School (Dunedin, New Zealand) has had an integrated, modular pre-clinical curriculum that emphasizes clinical relevance. It has proved popular with students. This study focused on faculty members' impressions. METHODS: We surveyed faculty members' opinions with a questionnaire identical to one used in studies at PBL schools. Faculty compared the students and their own levels of satisfaction in the old and new curricula on 7 to 10 items. The overall response rate was 85.4% (152 of 178). RESULTS: Perceptions of the new curriculum were positive among teachers who taught during the pre-clinical years and those who taught the students only after they reached the clinical years. Results for individual questions were in the same direction and generally similar in magnitude to those reported on identical items for PBL. CONCLUSION: We conclude that a hybrid curriculum that is more acceptable to many traditional teachers and students than is PBL has almost as great a positive effect on faculty members' perceptions of students' abilities and of the curriculum as does PBL.     

The role of basic sciences in a problem-based learning clinical curriculum

BACKGROUND: Very little is known about the use of problem-based learning (PBL) during the later years of the undergraduate medical course and how it influences further acquisition of basic science knowledge. Similarly to many other Faculties, the PBL approach is used at Manchester in years 1 and 2, but more unusually, the curriculum continues to be centred on PBL in the clinical modules. OBJECTIVES: To explore whether and how basic science learning was continued in year 3 of the PBL clinical curriculum. METHODS: 10 of the weekly problems from the two core modules in year 3 were analysed to determine: (a) whether the design teams were using basic science objectives in devising the problems, and (b) whether PBL student groups were setting basic science learning objectives. The basic science knowledge of year 3 and 4 students was also measured. RESULTS: Similar numbers of objectives were being set by the management groups for each weekly problem (Heart, lung and blood (HLB) module, median 15, range 11-20; Nutrition, metabolism and excretion (NME) module, median 13, range 9-21). In the basic sciences, there was a median of 3 objectives per problem (range 0-6) in the NME module, but only 1 objective (0-2) per problem in the HLB module. The objectives set by six PBL groups in each module were analysed. Overall, agreement was reached on 130 occasions (62%) between the design team basic science objectives and those set for themselves by the student groups. In addition, there was a median of 2 (range 1-8) new basic science objectives brought out by the PBL groups that were not listed by the HLB module design team. In the NME module, there was again a median of 2 new objectives (range 0-6). The performance of year 3 and year 4 students in the multiple-choice questions progress test was analysed. For the 65 basic science questions, the year 3 mark was 40.8 +/- 12.3% compared with 57.1 +/- 12.3% for year 4 (P < 0.0001). CONCLUSIONS: (a) The design teams are setting basic science objectives; (b) the working problems are triggering students to set learning objectives in the basic sciences; (c) most of the objectives being set by the design teams are being triggered in the majority of group sessions; (d) the students knowledge of basic sciences increases in years 3-4.     

Assessment of student performance in problem-based learning tutorial sessions

OBJECTIVES: To assess student performance during tutorial sessions in problem-based learning (PBL). DESIGN: A 24-item rating scale was developed to assess student performance during tutorial sessions in problem-based learning (PBL) as conducted during the pre-clinical years of Medical School at the National Autonomous University of Mexico. Items were divided into three categories: Independent study, Group interaction and Reasoning skills. Fourteen tutors assessed 152 first and second-year students in 16 tutorial groups. An exploratory factor analysis with an Oblimin rotation was carried out to identify the underlying dimensions of the questionnaire. SETTING: Medical School at the National Autonomous University of Mexico. SUBJECTS: Medical students. RESULTS: Factor analysis yielded four factors (Independent study, Group interaction, Reasoning skills, and Active participation) which together accounted for 76.6% of the variance. Their Cronbach reliability coefficients were 0.95, 0.83, 0.94 and 0. 93, respectively, and 0.96 for the scale as a whole. CONCLUSIONS: It was concluded that the questionnaire provides a reliable identification of the fundamental components of the PBL method as observable in tutorial groups and could be a useful assessment instrument for tutors wishing to monitor students' progress in each of these components.     

Effectiveness of problems used in problem-based learning

Where problem-based learning (PBL) is the main method used in medical curricula, the literature suggests that it is crucial that the problems used are effective in facilitating students to identify relevant learning issues. These learning issues guide the students' studying. The present investigation explores the extent to which students identify relevant issues following exposure to prepared paper problems. In the preparatory year, in an Introduction to Medicine module, four groups of students were exposed to six themes (Health Care System, Environment and Health, Alternative and Islamic Medicine, Chronic Illness, Infectious Diseases, and Prevention and Health Promotion). Each group had two facilitators per theme. Having discussed the prepared problems, the students identified learning issues which were collected for the purpose of the study. Two content experts, using a Likert scale, analysed learning issues for their concordance to staff objectives per theme. Kappa coefficients were computed for the six PBL themes in order to assess inter-rater agreement. Learning issues identified as having no relationship to theme objectives were further analysed for their relevance to theme objectives. No objective was totally omitted by any student group. There was a 100% concordance of objectives to learning issues demonstrated over four themes. The relationship of learning issues to theme objectives ranged from 55–85% in the theme on health care system, and 73–94% in the theme on environment and health. Irrelevant learning issues were identified in the first two PBL themes addressed. Kappa coefficients over the six PBL themes varied from 0·49 to 0·82.     

Student feedback in problem based learning: a survey of 103 final year students across five Ontario medical schools

CONTEXT: Problem based learning (PBL) has become an integral component of medical curricula around the world. In Ontario, Canada, PBL has been implemented in all five Ontario medical schools for several years. Although proper and timely feedback is an essential component of medical education, the types of feedback that students receive in PBL have not been systematically investigated. OBJECTIVES: In the first multischool study of PBL in Canada, we sought to determine the types of feedback (grades, written comments, group feedback from tutor, individual feedback from tutor, peer feedback, self-assessment, no feedback) that students receive as well as their satisfaction with these different feedback modalities. SUBJECTS AND METHODS: We surveyed a sample of 103 final year medical students at the five Ontario schools (University of Toronto, McMaster University, Queens University, University of Ottawa and University of Western Ontario). Subjects were recruited via E-mail and were asked to fill out a questionnaire. RESULTS: Many students felt that the most helpful type of feedback in PBL was individual feedback from the tutor, and indeed, individual feedback was one of the more common types of feedback provided. However, although students also indicated a strong preference for peer and group feedback, these forms of feedback were not widely reported. There were significant differences between schools in the use of grades, written comments, self-assessment and peer feedback, as well as the immediacy of the feedback given. CONCLUSIONS: Across Ontario, students do receive frequent feedback in PBL. However, significant differences exist in the types of feedback students receive, as well as the timing. Although rated highly by students at all schools, the use of peer feedback and self-assessment is limited at most, but not all, medical schools.     

Not knowing that they do not know: self-assessment accuracy of third-year medical students

PURPOSE: The development of self-regulated learning is a major focus of our problem-based learning (PBL) medical programme. Students who are unsuccessful in assessments often seem to lack insight into the standard of their own performance, yet the ability to self-assess accurately is essential for the effective self-management of learning. The aim of this project was to evaluate the accuracy of self- and peer-assessment according to academic performance. METHOD: In 2004, 175 3rd-year students undertook an integrated, case-based, short-essay, formative assessment. After the assessment they were provided with model answers and marking criteria. Students marked their own assessment paper and the paper of one of their peers. Assessment papers were subsequently marked by faculty members. The following data was available for each student: self-mark, faculty-mark, score awarded by a peer and the score that they awarded to their peer. Self-assessment and peer-assessment ability was compared to overall academic performance. RESULTS: Low-achieving students score themselves and their peers generously. High-achieving students score themselves more harshly than faculty. However, they score their peers accurately. CONCLUSION: In the 3rd year of the programme low-achieving students are unable to assess accurately the quality of their own work or the work of their peers in a formative written assessment. The PBL curriculum does not guarantee the appropriate development of self-assessment skills.     

Evaluating problem-based learning in a multilingual student population

OBJECTIVES: The University of Natal Medical School in South Africa provides training for a student body composed of two groups: one with English as a first language and the other with an African language as a first language and English as the second. A new methodology was developed to evaluate an innovative course using modified problem-based learning techniques in this heterogeneous environment. DESIGN: The learning model proposed required achieving a balance of three components: content, enquiry/learning process and social interaction/group process. A multidimensional system, felt to be consistent with this educational philosophy, was developed using seven different quantitative and qualitative techniques. SETTING: The University of Natal Medical School. SUBJECTS: First-year multilingual medical students. RESULTS: The results revealed that social interaction was highly successful in reducing barriers between the student groups and between students and facilitators. However, the emphasis on group participation may have overshadowed the enquiry process, leading to superficial discussions of problems and feelings of repetitiveness. During the course students and facilitators expressed concern that the innovative assessments used did not assess the course content adequately. While the group presentations and projects were useful exercises for consolidation and group interaction, they did not enable facilitators to identify struggling students. CONCLUSIONS: The outcome of the evaluation stressed the need of achieving an appropriate balance both in the curriculum and assessments of the three components of the learning model, particularly in a setting where student backgrounds and language ability differ. Multidimensional methodology is needed for effective evaluation that promotes critical reflection.