Early identification of 'at-risk' students by the parents of paediatric patients

INTRODUCTION: Assessment of medical student clinical skills is best carried out using multiple assessment methods. A programme was developed to obtain parent evaluations of medical student paediatric interview skills for feedback and to identify students at risk of poor performance in summative assessments. METHOD: A total of 130 parent evaluations were obtained for 67 students (parent participation 72%, student participation 58%). Parents completed a 13-item questionnaire [Interpersonal Skills Rating Scale (IPS) maximum score 91, higher scores = higher student skill level]. Students received their individual parent scores and de-identified class mean scores as feedback, and participants were surveyed regarding the programme. Parent evaluation scores were compared with student performance in formative and summative faculty assessments of clinical interview skills. RESULTS: Parents supported the programme and participating students valued parent feedback. Students with a parent score that was less than 1 standard deviation (SD) below the class mean (low IPS score students) obtained lower faculty summative assessment scores than did other students (mean +/- SD, 59% +/- 5 versus 64% +/- 7; P < 0.05). Obtaining 1 low IPS score was associated with a subsequent faculty summative assessment score below the class mean (sensitivity 0.38, specificity 0.88). Parent evaluations combined with faculty formative assessments identified 50% of students who subsequently performed below the class mean in summative assessments. CONCLUSIONS: Parent evaluations provided useful feedback to students and identified 1 group of students at increased risk of weaker performance in summative assessments. They could be combined with other methods of formative assessment to enhance screening procedures for clinically weak students.     

Self-directed,integrated clinical learning through a sign-up system

AIM: To help students in a self-directed, problem-based learning (PBL) curriculum learn from a breadth of hospital experience. METHODS: Clinical units were asked to identify what clinical activities they could make available to interested students on a 'first-come, first-served' basis. A centralised, electronic system was developed that allowed students to book a fair, but not excessive, share of those learning opportunities. This web-based programme evolved into a comprehensive timetabling system that managed activities for whole classes of students as well as 'sign-ups' booked by individuals. RESULTS: The sign-up system offered 1792 hours/week, or 6 hours/student/week, to the hospital's 291 students of learning opportunities additional to firm-based teaching and learning. There were sign-ups in all major specialties and they included rounds, clinics, theatre/investigation sessions and conferences. Students accessed the system as commonly from outside the hospital as within it, and used it to view their timetables as well as to book learning opportunities. The system was used particularly heavily when examinations were imminent, indicating that students found it supportive to their learning. CONCLUSIONS: We have developed an innovative way of providing experience that is relevant to problem-based, integrative clinical education. Students, as judged by their heavy use of it, value the system.     

Can student tutors act as examiners in an objective structured clinical examination?

Context  The dissemination of objective structured clinical examinations (OSCEs) is hampered by requirements for high levels of staffing and a significantly higher workload compared with multiple-choice examinations. Senior medical students may be able to support faculty staff to assess their peers. The aim of this study is to assess the reliability of student tutors as OSCE examiners and their acceptance by their peers.
Methods  Using a checklist and a global rating, teaching doctors (TDs) and student tutors (STs) simultaneously assessed students in basic clinical skills at 4 OSCE stations. The inter-rater agreement between TDs and STs was calculated by kappa values and paired t -tests. Students then completed a questionnaire to assess their acceptance of student peer examiners.
Results  All 214 Year 3 students at the University of Göttingen Medical School were evaluated in spring 2005. Student tutors gave slightly better average grades than TDs (differences of 0.02–0.20 on a 5-point Likert scale). Inter-rater agreement at the stations ranged from 0.41 to 0·64 for checklist assessment and global ratings; overall inter-rater agreement on the final grade was 0.66. Most students felt that assessment by STs would result in the same grades as assessment by TDs (64%) and that it would be similarly objective (69%). Nearly all students (95%) felt confident that they could evaluate their peers themselves in an OSCE.
Conclusions  On the basis of our results, STs can act as examiners in summative OSCEs to assess basic medical skills. The slightly better grades observed are of no practical concern. Students accepted assessment performed by STs.     

Paired basic science and clinical problem-based learning faculty teaching side by side: do students evaluate them differently?

INTRODUCTION: Many studies have evaluated the desirability of expert versus non-expert facilitators in problem-based learning (PBL), but performance differences between basic science and clinical facilitators has been less studied. In a PBL course at our university, pairs of faculty facilitators (1 clinician, 1 basic scientist) were assigned to student groups to maximise integration of basic science with clinical science. AIMS: This study set out to establish whether students evaluate basic science and clinical faculty members differently when they teach side by side. METHODS: Online questionnaires were used to survey 188 students about their faculty facilitators immediately after they completed each of 3 serial PBL cases. Overall satisfaction was measured using a scale of 1-7 and yes/no responses were gathered from closed questions describing faculty performance. results: Year 1 students rated basic science and clinical facilitators the same, but Year 2 students rated the clinicians higher overall. Year 1 students rated basic scientists higher in their ability to understand the limits of their own knowledge. Year 2 students rated the clinicians higher in several content expertise-linked areas: preparedness, promotion of in-depth understanding, and ability to focus the group, and down-rated the basic scientists for demonstrating overspecialised knowledge. Students' overall ratings of individual faculty best correlated with the qualities of stimulation, focus and preparedness, but not with overspecialisation, excessive interjection of the faculty member's own opinions, and encouragement of psychosocial issue discussion. CONCLUSION: When taught by paired basic science and clinical PBL facilitators, students in Year 1 rated basic science and clinical PBL faculty equally, while Year 2 students rated clinicians more highly overall. The Year 2 difference may be explained by perceived differences in content expertise.     

The effectiveness of problem-based learning compared to traditional teaching in undergraduate psychiatry

OBJECTIVES: A change from traditional to problem-based learning (PBL) methods in a psychiatry attachment was evaluated by comparing the learning styles, attitudes to psychiatry and examination performance of 2 cohorts of students. It was hypothesised that the PBL curriculum would result in increased deep learning, decreased surface learning, more favourable attitudes to psychiatry and improved examination performance. It was predicted that students' examination success would be related to the use of deep and strategic learning and favourable attitudes. METHODS: Consecutive cohorts of Year 2 clinical students taught using a traditional psychiatry curriculum (n = 188) and a PBL curriculum (n = 191) were compared. Students completed the Study Process Questionnaire to assess their learning styles and the Attitudes to Psychiatry Scale at the beginning and end of the attachment. Students completed 2 end-of-attachment examinations, a multiple-choice paper and a viva. RESULTS: The PBL curriculum resulted in significantly better examination performance than did the traditional teaching curriculum, both for multiple-choice questions and the viva. No differences in learning styles or attitudes to psychiatry were found between the curricula. Students were significantly more successful in the examinations if they had received the PBL curriculum, were female, and used strategic learning. CONCLUSIONS: Examination performance indicated that the PBL curriculum was more successful than the previous course, but that this improvement was not due to students using more effective learning styles or having more favourable attitudes towards psychiatry. It is possible that students learned more effectively during the teaching sessions in the PBL curriculum, but did not change their preferred learning styles.     

Peer assessment of professional competence

BACKGROUND: Current assessment formats for medical students reliably test core knowledge and basic skills. Methods for assessing other important domains of competence, such as interpersonal skills, humanism and teamwork skills, are less well developed. This study describes the development, implementation and results of peer assessment as a measure of professional competence of medical students to be used for formative purposes. METHODS: Year 2 medical students assessed the professional competence of their peers using an online assessment instrument. Fifteen randomly selected classmates were assigned to assess each student. The responses were analysed to determine the reliability and validity of the scores and to explore relationships between peer assessments and other assessment measures. RESULTS: Factor analyses suggest a 2-dimensional conceptualisation of professional competence: 1 factor represents Work Habits, such as preparedness and initiative, and the other factor represents Interpersonal Habits, including respect and trustworthiness. The Work Habits factor had moderate, yet statistically significant correlations ranging from 0.21 to 0.53 with all other performance measures that were part of a comprehensive assessment of professional competence. Approximately 6 peer raters were needed to achieve a generalisability coefficient of 0.70. CONCLUSIONS: Our findings suggest that it is possible to introduce peer assessment for formative purposes in an undergraduate medical school programme that provides multiple opportunities to interact with and observe peers.     

A comparison of learning outcomes and attitudes in student- versus faculty-led problem-based learning: an experimental study

OBJECTIVES: To compare learning outcomes and perceptions of facilitator behaviours and small-group process in problem-based learning (PBL) groups led by students and those led by faculty. DESIGN: A prospective, Latin-square cross-over design was employed. Second-year medical students participated in 11 PBL cases over the course of the academic year. For each case, half the student groups were led by faculty and the other half by a student group member selected randomly to serve in the facilitator role. Learning outcomes were assessed by performance on objective examinations covering factual materials pertinent to the case. Perceptions of facilitator behaviours and of group functioning were assessed with a questionnaire completed at the end of each individual case. Focus-group discussions were held to gain more in-depth information about student perceptions and experience. Student-led sessions were observed at random by the investigators. SETTING: A state-supported, US medical school with a hybrid lecture-based and problem-based curriculum. SUBJECTS: One hundred and twenty-seven second-year medical students and 30 basic science and clinical faculty. RESULTS: No differences were detected in student performance on the objective evaluation based on whether the facilitator was a faculty member or peer group member, nor were there any differences in the perceptions of group process. Students gave peer facilitators slightly higher ratings in the second semester of the experiment. In the focus-group discussions, students voiced a general preference for student-led groups because they felt they were more efficient. Observation and focus-group reports suggest that groups led by students sometimes took short cuts in the PBL process. CONCLUSION: In a hybrid lecture- and PBL-based curriculum, student performance on objective examinations covering PBL materials is unaffected by the status of the facilitator (student vs. faculty). However, in peer-facilitated groups, students sometimes took short cuts in the PBL process that may undermine some of the intended goals of PBL.     

Good for them or good for us? The role of academic guidance interviews

CONTEXT: Weak students may pass unnoticed through their undergraduate programme until they fail a summative assessment. Early recognition and support of struggling students is crucial if they are to avoid subsequent failure at high stakes assessments. METHODS: We introduced a whole-class formative assessment 5 months before a summative assessment. Subsequently, we supported poorly performing students with academic guidance interviews and remedial support. RESULTS: The programme was evaluated by questionnaires issued to staff and students involved in the guidance interviews, and by recording the performance of students who had failed the formative assessment in the subsequent summative assessment. CONCLUSIONS: We conclude that the rescue of failing students requires early recognition and tailored intervention, with support for all participants.     

In-training assessment: qualitative study of effects on supervision and feedback in an undergraduate clinical rotation

BACKGROUND: Supervision and feedback are essential factors that contribute to the learning environment in the context of workplace learning and their frequency and quality can be improved. Assessment is a powerful tool with which to influence students' learning and supervisors' teaching and thus the learning environment. OBJECTIVE: To investigate an in-training assessment (ITA) programme in action and to explore its effects on supervision and feedback. DESIGN: A qualitative study using individual, semistructured interviews. SUBJECTS AND SETTING: Eight students and 17 assessors (9 members of staff and 8 residents) in the internal medicine undergraduate clerkship at Vrije Universiteit Medical Centre, Amsterdam, the Netherlands. RESULTS: The ITA programme in action differed from the intended programme. Assessors provided hardly any follow-up on supervision and feedback given during assessments. Although students wanted more supervision and feedback, they rarely asked for it. Students and assessors failed to integrate the whole range of competencies included in the ITA programme into their respective learning and supervision and feedback. When giving feedback, assessors rarely gave borderline or fail judgements. DISCUSSION AND CONCLUSION: If an ITA programme in action is to be congruent with the intended programme, the implementation of the programme must be monitored. It is also necessary to provide full information about the programme and to ensure this information is given repeatedly. Introducing an ITA programme that includes the assessment of several competencies does not automatically lead to more attention being paid to these competencies in terms of supervision and feedback. Measures that facilitate change in the learning environment seem to be a prerequisite for enabling the assessment programme to steer the learning environment.     

Development of clinical reasoning from the basic sciences to the clerkships: a longitudinal assessment of medical students' needs and self-perception after a transitional learning unit

BACKGROUND: To facilitate students' transition from basic, science-oriented, problem-based learning (PBL) to clinical reasoning-oriented PBL, the University of Geneva School of Medicine introduced a 12-week unit of Introduction to Clinical Reasoning (ICR) at the beginning of its fourth or clerkship year. PURPOSE: The aims of the present study were to determine, after 12 weeks in the ICR unit, to what extent students had: (1) identified the learning content set by the faculty while adapting to the hypothetico-deductive reasoning approach; (2) familiarised themselves with the clinical reasoning-oriented learning process, and (3) transferred and further developed this process during the clinical years. METHOD: Students' derived objectives from the problems were compared to the objectives preset by the faculty to determine acquisition of intended learning content. To assess their adaptation to the clinical reasoning-oriented PBL approach, students (n = 124) were asked to list and freely comment on aspects of the unit they felt most at ease with or had difficulty with, and to complete a questionnaire on the clinical reasoning process (CRP). The same questionnaire was administered 6 and 12 months later to assess the evolution of the students' self-perception during clerkships. RESULTS: On average, student objectives matched 62% of faculty objectives. Half of the missed (38%) objectives were in basic sciences. Students generated 16% additional objectives, also predominantly in the basic sciences category (41%). Free comments indicated that the difficulties perceived by students were very similar to those previously reported in studies on reasoning and errors, such as difficulty in gathering, interpreting and weighting relevant data, synthesising information, and organising it hierarchically. These results were confirmed with the CRP questionnaire administered at the end of the unit. For most of the competencies assessed on the CRP questionnaire, a gradual improvement was seen to have occurred by 6 and 12 months after the unit. CONCLUSIONS: To ease students' transition from the preclinical to clinical years, a learning unit should give them the opportunity to train their clinical reasoning processes on standardised and prototypical problems, before encountering real patients with more ill-structured problems during clerkships. Such a transitional structure should particularly emphasise a developed repertoire of problem representations, recognition of key findings and a hierarchical classification of working hypotheses. It should foster the creation of links between the acquired basic clinical knowledge and the diagnostic, management and therapy steps of problem solving.     

Empirical evidence for symbiotic medical education: a comparative analysis of community and tertiary-based programmes

BACKGROUND: Flinders University has developed the Parallel Rural Community Curriculum (PRCC), a full year clinical curriculum based in rural general practice in South Australia. The examination performance of students on this course has been shown to be higher than that of their tertiary hospital-based peers. AIM: To compare the learning experiences of students in the community-based programme with those of students in the tertiary hospital in order to explain these improved academic outcomes. METHOD: A case study was undertaken, using an interpretivist perspective, with 3 structured interviews carried out over 2 academic years with each of 6 students from the community-based programme and 16 students from the tertiary hospital. The taped interviews were transcribed and analysed thematically using NUD*IST software. RESULTS: The community-based programme was successful in immersing the students in the clinical environment in a meaningful way. Four key themes were found in the data. These represented clear differences between the experiences of the community-based and hospital-based students. These differences involved: the value that students perceived they were given by supervising doctors and their patients; the extent to which the student's presence realised a synergy between the work of the university and the health service; opportunities for students to meet the aspirations of both the community and government policy, and opportunities for students to learn how professional expectations can mesh with their own personal values. CONCLUSION: This study has provided empirical evidence for the importance of the concept of symbiosis in understanding quality in medical education.     

Assessing the relationship between peer and facilitator evaluations in case-based learning

OBJECTIVES: Attempts to validate peer evaluation and to incorporate it into the curriculum have met with mixed results. The purpose of this study was to assess the use of peer evaluations in a Year 1 case-based learning course. METHODS: As part of the formal grading process for the course, all faculty facilitators (n = 69 over 3 years) completed a 12-item evaluation form for each student at the conclusion of each case. As part of a course assignment, students (n = 415 over 3 years) completed brief evaluations of their peers based on 2 criteria: the overall quality of written reports, and participation in group discussion. In addition, students provided anonymous feedback in the written end-of-course evaluation about the peer evaluation process, and faculty were asked to comment during the wrap-up luncheon for small-group facilitators. RESULTS: Response rates for the 3 Year 1 medical student classes ranged from 95% to 99%. The average number of peer evaluations completed for each student was 4.6. The G coefficients for the rater-nested-within-person generalisability study were 0.52 for written reports and 0.60 for group participation; both were based on an average of 4-5 ratings. Correlation coefficients between peer and faculty evaluations in each of the 3 consecutive years of the course ranged from 0.46 to 0.63; all were statistically significant at P < 0.001. A correction for attenuation suggests that the true score correlation between faculty and peer measures is near 1.0. DISCUSSION: This study provides strong evidence that facilitator and peer ratings measure similar constructs and shows that, even among Year 1 medical students, peer evaluation can be conducted in a valid manner.     

Approaches to medical student remediation after a comprehensive clinical skills examination

OBJECTIVE: Most US medical schools conduct comprehensive clinical skills assessments during Years 3 and 4. This study explores strategies used to identify and remediate students who perform poorly on these assessments. METHODS: In the academic year 2005-06, we conducted 33 semi-structured interviews with individuals responsible for standard setting in and remediation after their schools' comprehensive clinical skills assessments. We coded interviews to identify major themes. RESULTS: Prior to remediation, some schools employed a 'verification' step to ensure the accuracy of the failing score or need for remediation. Participants described a remediation process that included some or all of 3 steps. Firstly, students' specific learning deficits were diagnosed. Next, students participated in remedial activities such as performance review sessions or practice with standardised or actual patients. Lastly, students were re-tested, usually with a shorter, more formative examination. All participants reported using a diagnostic step, most offered or required remedial activities and many re-tested, although schools varied in the emphasis placed on each step. Many participants cited the individualised attention students received from remediation faculty staff as a strength of their approach, although they raised concerns about the substantial time demands placed on remediation faculty. Most respondents reported some dissatisfaction with their school's remediation process, particularly uncertainty about efficacy or rigour. CONCLUSIONS: Schools vary in the intensity and scope of remediation offered to students who perform poorly on clinical skills assessments. Although many schools invest significant resources in remediation, the effect of these efforts on students' subsequent clinical performance is unknown.     

Peer Assessment in Problem-Based Learning: A Qualitative Study

Peer assessment provides a powerful avenue for students to receive feedback on their learning. Although student perceptions of peer assessment have been studied extensively in higher education, little qualitative research has been undertaken with medical students in problem-based learning (PBL) curricula. A qualitative study of students’ attitudes to, and perceptions of, peer assessment was undertaken within the framework of a larger study of metacognition with first-year medical students at the University of Queensland. A highly structured format for provision of feedback was utilised in the study design. Many recommendations from the higher education literature on optimal implementation of peer-assessment procedures were put into practice. Results indicated the existence of six main themes: (1) increased responsibility for others, (2) improved learning, (3) lack of relevancy, (4) challenges, (5) discomfort, and (6) effects on the PBL process. Five of these themes have previously been described in the literature. However, the final theme represents a unique, although not unexpected, finding. Students expressed serious concerns about the negative impact of peer assessment on the cooperative, non-judgmental atmosphere of PBL tutorial groups. The practical implications of these findings are considered.     

Faculty opinions about a revised pre-clinical curriculum

INTRODUCTION: Medical schools having innovative curricula have been encouraged to ascertain the levels of satisfaction of faculty members with the curriculum. Faculty at schools that employ problem-based learning (PBL) have been shown to have positive perceptions, but not all schools are in a position to adopt PBL on a large scale. This study sought to determine faculty members' opinions about a new curriculum that is less ambitious than one utilizing true PBL. CONTEXT AND SETTING: Since 1997, the University of Otago Medical School (Dunedin, New Zealand) has had an integrated, modular pre-clinical curriculum that emphasizes clinical relevance. It has proved popular with students. This study focused on faculty members' impressions. METHODS: We surveyed faculty members' opinions with a questionnaire identical to one used in studies at PBL schools. Faculty compared the students and their own levels of satisfaction in the old and new curricula on 7 to 10 items. The overall response rate was 85.4% (152 of 178). RESULTS: Perceptions of the new curriculum were positive among teachers who taught during the pre-clinical years and those who taught the students only after they reached the clinical years. Results for individual questions were in the same direction and generally similar in magnitude to those reported on identical items for PBL. CONCLUSION: We conclude that a hybrid curriculum that is more acceptable to many traditional teachers and students than is PBL has almost as great a positive effect on faculty members' perceptions of students' abilities and of the curriculum as does PBL.     

Students' views of reflective learning techniques: an efficacy study at a UK medical school

OBJECTIVE: To describe the effects of a voluntary intervention using reflective learning techniques on students' learning. DESIGN: An interventional study with reflective learning techniques offered to medical students. SETTING: Year 3 of undergraduate medicine at Cardiff University where the curriculum is integrated with early clinical contact. PARTICIPANTS: All 232 Year 3 students were invited to participate. A total of 65 attended an introductory lecture. After the lecture 35 students agreed to take part; 15 of these subsequently dropped out (some before attending tutorial groups, others after taking part for some weeks). INTERVENTIONS: Participants kept learning journals for 2 terms and attended fortnightly, facilitated tutorial groups where they discussed their reflective journal entries. Main outcome measures were qualitative interviews and examination results. RESULTS: Interviews were carried out with 19 full participants, 4 initial participants and 7 non-participants. Participants perceived that they gained a greater ability to identify learning objectives and to integrate learning. The tutorial groups encouraged students to compare progress with their peers. Some students did not take part because they thought that the large factual content of the curriculum would make reflective learning less useful. There were no differences between the groups in examination results. CONCLUSIONS: Students among the small, self-selected group of participants were better able to identify what they needed to learn although there was no improvement in examination results. Students appear unlikely to take up voluntary reflective learning if they do not think it relates to the curriculum and assessments. Student culture exerts a potent effect on willingness to attend extra tutorial groups.     

Comparative academic performance of medical students in rural and urban clinical settings

OBJECTIVE: To determine whether the academic performance of medical students learning in rural settings differs from those learning in urban settings. DESIGN: Comparison of results of assessment for 2 full cohorts and 1 part cohort of medical students learning in rural and urban settings in 2002 (209 students), 2003 (226 students) and 2004 (220 students), including results for each specialist rotation in the 3rd year and end-of-year examinations in the 2nd and 4th years. SETTING: University of Queensland School of Medicine, Brisbane. Students spent the whole 3rd year (of a 4-year graduate entry programme) conducting 5 specialist 8-week rotations in either the rural clinical division (rural students) or in Brisbane (urban students), all following the same curriculum and taking the same examinations. RESULTS: For the 2002 cohort there were no statistically significant differences in academic performance between rural and urban students. For the 2003 cohort the only significant difference was a higher score for rural students in the end of the 4th-year clinical skills examination (65.7 versus 62.3%, P = 0.025). For the 2004 cohort, rural students scored higher in the 3rd-year mental health rotation (79.3 versus 76.2%, P = 0.038) and lower in the medicine rotation (65.5 versus 68.6%, P = 0.037). CONCLUSION: Academic performance among students studying in rural and urban settings is comparable.     

Problem-based learning improves the academic performance of medical students in South Africa

OBJECTIVES: To compare the academic performance of students on the previous, classical, discipline- and lecture-based, traditional curriculum with that of subsequent students who followed an innovative, problem- and community-based curriculum. METHODS: This was a retrospective study that analysed the records of students who enrolled on the doctor training programme between 1985 and 1995, and the records of students who graduated from the programme between 1989 and 2002. OUTCOMES: The educational outcomes assessed were the attrition and graduation rates on the traditional curriculum and those on the innovative curriculum. RESULTS: A total of 149 students on the traditional curriculum and 145 students on the innovative curriculum were studied. Overall, 23% of the traditional cohort as opposed to 10.3% of the innovative cohort dropped out of the course (P = 0.0041) and 55% of the traditional cohort as opposed to 67% of the innovative cohort graduated within the minimum period of 6 years (P < 0.001). The mean throughput period was 6.71 (0.09) years in the traditional cohort and 6.44 (0.07) years in the innovative cohort (P = 0.014) CONCLUSION: The introduction of the problem-based learning/community-based education (PBL/CBE) curriculum coincided with improved academic performance. The PBL/CBE approach to medical education may have contributed to this improvement.