Evaluating the impact of moving from discipline-based to integrated assessment

BACKGROUND: The move from discipline-based to problem-based learning (PBL) at Adelaide University in 2000 offered exciting opportunities to integrate the teaching and learning of the basic and clinical sciences for medical undergraduates. However, several cohorts of students still needed to progress through the first 3 years of the more traditional curriculum. Paradoxically, their readiness to function in the integrated learning and assessment environment of the last 3 years was assessed in 7 separate discipline-based examinations at the end of third year. When considerable examination-related stress was noted in the 1997 cohort and students petitioned formally for a reduced examination load, it was considered to be time for assessment to lead the way in integrating the disciplines. AIM: After introducing third year integrated written assessments in 1998, we aimed to develop an integrated practical examination (IPE) linking theory to practice, and evaluate its impact on staff and students. METHODS: After extensive staff collaboration, a structured objective multistation IPE was developed and administered in 1999 and 2000. Its utility was evaluated using a model proposed earlier. RESULTS: Assessment validity was maximised by an extensive item review process. Reliability, as measured by Cronbach's alpha, was 0.79 and 0.80 in 1999 and 2000, respectively. An independent evaluation yielded qualitative data on the examination's educational impact, cost and acceptability. CONCLUSIONS: Investing time in changing from discipline-based to integrated assessment, integrating theory and practice, resulted in gains in assessment reliability, validity and educational impact on both staff and students.     

Evaluation of interns by senior residents and faculty: is there any difference?

INTRODUCTION: Both senior residents and faculty members evaluate family practice interns (PGY-1) on the inpatient family medicine service at the University of Missouri-Columbia. The purpose of this study was to investigate the content and nature of narrative comments on a clinical evaluation sheet. METHODS: Objective 1. The authors placed the subjective comments made by faculty and senior residents in their evaluations of PGY-1 residents into 12 distinctive categories. Objective 2. Comments were coded with a positive or negative valence. Objective 3. The genders of the evaluator and learner were recorded. RESULTS: All evaluations made between 1996 and 1999 were analysed. A total of 1341 individual comments were reviewed. Objective 1. Categories used most often were generic comments (20.2%), personal attributes (18%), and clinical competence (14.1%). There was no difference in category use based on the experience level of the evaluator (P = 0.17). Objective 2. The majority of the comments (81.9%) were positive in nature. Senior faculty members were significantly less likely to make negative comments than were junior faculty members or senior residents (P = 0.004). Objective 3. There were no differences in category use based on the gender of the evaluator (P = 0.13). CONCLUSIONS: Objective 1. Narrative evaluation comments may be placed into 12 distinctive categories. Most comments are generic and do not help to inform learning. Objective 2. A total of 82% of comments were positive. Residents were more likely to make negative comments than senior faculty members. Objective 3. There was no demonstrable gender bias in writing negative comments.     

Growth of knowledge in psychiatry and behavioural sciences in a problem-based learning curriculum

PURPOSE: To evaluate the effectiveness of undergraduate medical education in the domains of psychiatry and behavioural sciences, we examined the growth of knowledge in those disciplines in a 6-year, problem-based learning (PBL) curriculum. Psychiatry and behavioural sciences are taught in the 4 preclinical years and in the psychiatric clerkship. The integrative nature of this PBL curriculum led us to hypothesise that the knowledge growth curves for these disciplines are similar and show a steady upward trend throughout the curriculum. METHODS: All items pertaining to psychiatry and behavioural sciences in the progress tests administered in the period from September 1993 through May 2001 were identified. For those items, the percentage of correct scores in the 6 year groups were considered a multivariate observation reflecting knowledge growth across the 6-year programme. RESULTS: Knowledge growth for psychiatry and behavioural sciences increased significantly, from 12% to 59% and from 28% to 60%, respectively, between Year 1 and the end of Year 6. Apparently, students know more about behavioural sciences than about psychiatry when they enter medical school, but this difference vanishes in the last 2 years of training. Moreover, the growth curves for psychiatry and behavioural sciences started to level off after Years 3 and 4, respectively, with no additional significant growth in any of the later years. CONCLUSIONS: Psychiatry and behavioural sciences showed different patterns of knowledge growth and the 2 growth curves levelled off in Years 5 through 6. Because a student-centred, horizontally and vertically integrated PBL curriculum is aimed at effecting steady growth in knowledge in all disciplines, the slowdown in growth in the later years was among the reasons for initiating a major curricular innovation in 2001.     

Changing education,changing assessment,changing research?

BACKGROUND: In medical education, assessment of medical competence and performance, important changes have taken place in the last 5 decades. These changes have affected the basic concepts in all 3 domains. DEVELOPMENTS IN EDUCATION AND ASSESSMENT: In education constructivism has provided a completely new view on how students learn best. In assessment the change from trait-orientated to competency- or role-orientated thinking has given rise to a whole range of new approaches. Certain methods of education, such as problem-based learning (PBL), and assessment, however, are often seen as almost synonymous with the underlying concepts, and one tends to forget that it is the concept that is important and that a particular method is but 1 way of using a concept. When doing this, one runs the risk of confusing means and ends, which may hamper or slow down new developments. LESSONS FOR RESEARCH: A similar problem seems to occur often in research of medical education. Here too, methods--or, rather, methodologies--are confused with research questions. This may lead to an overemphasis on research that fits well known methodologies (e.g. the randomised controlled trial) and neglect of what are sometimes even more important research questions because they do not fit well known methodologies. CONCLUSION: In this paper we advocate a return to the underlying concepts and a careful reflection of their use in various situations.     

Feasibility and reliability of an in-training assessment programme in an undergraduate clerkship

INTRODUCTION: Structured assessment, embedded in a training programme, with systematic observation, feedback and appropriate documentation may improve the reliability of clinical assessment. This type of assessment format is referred to as in-training assessment (ITA). The feasibility and reliability of an ITA programme in an internal medicine clerkship were evaluated. The programme comprised 4 ward-based test formats and 1 outpatient clinic-based test format. Of the 4 ward-based test formats, 3 were single-sample tests, consisting of 1 student-patient encounter, 1 critical appraisal session and 1 case presentation. The other ward-based test and the outpatient-based test were multiple sample tests, consisting of 12 ward-based case write-ups and 4 long cases in the outpatient clinic. In all the ITA programme consisted of 19 assessments. METHODS: During 41 months, data were collected from 119 clerks. Feasibility was defined as over two thirds of the students obtaining 19 assessments. Reliability was estimated by performing generalisability analyses with 19 assessments as items and 5 test formats as items. RESULTS: A total of 73 students (69%) completed 19 assessments. Reliability expressed by the generalisability coefficients was 0.81 for 19 assessments and 0.55 for 5 test formats. CONCLUSIONS: The ITA programme proved to be feasible. Feasibility may be improved by scheduling protected time for assessment for both students and staff. Reliability may be improved by more frequent use of some of the test formats.     

The effect of in-training assessment on clinical confidence in postgraduate education

INTRODUCTION: The literature on how in-training assessment (ITA) works in practice and what educational outcomes can actually be achieved is limited. One of the aims of introducing ITA is to increase trainees' clinical confidence; this relies on the assumption that assessment drives learning through its content, format and programming. The aim of this study was to investigate the effect of introducing a structured ITA programme on junior doctors' clinical confidence. The programme was aimed at first year trainees in anaesthesiology. METHODS: The study involved a nationwide survey of junior doctors' self-confidence in clinical performance before (in 2001) and 2 years after (in 2003) the introduction of an ITA programme. Respondents indicated confidence on a 155-item questionnaire related to performance of clinical skills and tasks reflecting broad aspects of competence. A total of 23 of these items related to the ITA programme. RESULTS: The response rate was 377/531 (71%) in 2001 and 344/521 (66%) in 2003. There were no statistically significant differences in mean levels of confidence before and 2 years after the introduction of the ITA programme - neither in aspects that were related to the programme nor in those that were unrelated to the programme. DISCUSSION: This study demonstrates that the introduction of a structured ITA programme did not have any significant effect on trainees' mean level of confidence on a broad range of aspects of clinical competence. The importance of timeliness and rigorousness in the application of ITA is discussed.     

The interactive examination: assessing students' self-assessment ability

BACKGROUND: The ability to self-assess one's competence is a crucial skill for all health professionals. The interactive examination is an assessment model aiming to evaluate not only students' clinical skills and competence, but also their ability to self-assess their proficiency. METHODS: The methodology utilised students' own self-assessment, an answer to a written essay question and a group discussion. Students' self-assessment was matched to the judgement of their instructors. As a final task, students compared their own essay to one written by an "expert". The differences pointed by students in their comparison documents and the accompanying arguments were analysed and categorised. Students received individual feedback on their performance and learning needs. The model was tested on 1 cohort of undergraduate dental students (year 2001, n = 52) in their third semester of studies, replacing an older form of examination in the discipline of clinical periodontology. RESULTS: Students' acceptance of the methodology was very positive. Students tended to overestimate their competence in relation to the judgement of their instructors in diagnostic skills, but not in skills relevant to treatment. No gender differences were observed, although females performed better than males in the examination. Three categories of differences were observed in the students' comparison documents. The accompanying arguments may reveal students' understanding and methods of prioritising. CONCLUSIONS: Students tended to overestimate their competence in diagnostic rather than treatment skills. The interactive examination appeared to be a convenient tool for providing deeper insight into students' ability to prioritise, self-assess and steer their own learning.     

Setting a standard for performance assessment of doctor-patient communication in general practice

CONTEXT: Continuing professional development (CPD) of general practitioners. OBJECTIVE: Criterion-referenced standards for assessing performance in the real practice of general practitioners (GPs) should be available to identify learning needs or poor performers for CPD. The applicability of common standard setting procedures in authentic assessment has not been investigated. METHODS: To set a standard for assessment of GP-patient communication with video observation of daily practice, we investigated 2 well known examples of 2 different standard setting approaches. An Angoff procedure was applied to 8 written cases. A borderline regression method was applied to videotaped consultations of 88 GPs. The procedures and outcomes were evaluated by the applicability of the procedure, the reliability of the standards and the credibility as perceived by the stakeholders, namely, the GPs. RESULTS: Both methods are applicable and reliable; the obtained standards are credible according to the GPs. CONCLUSIONS: Both modified methods can be used to set a standard for assessment in daily practice. The context in which the standard will be used - i.e. the specific purpose of the standard, the moment the standard must be available or if specific feedback must be given - is important because methods differ in practical aspects.     

Computer‐aided learning in the real world of medical education: does the quality of interaction with the computer affect student learning?

BACKGROUND: Computerised learning clearly offers exciting potential for improving student learning, either as an aid to or as a replacement for traditional formats, or for the development of innovative approaches. However, rigorous evaluation of the utility of computer-aided learning (CAL) in enhancing student learning can be difficult. Many studies have compared CAL to more traditional learning formats, but there is little evidence to show which style of CAL leads to the best learning outcomes. AIM: This study aimed to test the hypothesis that a CAL tutorial, in which the learner actively interacts with the computer, will result in superior learning (ability to apply and retain knowledge) to that obtained in more passive CAL formats. METHODS: Third year medical undergraduates at Adelaide University, South Australia were randomly assigned to 4 groups. Following a pretest, only students in the "didactic", "problem-based" and "free text" groups had 2 weeks of free access to a neuroradiology CAL tutorial in their assigned format. Tutorial access was denied to all students 2 weeks before post-testing. Learning was quantified by comparing the post- to pretest scores for each of the 4 groups. RESULTS: After active interaction with the computer material, students in the free text group demonstrated a statistically significant improvement in their ability to apply and retain knowledge compared to the control group, but no advantage compared to the didactic group. CONCLUSIONS: While users of an interactive CAL tutorial demonstrated significant learning gains compared to non-CAL users, these gains were not superior to those achieved from non-interactive CAL. When evaluating education interventions such as CAL packages, it is important to use a valid assessment tool to measure learning.