Interpreting course evaluation results: insights from thinkaloud interviews with medical students

PURPOSE: To determine whether some of the fundamental assumptions that frequently underlie interpretation of course evaluation results are justified by investigating what medical students are thinking as they complete a typical basic science course evaluation. METHODS: A total of 24 students participated in thinkaloud cognitive interviews, voicing their thoughts while completing a typical evaluation instrument that included items on overall course design, educational materials and methods, and faculty teaching. Students' responses were organised to consider how they interpreted questions, formed judgements and selected response options. Major themes relevant to the meaningful interpretation of course evaluation data were identified. RESULTS: Medical students understood educational terms such as 'independent learning' in different ways from both one another and common usage. When formulating responses, students' judgements were sometimes based on unique or unexpected criteria, and they described editing their judgements by considering factors such as effort or caring on the part of teaching faculty. Students tended to avoid using the lower end of the rating scale, used the highest rating option selectively, but chose the second highest category indiscriminately. CONCLUSIONS: These results call into question fundamental assumptions that frequently underlie interpretation of course evaluation results, such as whether students understand the intended meanings of terms used in items; whether faculty members who receive the same rating are perceived similarly; whether ratings actually reflect teaching effectiveness, and whether 'positive' ratings reflect positive opinions. This study also demonstrates how thinkaloud interviews can be used in validity studies, providing information to supplement statistical and psychometric analyses.     

Does problem‐based learning lead to deficiencies in basic science knowledge? An empirical case on anatomy

INTRODUCTION: Problem-based learning (PBL) is supposed to enhance the integration of basic and clinical sciences. In a non-integrative curriculum, these disciplines are generally taught in separate courses. Problem-based learning students perceive deficiencies in their knowledge of basic sciences, particularly in important areas such as anatomy. Outcome studies on PBL show controversial results, sometimes indicating that medical students at PBL schools have less knowledge of basic sciences than do their colleagues at more traditional medical schools. We aimed to identify differences between PBL and non-PBL students in perceived and actual levels of knowledge of anatomy. METHODS: Samples of Year 4 students in all eight medical schools in the Netherlands completed a questionnaire on perceived knowledge and took part in a computerised anatomy test consisting of both clinically contextualised items and items without context. RESULTS: Problem-based learning students were found to have the same perceived level of anatomy knowledge as students at other medical schools. Differences in actual levels of knowledge were found between schools. No significant effects on knowledge levels were found for PBL schools versus non-PBL schools. CONCLUSION: The results of this study show that PBL does not result in a lower level of anatomy knowledge than more traditional educational approaches. It remains to be ascertained whether the levels students attain are adequate. Subjects for further study are the desired level of anatomy knowledge at the end of undergraduate medical education and the effectiveness of basic science learning within a clinical context and with repetition over the course of the curriculum.     

The acquisition of tacit knowledge in medical education: learning by doing

AIM: This study was designed to assess medical school teachers' tacit knowledge of basic pedagogic principles and to explore the specific character of the knowledge base. METHODS: We developed a 50-item, multiple-choice question test based on important pedagogic principles, and classified all questions as requiring either declarative or procedural knowledge. A total of 72 medical teachers representing 5 different groups of clinicians and educators agreed to sit the test. RESULTS: Teachers in all 5 groups performed well on the test of tacit pedagogic knowledge but those with advanced education degrees, or local recognition as experts, performed best. All test takers performed best on questions requiring procedural knowledge. CONCLUSION: Medical teachers possess tacit knowledge of basic pedagogic principles. Superior test performance on questions requiring procedural knowledge is consistent with their working in a clinical environment characterised by repeated procedural activities.     

Relevant behavioural and social science for medical undergraduates: a comparison of specialist and non-specialist educators

AIM: To compare what medical educators who are specialists in the behavioural and social sciences and their non-specialist counterparts consider to be core concepts that medical graduates should understand. BACKGROUND: Previously perceived as 'nice to know' rather than 'need to know', the General Medical Council (GMC) now places behavioural and social sciences on the same need-to-know basis as clinical and basic sciences. Attempts have been made to identify what components of these topics medical students need to know; however, it remains unknown if decisions over programme content differ depending on whether or not educationalists have specialist knowledge of the behavioural and social sciences. METHODS: In a survey of medical educationalists within all UK medical schools, respondents were asked to indicate from a comprehensive list of psychological, sociological and anthropological concepts what they considered a minimally competent graduate should understand. Comparisons were made between the concepts identified by specialist behavioural and social science (BSS) educators and those without such training. RESULTS: Despite different disciplinary backgrounds, non-specialist educators largely concurred with BSS specialist educators in the concepts they considered tomorrow's doctors should know about. However, among BSS specialists there remained disagreement on what BSS content was relevant for graduates. Differences reflect specialist knowledge and recognition of the role of theoretical underpinning of BSS and reveal gaps in non-specialists knowledge. CONCLUSIONS: Educationalists with formal training in the full range of behavioural and social sciences should be involved in the development of BSS curriculum content at both national and school levels.     

Panel expertise for an Angoff standard setting procedure in progress testing: item writers compared to recently graduated students

INTRODUCTION: An earlier study showed that an Angoff procedure with > or = 10 recently graduated students as judges can be used to estimate the passing score of a progress test. As the acceptability and feasibility of this approach are questionable, we conducted an Angoff procedure with test item writers as judges. This paper reports on the reliability and credibility of this procedure and compares the standards set by the two different panels. METHODS: Fourteen item writers judged 146 test items. Recently graduated students had assessed these items in a previous study. Generalizability was investigated as a function of the number of items and judges. Credibility was judged by comparing the pass/fail rates associated with the Angoff standard, a relative standard and a fixed standard. The Angoff standards obtained by item writers and graduates were compared. RESULTS: The variance associated with consistent variability of item writers across items was 1.5% and for graduate students it was 0.4%. An acceptable error score required 39 judges. Item-Angoff estimates of the two panels and item P-values correlated highly. Failure rates of 57%, 55% and 7% were associated with the item writers' standard, the fixed standard and the graduates' standard, respectively. CONCLUSION: The graduates' and the item writers' standards differed substantially, as did the associated failure rates. A panel of 39 item writers is not feasible. The item writers' passing score appears to be less credible. The credibility of the graduates' standard needs further evaluation. The acceptability and feasibility of a panel consisting of both students and item writers may be worth investigating.     

Validation of a checklist to assess ward round performance in internal medicine

BACKGROUND: Ward rounds are an essential responsibility for doctors in hospital settings. Tools for guiding and assessing trainees' performance of ward rounds are needed. A checklist was developed for that purpose for use with trainees in internal medicine. OBJECTIVE: To assess the content and construct validity of the task-specific checklist. METHODS: To determine content validity, a questionnaire was mailed to 295 internists. They were requested to give their opinion on the relevance of each item included on the checklist and to indicate the comprehensiveness of the checklist. To determine construct validity, an observer assessed 4 groups of doctors during performance of a complete ward round (n = 32). The nurse who accompanied the doctor on rounds made a global assessment of the performance. RESULTS: The response rate to the questionnaire was 80.7%. The respondents found that all 10 items on the checklist were relevant to ward round performance and that the item collection was comprehensive. Checklist mean-item scores differed between levels of expertise: junior house officers 1.4 (1.0-1.9); senior house officers 2.0 (1.5-2.9); specialist trainees 2.5 (1.8-2.8), and specialists 2.7 (2.3-3.5); median (range) (P < 0.001). A significant correlation was found between global observer scores and nurse scores (r = 0.56, P < 0.001). CONCLUSION: The checklist, developed for assessing trainees' performance of ward rounds in internal medicine, showed high content validity. Construct validity was supported by the higher scores of experienced doctors compared to those with less experience and the significant correlation between the observer's and nurses' global scores. The developed checklist should be valuable in guiding and assessing trainees on ward round performance.     

Predictive validity of a written knowledge test of skills for an OSCE in postgraduate training for general practice

PURPOSE: To examine the validity of a written knowledge test of skills for performance on an OSCE in postgraduate training for general practice. METHODS: A randomly-selected sample of 47 trainees in general practice took a knowledge test of skills, a general knowledge test and an OSCE. The OSCE included technical stations and stations including complete patient encounters. Each station was checklist rated and global rated. RESULTS: The knowledge test of skills was better correlated to the OSCE than the general knowledge test. Technical stations were better correlated to the knowledge test of skills than stations including complete patient encounters. For the technical stations the rating system had no influence on the correlation. For the stations including complete patient encounters the checklist rating correlated better to the knowledge test of skills than the global rating. CONCLUSION: The results of this study support the predictive validity of the knowledge test of skills. In postgraduate training for general practice a written knowledge test of skills can be used as an instrument to estimate the level of clinical skills, especially for group evaluation, such as in studies examining the efficacy of a training programme or as a screening instrument for deciding about courses to be offered. This estimation is more accurate when the content of the test matches the skills under study. However, written testing of skills cannot replace direct observation of performance of skills.     

The development of medical teachers: an enquiry into the learning histories of 10 experienced medical teachers

AIM: The aim of the study was to explore the different ways in which doctors have learned to teach and train. INTRODUCTION: There is no coherent theory of medical teacher development. Doctors are experts in what they teach; most have had little or no training in how they teach. Research has mostly concentrated on the acquisition and improvement of pedagogical skills by attendance at formal, generally short courses. These may have limited impact. METHODS: We carried out semistructured interviews with 10 experienced medical teachers. A review of the literature had suggested areas to explore. Interviews were transcribed and coded and thematic analysis and grounded theory used as the framework for qualitative analysis. RESULTS: Four areas were identified as important in teacher development: acquisition of educational knowledge and skills; modelling and practice of teaching skills; encouragement and motivation of teachers, and constraints on teaching and learning. DISCUSSION: The results suggest a model for teacher development that begins with doctors as learners, learning to learn and watching teachers teach. They then start to teach, acquiring and practising skills, and subsequently move on to reflect on their teaching. They can be encouraged to teach but may also be prevented from teaching. CONCLUSIONS: This inductive study proposes a model for medical teacher development that attempts to explain how doctors learn to teach and train. More research is needed to clarify the findings. There are implications for faculty development.     

Do students have sufficient knowledge of clinical anatomy?

INTRODUCTION: Comparisons of anatomy knowledge levels of students from various curricula show either no differences or small differences to the detriment of innovative schools. To pass judgement on the general level of students' anatomy knowledge, we need an absolute standard. The purpose of this study was to compare students' levels of anatomy knowledge as measured by a case-based anatomy test with standards set by different groups of experts. METHODS: A modified Angoff procedure was used to establish an absolute standard against which the students' results could be evaluated. Four panels of 9 anatomists, 7 clinicians, 9 recent graduates and 9 Year 4 students, respectively, judged 107 items of an anatomy test. The students' results on these items were compared with the standards obtained by the panels. RESULTS: If the standard established by the panel of Year 4 students was used, 64% of the students would fail the test. The standards established by the anatomists, clinicians and recent graduates would yield failure rates of 42%, 58% and 26%, respectively. CONCLUSION: According to the panels' standards, many students did not know enough about anatomy. The high expectations that the Year 4 students appeared to have of their peers may contribute to students' uncertainty about their level of anatomy knowledge.     

The problem with outcomes-based curricula in medical education: insights from educational theory

BACKGROUND: Educators across the world are charged with the responsibility of producing core learning outcomes for medical curricula. However, much educational theory exists which deliberates the value of learning outcomes in education. AIMS: This paper aims to discuss the problems surrounding outcomes-based curricula in medical education, using insights from educational theory. DISCUSSION: The paper begins with a discussion of the traditions, values and ideologies of medical curricula. It continues by analysing the issue of control within the curriculum and argues that curriculum designers and teachers control product-orientated curricula, leading to student disempowerment. The paper debates outcomes-based curricula from an ideological perspective and argues that learning outcomes cannot specify exactly what is to be achieved as a result of learning. CONCLUSIONS: The paper argues that medical schools should adopt a model for co-operative control of the curriculum, thus empowering learners. The paper also suggests that medical educators should determine the value of precise learning outcomes before blindly adopting an outcomes-based model.     

The rating of pre-clerkship examination questions by postgraduate medical students: an assessment of quality and relevancy to medical practice

OBJECTIVE: To determine whether postgraduate students are able to assess the quality of undergraduate medical examinations and to establish whether faculty can use their results to troubleshoot the curriculum in terms of its content and evaluation. SUBJECTS: First and second year family medicine postgraduate students. MATERIALS: A randomly generated sample of undergraduate medical examination questions. METHODS: Postgraduate students were given two undergraduate examinations which included questions with an item difficulty (ID) > 0.60. The students answered and then rated each question on a scale of 1-7. RESULTS: The percentage of postgraduate students answering each question correctly correlated significantly with the average perceived relevance (Examination 1: r=0.372; P < 0.05; Examination 2: r=0.458; P < 0.05). Questions plotted for average postgraduate/undergraduate performance ratio versus the average perceived relevance were significantly correlated (Examination 1: r=0.462; P < 0.01; Examination 2: r=0.458; P < 0.05). CONCLUSIONS: This study offers a method of validating question appropriateness prior to examination administration. The design has the potential to be used as a model for determining the relevancy of a medical curriculum.     

Using automatic item generation to create multiple-choice test items

Medical Education 2012: 46: 757–765 Context Many tests of medical knowledge, from the undergraduate level to the level of certification and licensure, contain multiple‐choice items. Although these are efficient in measuring examinees’ knowledge and skills across diverse content areas, multiple‐choice items are time‐consuming and expensive to create. Changes in student assessment brought about by new forms of computer‐based testing have created the demand for large numbers of multiple‐choice items. Our current approaches to item development cannot meet this demand. Methods We present a methodology for developing multiple‐choice items based on automatic item generation (AIG) concepts and procedures. We describe a three‐stage approach to AIG and we illustrate this approach by generating multiple‐choice items for a medical licensure test in the content area of surgery. Results To generate multiple‐choice items, our method requires a three‐stage process. Firstly, a cognitive model is created by content specialists. Secondly, item models are developed using the content from the cognitive model. Thirdly, items are generated from the item models using computer software. Using this methodology, we generated 1248 multiple‐choice items from one item model. Conclusions Automatic item generation is a process that involves using models to generate items using computer technology. With our method, content specialists identify and structure the content for the test items, and computer technology systematically combines the content to generate new test items. By combining these outcomes, items can be generated automatically.     

Senior medical students' appraisal of CanMEDS competencies

CONTEXT: In 2003 the Dutch Central College of Medical Specialties presented guidelines for the modernisation of all medical specialty training programmes in the Netherlands. These guidelines are based to a large extent on the CanMEDS (Canadian Medical Education Directives for Specialists) 2000 model, which defines 7 roles for medical specialists. This model was adjusted to the Dutch situation. The roles were converted to 7 fields of competency: Medical Performance; Communication; Collaboration; Knowledge and Science; Community Performance; Management, and Professionalism. OBJECTIVE: As changes in postgraduate training will probably be most effective if future trainees recognise their value, we set out to determine how senior medical students rated these fields of competency in terms of their importance. METHODS: We carried out a study at University Medical Centre (UMC) Utrecht, the Netherlands, in which 80 Year 6 medical students answered a questionnaire in which they rated the importance of each of 28 key competencies within the 7 competency fields. RESULTS: Although all key competencies were regarded as important (averages > or = 3.8), Professionalism and Communication scored highest on the student ratings. Management was assessed as least important. CONCLUSIONS: It is interesting that medical students acknowledged the importance of competencies other than those involving medical expertise and performance. It confirms the opinion that educating doctors is currently viewed as much more than providing theoretical and clinical knowledge and skills. The CanMEDS framework is appreciated by Dutch medical students. The fact that all competencies are seen as important adds to their face validity and therefore to their usefulness as a basis for postgraduate training.     

The impact of curricular change on medical students' knowledge of anatomy

BACKGROUND: In recent years, following the publication of Tomorrow's Doctors, the undergraduate medical curriculum in most UK medical schools has undergone major revision. This has resulted in a significant reduction in the time allocated to the teaching of the basic medical sciences, including anatomy. However, it is not clear what impact these changes have had on medical students' knowledge of surface anatomy. AIM: This study aimed to assess the impact of these curricular changes on medical students' knowledge of surface anatomy. SETTING: Medical student intakes for 1995-98 at the Queen's University of Belfast, UK. METHODS: The students were invited to complete a simple examination paper testing their knowledge of surface anatomy. Results from the student intake of 1995, which undertook a traditional, 'old' curriculum, were compared with those from the student intakes of 1996-98, which undertook a new, 'systems-based' curriculum. To enhance linear response and enable the use of linear models for analysis, all data were adjusted using probit transformations of the proportion (percentage) of correct answers for each item and each year group. RESULTS: The student intake of 1995 (old curriculum) were more likely to score higher than the students who undertook the new, systems-based curriculum. CONCLUSION: The introduction of the new, systems-based course has had a negative impact on medical students' knowledge of surface anatomy.     

Effects of item and rater characteristics on checklist recording: what should we look for?

OBJECTIVE: Examinations based on using standardised patients (SPs) commonly use checklist recordings to evaluate students' clinical performance. This paper examines whether and to what extent item and rater characteristics affect the reliability of history checklist recording in an SP-based assessment. METHODS: Checklist items were reviewed for the presence or absence of 5 item characteristics and a 2-point versus 3-point scoring scale. Agreement between checklist recordings obtained from SPs and clinician-examiners (CEs) were compared by item characteristics, scoring scale and CEs' level of involvement in the assessment. RESULTS: Based on 3179 pairs of recordings, the overall percentage of agreement between SPs and CEs was 83% (kappa = 0.64). Agreement was significantly higher for items scored on a 2-point than on a 3-point scale, and when the CE was also the author and the trainer of the station. After controlling for other factors, item characteristics were only marginally associated with level of interrater agreement. CONCLUSIONS: This study suggests that attention should be paid to specific aspects of checklist development and checklist recording training when an SP or CE is used as recorder.     

What can we expect of clinical teachers? Establishing consensus on applicable skills,attitudes and practices

CONTEXT: Despite myriad advances in medical education, we have not yet established a universally accepted set of attributes we can reasonably expect from our teachers. METHODS: A modified Delphi technique established the skills, attitudes and practices thought to be core for clinical teachers within our region. We identified relevant statements from the literature. Individuals with significant involvement in undergraduate teaching acted as Delphi panelists. Four statement categories emerged: Preparing to Teach, Delivery of Teaching, Teacher Conduct and Supporting Activities. Two iterations of the Delphi round then took place. In the first round, panelists were asked to accept, reject or develop the statements identified from the literature. In the second round, they were asked to accept or reject modified statements. Throughout the exercise panelists were expected to differentiate between what could be expected from both those involved in clinical undergraduate education and those with a specialist educational remit. Agreement of > or = 80% was used to assign statements to basic or advanced categories. RESULTS: A total of 38 regional panelists participated in the Delphi process. After the 2 iterations, 27 statements were accepted at basic level, mostly in Teacher Conduct (11), and least in Supporting Activities (2). Overall, only 4 statements scored > 80% at advanced level. Many statements (25 of 56), were not clearly defined as either basic or advanced and failed to gain acceptance > 80% for either category. DISCUSSION: A useful set of attributes has been developed that can be applied to a majority of clinical teachers. There was less agreement than expected around higher level attributes. Further debate is invited.     

The effectiveness of a computer-assisted instruction programme on communication skills of medical specialists in oncology

BACKGROUND: Although doctor--patient communication is important in health care, medical specialists are generally not well trained in communication skills. Conventional training programmes are generally time consuming and hard to fit into busy working schedules of medical specialists. A computer-assisted instruction (CAI) programme was developed -- 'Interact-Cancer' -- which is a time-efficient learning method and easily accessible at the workplace. OBJECTIVE: To investigate the effect of the CAI training, 'Interact-Cancer', on the communication behaviour of medical specialists, and on satisfaction of patients about their physician interaction. DESIGN: Consultations of medical specialists with cancer outpatients were videotaped at 4 specific stages, 2 before and 2 after Interact-Cancer, with intervals of 4 weeks. PATIENTS/PARTICIPANTS: Participants were 21 medical specialists, mainly internists, working in 7 hospitals, and 385 cancer outpatients. METHODS: Communication behaviour was assessed on 23 observation categories derived from the course content. Frequencies were rated as well as judgements about the quality of the performance of each target skill. Satisfaction was measured by the Medical Interview Satisfaction Scale. Data were analyzed by means of multilevel statistical methods. RESULTS: The behavioural assessment showed course effects on ratings of the physicians' quality of performance. No course effects were found on the frequencies of physicians' behaviours and on the patient satisfaction ratings. CONCLUSIONS: CAI is a promising method to supply medical specialists with postgraduate training of communication skills. The application of judgement ratings of communication behaviour proved to be valuable to evaluate course effects in real-life patient encounters.     

Growth of medical knowledge

BACKGROUND: Knowledge is an essential component of medical competence and a major objective of medical education. Thus, the degree of acquisition of knowledge by students is one of the measures of the effectiveness of a medical curriculum. We studied the growth in student knowledge over the course of Maastricht Medical School's 6-year problem-based curriculum. METHODS: We analysed 60 491 progress test (PT) scores of 3226 undergraduate students at Maastricht Medical School. During the 6-year curriculum a student sits 24 PTs (i.e. four PTs in each year), intended to assess knowledge at graduation level. On each test occasion all students are given the same PT, which means that in year 1 a student is expected to score considerably lower than in year 6. The PT is therefore a longitudinal, objective assessment instrument. Mean scores for overall knowledge and for clinical, basic, and behavioural/social sciences knowledge were calculated and used to estimate growth curves. FINDINGS: Overall medical knowledge and clinical sciences knowledge demonstrated a steady upward growth curve. However, the curves for behavioural/social sciences and basic sciences started to level off in years 4 and 5, respectively. The increase in knowledge was greatest for clinical sciences (43%), whereas it was 32% and 25% for basic and behavioural/social sciences, respectively. INTERPRETATION: Maastricht Medical School claims to offer a problem-based, student-centred, horizontally and vertically integrated curriculum in the first 4 years, followed by clerkships in years 5 and 6. Students learn by analysing patient problems and exploring pathophysiological explanations. Originally, it was intended that students' knowledge of behavioural/social sciences would continue to increase during their clerkships. However, the results for years 5 and 6 show diminishing growth in basic and behavioural/social sciences knowledge compared to overall and clinical sciences knowledge, which appears to suggest there are discrepancies between the actual and the planned curricula. Further research is needed to explain this.     

A systematic review of medical skills laboratory training: where to from here?

OBJECTIVES: The aim of this review was to evaluate the effectiveness of medical skills laboratories or simulators. In particular, it aimed to determine if performance in medical skills laboratories is transferable to actual clinical performance and maintained over time. METHODS: A range of databases was utilised to search for relevant papers published from 1998 to June 2006. Articles were included in the review if they met a number of criteria that included the evaluation of a skills laboratory or simulator for the purpose of procedural skills training, that participants were either undergraduate medical students or postgraduate medical trainees, and that the study used a randomised, controlled trial (RCT) research design in evaluation. RESULTS: A total of 44 RCTs were identified for inclusion in the review. Overall, 32 (70%) studies reported that simulator training significantly improved procedural skills performance in comparison with standard or no training. Twenty (45%) RCTs assessed the transfer of simulator performance to clinical skills performance; however, 8 of these used animal models, not real patients. Only 2 studies assessed the maintenance of skills post-intervention, both at 4-month follow-up periods. CONCLUSIONS: Medical skills laboratories do lead to improvement in procedural skills compared with standard or no training at all when assessed by simulator performance and immediately post-training. However, there is a lack of well designed trials addressing the crucial issues of transferability to clinical practice and retention of skills over time. Further research must be carried out to address these matters if medical skills laboratories are to remain an integral component of medical education.