Temporal and group-related trends in peer assessment amongst medical students

Peer assessment has been increasingly recommended as a way to evaluate the professional competencies of medical trainees. Prior studies have only assessed single groups measured at a single timepoint. Thus, neither the longitudinal stability of such ratings nor differences between groups using the same peer-assessment instrument have been reported previously. Participants were all members of 2 consecutive classes of medical students (n = 77 and n = 85) at the University of Rochester School of Medicine and Dentistry who completed Years 2 and 3 of medical school consecutively. All participants were evaluated by 6-12 classmates near the end of both Years 2 and 3. Main outcome measures were mean numerical ratings on peer-assessed scales of professional work habits (WH) and interpersonal attributes (IA). Both scales had high internal consistencies in both years (Cronbach's alpha 0.84-0.94). The IA and WH scales were moderately correlated with one another (r = 0.36 in Year 2, r = 0.28 in Year 3). Year 2 scores were predictive of Year 3 scores for both scales (WH: r = 0.64; IA; r = 0.62). Generalisability and decision analyses revealed that 1 class was consistently more discriminating with the WH scale, while the other was more discriminating with of the IA scale. Depending on the class, year and scale, the number of raters needed to achieve a reasonable reliability ranged between 7 and 28. Although Year 3 peer ratings were consistently higher than Year 2 peer ratings for both WH and IA, individual scores were highly correlated across the 2 years, despite the fact that different individuals were chosen as peer raters. Abilities appear to be stable between Years 2 and 3 of medical school. Groups may differ in their ability to discriminate different kinds of skills. Generalisability analysis can be used to discover these patterns within groups.     

Teaching research methodology in medical schools: students' attitudes towards and knowledge about science

OBJECTIVE: To explore the relationship between teaching scientific methodology in Year 2 of the medical curriculum and student attitudes towards and knowledge about science and scientific methodology. DESIGN: Anonymous questionnaire survey developed for this purpose. SETTING: Zagreb University School of Medicine, Croatia. PARTICIPANTS: A total of 932 students (response rate 58%) from all 6 years were invited to participate. MAIN OUTCOME MEASURES: Score on attitude scale with 45 Likert-type statements and score on knowledge test consisting of 8 multiple choice questions. RESULTS: The average attitude score for all students was 166 +/- 22 out of a maximum of 225, indicating a positive attitude towards science and scientific research. The students' average score on the knowledge test was 3.2 +/- 1.7 on 8 questions. Students who had finished Year 2 had the highest mean attitude (173 +/- 24) and knowledge (4.7 +/- 1.7) scores compared with other year groups (P < 0.001, anova and Tukey posthoc test). For students who had attended a mandatory Year 2 course on the principles of scientific research in medicine (Years 3 to 6), multiple linear regression analysis showed that knowledge test score (B = 3.4; SE = 0.4; 95% confidence interval 2.5-4.2; P < 0.001) and average grades (B = 7.6; SE = 1.5; 95% CI 4.6-10.6; P < 0.001) were significant predictors of attitude towards science, but not sex or failure to pass a year (B = - 0.6; SE = 1.7; 95% CI - 3.9-2.6; P = 0.707; and B = - 3.1; SE = 1.9; 95% CI - 6.8-5.7; P = 0.097, respectively). CONCLUSION: Medical students have generally positive attitudes towards science and scientific research in medicine. Attendance of a course on research methodology is related to a positive attitude towards science.     

Change in knowledge and skills of Year 3 undergraduates in evidence-based medicine seminars

PURPOSE: To evaluate the effect of a compulsory evidence-based medicine (EBM) seminar in critical appraisal skills and the overall acceptance of compulsory EBM seminars for Year 3 medical undergraduate students. METHODS: Small group seminars by peer teaching were conducted for up to 23 undergraduates. Knowledge and skills in EBM before and after the compulsory seminars were evaluated by 2 different sets of 20 questions. To apply knowledge, each undergraduate had to analyse an individual paper case using the principles of EBM. Undergraduates gave anonymous feedback using separate evaluation sheets at the end of the seminar. Main outcome variables were changes in knowledge and skills. RESULTS: A total of 132 Year 3 undergraduates at the University of Frankfurt participated in a compulsory EBM seminar during the academic half-year 2003/04 as part of their regular curriculum. Complete datasets were available for evaluation from 124 undergraduates (94%). The seminars led to an overall increase in knowledge (question paper score increase from 2.37 to 7.48, 99% CI 6.61-8.36, or 216%). Transfer of knowledge into a paper case scenario was generally good, with a mean score of 49.5 (SD 5.24) out of 55 points. Feedback indicated good overall acceptance of the seminars, with a median of 2 (score range from 1 = excellent to 6 = failed). CONCLUSION: Trained medical students are effective and well accepted EBM trainers in compulsory undergraduate seminars.     

Conditional reliability of admissions interview ratings: extreme ratings are the most informative

CONTEXT: Admissions interviews are unreliable and have poor predictive validity, yet are the sole measures of non-cognitive skills used by most medical school admissions departments. The low reliability may be due in part to variation in conditional reliability across the rating scale. OBJECTIVES: To describe an empirically derived estimate of conditional reliability and use it to improve the predictive validity of interview ratings. METHODS: A set of medical school interview ratings was compared to a Monte Carlo simulated set to estimate conditional reliability controlling for range restriction, response scale bias and other artefacts. This estimate was used as a weighting function to improve the predictive validity of a second set of interview ratings for predicting non-cognitive measures (USMLE Step II residuals from Step I scores). RESULTS: Compared with the simulated set, both observed sets showed more reliability at low and high rating levels than at moderate levels. Raw interview scores did not predict USMLE Step II scores after controlling for Step I performance (additional r2 = 0.001, not significant). Weighting interview ratings by estimated conditional reliability improved predictive validity (additional r2 = 0.121, P < 0.01). CONCLUSIONS: Conditional reliability is important for understanding the psychometric properties of subjective rating scales. Weighting these measures during the admissions process would improve admissions decisions.     

Medical students' personality characteristics and academic performance: a five-factor model perspective

OBJECTIVES: This study investigates: (1) which personality traits are typical of medical students as compared to other students, and (2) which personality traits predict medical student performance in pre-clinical years. DESIGN: This paper reports a cross-sectional inventory study of students in nine academic majors and a prospective longitudinal study of one cohort of medical students assessed by inventory during their first preclinical year and by university examination at the end of each pre-clinical year. SUBJECTS AND METHODS: In 1997, a combined total of 785 students entered medical studies courses in five Flemish universities. Of these, 631 (80.4%) completed the NEO-PI-R (i.e. a measure of the Five-Factor Model of Personality). This was also completed by 914 Year 1 students of seven other academic majors at Ghent University. Year end scores for medical students were obtained for 607 students in Year 1, for 413 in Year 2, and for 341 in Year 3. RESULTS: Medical studies falls into the group of majors where students score highest on extraversion and agreeableness. Conscientiousness (i.e. self-achievement and self-discipline) significantly predicts final scores in each pre-clinical year. Medical students who score low on conscientiousness and high on gregariousness and excitement-seeking are significantly less likely to sit examinations successfully. CONCLUSIONS: The higher scores for extraversion and agreeableness, two dimensions defining the interpersonal dynamic, may be beneficial for doctors' collaboration and communication skills in future professional practice. Because conscientiousness affects examination results and can be reliably assessed at the start of a medical study career, personality assessment may be a useful tool in student counselling and guidance.     

Influence of curriculum type on student performance in the United States Medical Licensing Examination Step 1 and Step 2 exams: problem-based learning vs. lecture-based curriculum

INTRODUCTION: The results of the United States Medical Licensing Examination Step 1 and 2 examinations are reported for students enrolled in a problem-based and traditional lecture-based curricula over a seven-year period at a single institution. There were no statistically significant differences in mean scores on either examination over the seven year period as a whole. There were statistically significant main effects noted by cohort year and curricular track for both the Step 1 and 2 examinations. These results support the general, long-term effectiveness of problem-based learning with respect to basic and clinical science knowledge acquisition. CONTEXT: This paper reports the United States Medical Licensing Examination Step 1 and Step 2 results for students enrolled in a problem-based and traditional lecture-based learning curricula over the seven-year period (1992-98) in order to evaluate the adequacy of each curriculum in supporting students learning of the basic and clinical sciences. METHODS: Six hundred and eighty-nine students who took the United States Medical Licensing Examination Step 1 and 540 students who took Step 2 for the first time over the seven-year period were included in the analyses. T-test analyses were utilized to compare students' Step 1 and Step 2 performance by curriculum groups. RESULTS: United States Medical Licensing Examination Step 1 scores over the seven-year period were 214 for Traditional Curriculum students and 208 for Parallel Curriculum students (t-value = 1.32, P=0.21). Mean Step 2 scores over the seven-year period were 208 for Traditional Curriculum students and 206 for Parallel Curriculum students (t-value=1.08, P=0.30). Statistically significant main effects were noted by cohort year and curricular track for both the Step 1 and Step 2 examinations. CONCLUSION: The totality of experience in both groups, although differing by curricular type, may be similar enough that the comparable scores are what should be expected. These results should be reassuring to curricular planners and faculty that problem-based learning can provide students with the knowledge needed for the subsequent phases of their medical education.     

Peer assessment of professional competence

BACKGROUND: Current assessment formats for medical students reliably test core knowledge and basic skills. Methods for assessing other important domains of competence, such as interpersonal skills, humanism and teamwork skills, are less well developed. This study describes the development, implementation and results of peer assessment as a measure of professional competence of medical students to be used for formative purposes. METHODS: Year 2 medical students assessed the professional competence of their peers using an online assessment instrument. Fifteen randomly selected classmates were assigned to assess each student. The responses were analysed to determine the reliability and validity of the scores and to explore relationships between peer assessments and other assessment measures. RESULTS: Factor analyses suggest a 2-dimensional conceptualisation of professional competence: 1 factor represents Work Habits, such as preparedness and initiative, and the other factor represents Interpersonal Habits, including respect and trustworthiness. The Work Habits factor had moderate, yet statistically significant correlations ranging from 0.21 to 0.53 with all other performance measures that were part of a comprehensive assessment of professional competence. Approximately 6 peer raters were needed to achieve a generalisability coefficient of 0.70. CONCLUSIONS: Our findings suggest that it is possible to introduce peer assessment for formative purposes in an undergraduate medical school programme that provides multiple opportunities to interact with and observe peers.     

The rating of pre-clerkship examination questions by postgraduate medical students: an assessment of quality and relevancy to medical practice

OBJECTIVE: To determine whether postgraduate students are able to assess the quality of undergraduate medical examinations and to establish whether faculty can use their results to troubleshoot the curriculum in terms of its content and evaluation. SUBJECTS: First and second year family medicine postgraduate students. MATERIALS: A randomly generated sample of undergraduate medical examination questions. METHODS: Postgraduate students were given two undergraduate examinations which included questions with an item difficulty (ID) > 0.60. The students answered and then rated each question on a scale of 1-7. RESULTS: The percentage of postgraduate students answering each question correctly correlated significantly with the average perceived relevance (Examination 1: r=0.372; P < 0.05; Examination 2: r=0.458; P < 0.05). Questions plotted for average postgraduate/undergraduate performance ratio versus the average perceived relevance were significantly correlated (Examination 1: r=0.462; P < 0.01; Examination 2: r=0.458; P < 0.05). CONCLUSIONS: This study offers a method of validating question appropriateness prior to examination administration. The design has the potential to be used as a model for determining the relevancy of a medical curriculum.     

Impact of a new curriculum on medical students' interpersonal and interviewing skills

OBJECTIVES: In 1998 we reported on the rise and fall of medical student communication skills during the 4 years of medical school. Since then, the University of Connecticut School of Medicine has completed a major curriculum renewal project with an emphasis on early clinical work, lifelong learning and more ambulatory training. The goals of this study were to compare students' interviewing and interpersonal skills in standardised patient (SP) assessments in the old and new curricula and to assess the success of the new curriculum in preventing a decline in student skills in this domain. METHODS: The clinical skills of 202 students were measured longitudinally during encounters with SPs in each of their 4 years of medical school. Students in this study and the earlier study were evaluated using the Arizona Clinical Interviewing Rating (ACIR) Scale. RESULTS: Compared with students from the previous curriculum, students on the new curriculum in this study showed an improvement in ACIR scores. Year 1 mean ACIR scores (1 = poor to 5 = excellent) were, respectively, 3.6 for the old curriculum cohort and 4.0 for the new curriculum group. In Year 4 the mean score for the old curriculum cohort was 3.7 and that for the new curriculum group was 3.8. Students on the new curriculum still showed a decline in ACIR scores from Years 1 to 4, but it was not as severe a decline as it had been previously. CONCLUSIONS: Pre-clinical medical students perform better on measures of interpersonal communication than their clinical counterparts. The students who participated in the new curriculum demonstrated an earlier acquisition of and a less steep decline in interviewing and interpersonal skills during the course of medical school.     

Competence in the musculoskeletal system: assessing the progression of knowledge through an undergraduate medical course

BACKGROUND: Professional bodies have expressed concerns that medical students lack appropriate knowledge in musculoskeletal medicine despite its high prevalence of use within the community. Changes in curriculum and teaching strategies may be contributing factors to this. There is little evidence to evaluate the degree to which these concerns are justified. OBJECTIVES: To design and evaluate an assessment procedure that tests the progress of medical students in achieving a core level of knowledge in musculoskeletal medicine during the course. PARTICIPANTS AND SETTING: A stratified sample of 136 volunteer students from all 5 years of the medical course at Sheffield University. METHODS: The progress test concept was adapted to provide a cross-sectional view of student knowledge gain during each year of the course. A test was devised which aimed to provide an assessment of competence set at the standard required of the newly qualified doctor in understanding basic and clinical sciences relevant to musculoskeletal medicine. The test was blueprinted against internal and external guidelines. It comprised 40 multiple-choice and extended matching questions administered by computer. Six musculoskeletal practitioners set the standard using a modified Angoff procedure. RESULTS: Test reliability was 0.6 (Cronbach's alpha). Mean scores of students increased from 41% in Year 1 to 84% by the final year. Data suggest that, from a baseline score in Year 1, there is a disparate experience of learning in Year 2 that evens out in Year 3, with knowledge progression becoming more consistent thereafter. All final year participants scored above the standard predicted by the Angoff procedure. CONCLUSIONS: This short computer-based test was a feasible method of estimating student knowledge acquisition in musculoskeletal medicine across the undergraduate curriculum. Tested students appear to have acquired a satisfactory knowledge base by the end of the course. Knowledge gain seemed relatively independent of specialty-specific clinical training. Proposals from specialty bodies to include long periods of disciplinary teaching may be unnecessary.     

Assessing student reflection in medical practice. The development of an observer-rated instrument: reliability, validity and initial experiences

INTRODUCTION: This study describes the development of an instrument to measure the ability of medical students to reflect on their performance in medical practice. METHODS: A total of 195 Year 4 medical students attending a 9-hour clinical ethics course filled in a semi-structured questionnaire consisting of reflection-evoking case vignettes. Two independent raters scored their answers. Respondents were scored on a 10-point scale for overall reflection score and on a scale of 0-2 for the extent to which they mentioned a series of perspectives in their reflections. We analysed the distribution of scores, the internal validity and the effect of being pre-tested with an alternate form of the test on the scores. The relationships between overall reflection score and perspective score, and between overall reflection score and gender, career preference and work experience were also calculated. RESULTS: The interrater reliability was sufficient. The range of scores on overall reflection was large (1-10), with a mean reflection score of 4.5-4.7 for each case vignette. This means that only 1 or 2 perspectives were mentioned, and hardly any weighing of perspectives took place. The values over the 2 measurements were comparable and were strongly related. Women had slightly higher scores than men, as had students with work experience in health care, and students considering general practice as a career. CONCLUSIONS: Reflection in medical practice can be measured using this semistructured questionnaire built on case vignettes. The mean score allows for the measurement of improvement by future educational efforts. The wide range of individual differences allows for comparisons between groups. The differences found between groups of students were as expected and support the validity of the instrument.     

Growth of analytical thinking skills over time as measured with the MATCH test

Context Ber’s Comprehensive Integrative Puzzle aims to assess analytical clinical thinking in medical students. We developed a paediatric version, the MATCH test, in which we added two irrelevant options to each question in order to reduce guessing behaviour. We tested its construct validity and studied the development of integrative skills over time. Methods We administered a test (MATCH 1) to subjects from two universities, both with a 6‐year medical training course. Subjects included 30 students from university 1 who had completed a paediatric clerkship in Year 4, 23 students from university 2 who had completed a paediatric clerkship in Year 5, 13 students from both universities who had completed an advanced paediatric clerkship in Year 6, 28 paediatric residents and 17 paediatricians. We repeated this procedure using a second test with different domains in a new, comparable group of subjects (MATCH 2). Results Mean MATCH 1 scores for the respective groups were: Year 4 students: 61.2% (standard deviation [SD] 1.3); Year 5 students: 71.3% (SD 1.6); Year 6 students: 76.2% (SD 1.5); paediatric residents: 88.5% (SD 0.7), and paediatricians: 92.2% (SD 1.1) (one‐way anova F = 104.00, P < 0.0001). Students of both universities had comparable scores. MATCH 1 and 2 scores were comparable. Cronbach’s α‐values in MATCH 1 and 2 were 0.92 and 0.91, respectively, for all subjects, and 0.82 and 0.87, respectively, for all students. Conclusions Analytical clinical thinking develops over time, independently of the factual content of the course. This implies that shortened medical training programmes could produce less skilled graduates.     

Active learning in a Year 2 pathology curriculum

PURPOSE: Team-based learning (TBL) has been successfully used in non-medical curricula, but its effectiveness in medical education has not been studied extensively. We evaluated the impact of TBL on the academic performance of Year 2 medical students at Wright State University by comparing this active learning strategy against a traditional method of case-based group discussion (CBGD). METHODS: A prospective crossover design assigned 83 Year 2 medical students to either CBGD or TBL for 8 pathology modules in the systems-based curriculum. The effectiveness of both learning methods was assessed by performance on pathology-based examination questions contained in end-of-course examinations. The highest and lowest academic quartiles of students were evaluated separately. Students' opinions of both methods were surveyed. RESULTS: No significant differences in whole group performance on pathology-based examination questions were observed as a consequence of experiencing TBL versus CBGD. However, students in the lowest academic quartile showed better examination performance after experiencing TBL than CBGD in 4 of 8 modules (P = 0.035). Students perceived that the contributions of peers to learning were more helpful during TBL than CBGD (P = 0.003). CONCLUSION: This study demonstrates that TBL and CBGD are equally effective active learning strategies when employed in a systems-based pre-clinical pathology curriculum, but students with lower academic performance may benefit more from TBL than CBGD.     

The development and validation of a knowledge, attitude and behaviour questionnaire to assess undergraduate evidence-based practice teaching and learning

OBJECTIVES: Most evidence-based practice (EBP) educational assessment tools evaluated to date have focused on specific knowledge components or technical skills. Other important potential barriers to the adoption of EBP, such as attitudinal, perceptual and behavioural factors, have yet to be studied, especially in the undergraduate setting. Therefore, we developed and validated a knowledge, attitude and behaviour questionnaire designed to evaluate EBP teaching and learning in an undergraduate medical curriculum. METHODS: We derived the questionnaire from a comprehensive literature review, informed by international and local experts and a Year 5 student focus group. We determined its factor structure and refined and validated the questionnaire according to the responses of a cohort of Year 5 and a combined group of Years 2 and 3 students using principal components factor analysis with varimax rotation. Factor reliability was computed using Cronbach's alpha coefficient. We assessed construct validity by correlating the factors with other measures of EBP activity and examined responsiveness through paired t-test of the pre/post factor mean scores. RESULTS: A 43-item questionnaire was developed. Four factors were identified from both student groups. The overall questionnaire as well as each factor had high construct validity (Cronbach's alpha > 0.7 for each scale). No significant correlations were found between the 4 factors, confirming their orthogonality. Positive correlations, however, resulted between factor mean scores and other EBP activities. The responsiveness of the questionnaire was satisfactory. CONCLUSION: A reliable knowledge, attitude and behaviour measure of EBP teaching and learning appropriate for undergraduate medical education has been developed and validated.     

The determination of the relevance of basic sciences learning objectives to clinical practice using a questionnaire survey

OBJECTIVES: Facilitating sufficient understanding of the basic sciences to underpin clinical practice is important in producing the good doctor. However, the inclusion of irrelevant material in the curriculum not only wastes valuable learning time, but may also hinder learning. The aim of this study was to determine how relevant staff and students thought respiratory basic science learning objectives were to medical practice. DESIGN: The study involved a survey using an anonymous questionnaire to determine whether the respiratory learning objectives stated in Year 1 were perceived as relevant to clinical practice. Each learning objective was rated as being 'relevant', 'not relevant' or of 'uncertain relevance'. SETTING: Dundee Medical School, UK. SUBJECTS: Junior and senior students and staff. RESULTS: Year 1 students considered the majority of the learning objectives to be relevant to clinical practice. Staff and senior students identified some respiratory learning objectives as not relevant to clinical practice, most of which were related to biochemistry. CONCLUSIONS: The identification of learning objectives with questionable relevance to clinical practice requires careful consideration to determine whether these should be removed from the course. Attention needs to be given to both the presentation and process by which material is delivered to students. Strategies to emphasise the clinical relevance of the basic sciences to students are discussed. Further research needs to be conducted to evaluate what knowledge is essential for producing good doctors.     

Faculty training in evidence-based medicine: improving evidence acquisition and critical appraisal

INTRODUCTION: Evidence-based medicine (EBM) integrates published clinical evidence with patient values and clinical expertise, the output of which is informed medical decision making. Key skills for evidence-based practice include acquisition and appraisal of clinical information. Faculty clinicians often lack expertise in these skills and are therefore unable to demonstrate this process for students and residents. METHODS: We conducted a yearlong case-based EBM workshop for 28 clinician educators, with precourse and postcourse evaluations of EBM resource use and literature appraisal skills. RESULTS: Of the original 28 participants, 26 completed the course. Self-assessed EBM resource use improved significantly. Self-reported EBM knowledge correlated with measured skill (r = 0.45), and both improved with the intervention (both p < .001). Higher EBM skills scores correlated with time logged on the course's EBM Web sites (r = 0.56; p < .05), workshop attendance rates (r = 0.55; p = .003), and fewer years since medical school graduation (r = -0.56; p < .005). DISCUSSION: An interactive, longitudinal, EBM course derived from a needs assessment can improve 2 skills important for evidence-based practice: online literature retrieval and critical appraisal skills.     

Medical students' learning of anatomy: memorisation, understanding and visualisation

INTRODUCTION: There is much subjective discussion, but few empirical data that explore how students approach the learning of anatomy. AIMS: Students' perceptions of successful approaches to learning anatomy were correlated with their own approaches to learning, quality of learning and grades. METHODS: First-year medical students (n = 97) studying anatomy at an Australian university completed an online survey including a version of the Study Process Questionnaire (SPQ) that measures approaches to learning. The quality of students' written assessment was rated using the Structure of Observed Learning Outcomes (SOLO) taxonomy. Final examination data were used for correlation with approaches and quality of learning. RESULTS: Students perceived successful learning of anatomy as hard work, involving various combinations of memorisation, understanding and visualisation. Students' surface approach (SA) scores (mean 30 +/- 3.4) and deep approach (DA) scores (mean 31 +/- 4.2) reflected the use of both memorisation and understanding as key learning strategies in anatomy. There were significant correlations between SOLO ratings and DA scores (r = 0.24, P < 0.01), between SA scores and final grades (r = - 0.30, P < 0.01) and between SOLO ratings and final grades (r = 0.61, P < 0.01) in the subject. CONCLUSIONS: Approaches to learning correlate positively with the quality of learning. Successful learning of anatomy requires a balance between memorisation with understanding and visualisation. Interrelationships between these three strategies for learning anatomy in medicine and other disciplines require further investigation.     

Unintentional failure to assess for experience in senior undergraduate OSCE scoring

Context One goal of undergraduate assessment is to test students’ (future) performance. In the area of skills testing, the objective structured clinical examination (OSCE) has been of great value as a tool with which to test a number of skills in a limited time, with bias reduction and improved reliability. But can OSCEs measure undergraduate internship expertise in basic clinical skills? Methods Undergraduate students (n = 32) were given a questionnaire listing 182 basic clinical skills. We asked them to score the number of times they had performed each skill during their internships (a 12‐month period in Year 6). We assessed the students at the end of Year 5 (before the start of their internships) and again at the start of Year 7 (undergraduate training takes 7 years in Belgium, with internships during Year 6), using a 14‐station OSCE assessing basic clinical skills. Global ratings were used to score performance. The relationship between internship experience and the OSCE Year 7 score was analysed using a linear regression model, controlling for variation in OSCE scores from Year 5. A multi‐level analysis was performed considering students as level‐1 units and stations as level‐2 units. Results Year 7 OSCE scores (post‐internships) were not affected by the number of times that students practised basic medical skills during their internships. Discussion Scores on OSCEs do not seem to reflect clinical expertise acquired during internships. Other more integrated assessment methods may prove to be more valid for testing final undergraduate skills levels.     

The impact of a modern medical curriculum on students' proposed behaviour on meeting ethical dilemmas

OBJECTIVE: To evaluate the impact of a modern medical curriculum on students' proposed behaviour on encountering ethical dilemmas. DESIGN: Cohort design. SETTING: University of Glasgow Medical School. SUBJECTS: The first intake of students into Glasgow's new curriculum (n = 238). Main outcome measure Student answers consistent with consensus professional judgement on the ethical dilemmas posed by the vignettes of the Ethics and Health Care Survey Instrument. RESULTS: The probability of giving a consensus answer was lowest pre-Year 1 and highest post-Year 1. It reduced slightly post-Years 3 and 5, but remained significantly higher than at pre-Year 1. The performance of students undertaking a 1-year intercalated BSc, however, appeared to regress on testing post-Year 4. CONCLUSIONS: While the first year of the curriculum had a positive impact on students, the remainder of the curriculum did not impact to the same extent. These findings support the recommendation that small group teaching, the predominant teaching method in Year 1, should be preferred to lecture and large group teaching, the predominant method of the remaining curricular years. Full integration of ethics and law teaching within the rest of the curriculum is recommended, particularly during the clinical years. This has training implications for all medical teachers involved in the curriculum. The assessment of ethics should be incorporated into all formal examinations. It is recommended that ethics be addressed as part of a wider approach to professionalism in order to promote integration.     

Origin bias of test items compromises the validity and fairness of curriculum comparisons

OBJECTIVE: To determine whether items of progress tests used for inter-curriculum comparison favour students from the medical school where the items were produced (i.e. whether the origin bias of test items is a potential confounder in comparisons between curricula). METHODS: We investigated scores of students from different schools on subtests consisting of progress test items constructed by authors from the different schools. In a cross-institutional collaboration between 3 medical schools, progress tests are jointly constructed and simultaneously administered to all students at the 3 schools. Test score data for 6 consecutive progress tests were investigated. Participants consisted of approximately 5000 undergraduate medical students from 3 medical schools. The main outcome measure was the difference between the scores on subtests of items constructed by authors from 2 of the collaborating schools (subtest difference score). RESULTS: The subtest difference scores showed that students obtained better results on items produced at their own schools. This effect was more pronounced in Years 2-5 of the curriculum than in Year 1, and diminished in Year 6. CONCLUSIONS: Progress test items were subject to origin bias. As a consequence, all participating schools should contribute equal numbers of test items if tests are to be used for valid and fair inter-curriculum comparisons.