AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician

Two important features of contemporary medical education are recognized. The first is an emphasis on assessment as a tool to ensure quality in training programmes, to motivate students and to direct what they learn. The second is a move to outcome-based education where the learning outcomes are defined and decisions about the curriculum are based on these. These two trends are closely related. If teachers are to do a better job of assessing their students, they need an understanding of the assessment process, an appreciation of the learning outcomes to be assessed and a recognition of the most appropriate tools to assess each outcome. Assessment tools selected should be valid, reliable, practical and have an appropriate impact on student learning. The preferred assessment tool will vary with the outcome to be assessed. It is likely to be some form of written test, a performance test such as an OSCE in which the student's competence can be tested in a simulated situation, and a test of the student's behaviour over time in clinical practice, based on tutors' reports and students' portfolios. An assessment profile can be produced for each student which highlights the learning outcomes the student has achieved at the required standard and other outcomes where this is not the case. For educational as well as economic reasons, there should be collaboration across the continuum of education in test development as it relates to the assessment of learning outcomes and in the implementation of a competence-based approach to assessment.     

OSCE: the Dundee experience

The Dundee Medical School has bean running OSCEs since 1977. In 1995, an integrated systems-based spiral curriculum on the core and options model was introduced. In 1997, outcome-based education was introduced as the basis for instruction, with a task-based educational strategy employed for students in years 4 and 5. This blend of educational strategies was considered in the design of the student assessment process. Assessment instruments, appropriate for use at each of the four levels of Miller's pyramid, were identified and included in the assessment process. The OSCE was used for summative assessment of students at the level of 'shows how' or simulation in years 2, 3 and 4. A year 2 OSCE is described here. Features of the Dundee OSCE are identified, relating to number and length of individual stations, practicalities or assessing a year group of students without student contamination with examination information and the blueprints used to design the examinations. Suggestions made for future development of the OSCE include the OSSE, the Objective Structured Selection Examination, and an exploration of the potential of the OSCE to assess attitudes, personal attributes and professionalism. The need is identified for a platform to debate issues such as should individual medical schools attempt to achieve national test centre standards with their examinations.     

The outcomes of global minimum essential requirements (GMER) pilot implementation in China

Defining global standards for medical education in the form of competencies and the methods to evaluate whether an individual student possesses these competencies at graduation has long been a dream of some medical educators. The development of such standards, the methods to assess their presence and the pilot test study of the standards in graduating students at eight medical schools in China, as well as the process for establishing student and school performance "cut points", has been previously described. This paper reports on the performance of a single student who went through the assessment process, the performance of all students at one of the eight medical schools and the collective performance of all students at all eight medical schools. The actual quantitative data is presented, as is the conclusion of where the student, the school and all schools had strengths, where they were borderline in performance and/or where they need improvement. The results are serving as a blueprint for medical education reform in China. Implications of the pilot test and the entire process are discussed, as is the potential for global adoption of outcome based assessments.     

Cross institutional collaboration in assessment: a case on progress testing

The practice of assessment is governed by an interesting paradox. On the one hand good assessment requires substantial resources which may exceed the capacity of a single institution and we have reason to doubt the quality of our in-house examinations. On the other hand, our parsimonity with regard to our resources makes us reluctant to pool efforts and share our test material. This paper reports on an initiative to share test material across different medical schools. Three medical schools in The Netherlands have successfully set up a partnership for a specific testing method: progress testing. At present, these three schools collaboratively produce high-quality test items. The jointly produced progress tests are administered concurrently by these three schools and one other school, which buys the test. The steps taken in establishing this partnership are described and results are presented to illustrate the unique sort of information that is obtained by cross-institutional assessment. In addition, plans to improve test content and procedure and to expand the partnership are outlined. Eventually, the collaboration may even extend to other test formats. This article is intended to give evidence of the feasibility and exciting potential of between school collaboration in test development and test administration. Our experiences have demonstrated that such collaboration has excellent potential to combine economic benefit with educational advantages, which exceed what is achievable by individual schools.     

Correlation of a written test of skills and a performance based test: a study in two traditional medical schools

Studies in innovative curricula have shown that a written test of skills is potentially able to predict OSCE scores. In this study we verified the potential of a written test of skills as an alternative to an OSCE in two traditional medical schools. A 12 station OSCE was scored using detailed checklists and global rating scales. The written test of skills consisted of 132 true-false questions. Students were assessed immediately before graduation in 1997. The size of the reported correlations confirms that the written test of skills can be used to predict performance-based test scores in traditional medical schools to some extent. For research purposes, particularly for outcome research to compare curricula of medical schools, a written test of skills is a viable alternative to performance-based testing. A written test of skills is unable to replace the OSCE for assessment of individual students.     

Post-examination interpretation of objective test data: monitoring and improving the quality of high-stakes examinations--a commentary on two AMEE Guides

As great emphasis is rightly placed upon the importance of assessment to judge the quality of our future healthcare professionals, it is appropriate not only to choose the most appropriate assessment method, but to continually monitor the quality of the tests themselves, in a hope that we may continually improve the process. This article stresses the importance of quality control mechanisms in the exam cycle and briefly outlines some of the key psychometric concepts including reliability measures, factor analysis, generalisability theory and item response theory. The importance of such analyses for the standard setting procedures is emphasised. This article also accompanies two new AMEE Guides in Medical Education (Tavakol M, Dennick R. Post-examination Analysis of Objective Tests: AMEE Guide No. 54 and Tavakol M, Dennick R. 2012. Post examination analysis of objective test data: Monitoring and improving the quality of high stakes examinations: AMEE Guide No. 66) which provide the reader with practical examples of analysis and interpretation, in order to help develop valid and reliable tests.     

Developing performance-based medical school assessment programs in resource-limited environments

While the use of performance-based assessments in medical schools is widespread, there are unique issues associated with developing and establishing simulated clinical examinations when resources are limited. The faculty at the National University of Cuyo (NUC) in Mendoza, Argentina, has successfully implemented an OSCE as part of the medical school graduation requirements. This paper provides an overview of the issues faced by NUC faculty in setting up a standardized patient (SP)-based assessment with only limited resources, and contrasts their experiences with those of other groups, including certification and licensure bodies who have implemented similar types of assessments. Despite their novelty and a lack of technological, staffing, and budgetary resources, with adequate planning, an emphasis on faculty involvement, quality case development, local SP recruitment and training activities, educational programs focused on assessment, and flexibility in creating a physical space to conduct the exam, an effective program for assessing the clinical skills of medical students can be realized.     

The versatility of progress testing assessed in an international context: a start for benchmarking global standardization?

Sharing and collaboration relating to progress testing already takes place on a national level and allows for quality control and comparisons of the participating institutions. This study explores the possibilities of international sharing of the progress test after correction for cultural bias and translation problems. Three progress tests were reviewed and administered to 3043 Pretoria and 3001 Maastricht medical students. In total, 16% of the items were potentially biased and removed from the test items administered to the Pretoria students (9% due to translation problems; 7% due to cultural differences). Of the three clusters (basic, clinical and social sciences) the social sciences contained most bias (32%), basic sciences least (11%). The differences that were found, comparing the student results of both schools, seem a reflection of the deliberate accentuations that both curricula pursue. The results suggest that the progress test methodology provides a versatile instrument that can be used to assess medical schools across the world. Sharing of test material is a viable strategy and test outcomes are interesting and can be used in international quality control.     

The prospects of sitting ‘end of year’ open book exams in the light of COVID-19: A medical student’s perspective

Abstract

Medical schools in the UK have been forced to dramatically restructure teaching and assessment amidst the Coronavirus (COVID-19) pandemic. As part of this, some have opted to assess progression through open book examinations (OBE). I aim to share my thoughts as an unsettled 4th year medical student about to embark on my first set of clinical exams conducted in this format. The difficulties associated with preparing for examinations under such unique and challenging circumstances cannot be underestimated. Working from home, during social distancing, surrounds students with the extra family stresses that we are all facing during this pandemic. This combined with a new, unfamiliar examination format will inevitably lead to students feeling daunted. While some would argue that an OBE may reward good problem solvers, students still require a strong foundation of knowledge. The luxury of reference will not be afforded in all clinical settings thus leading to concerns regarding students skimming over essential learning points. Furthermore, we cannot ignore the increased opportunity for academic misconduct resulting from an open book assessment format. Why are medical schools placing undue stress on students who could instead focus their attention on living compassionately for others during this difficult time?     

AMEE Medical Education Guide No. 23 (Part 2): Curriculum,environment, climate,quality and change in medical education - a unifying perspective

This paper looks at five focal terms in education - curriculum, environment, climate, quality and change - and the interrelationships and dynamics bemeen and among them. It emphasizes the power and utility of the concept of climate as an operationalization or manifetation of the curriculum and the other three concepts. Ideas pertaining w the theory of climate and its measurement can provide a greater understanding of the medical cumadurn. The environment is an impoltant detemzinant of behaviour. Environment is perceived by students and it is perceptions of environment that are related w behaviour. The environment, as perceived, may be designated as climate. It is argued that the climate is the soul and spirit of the medical school environment and curriculum. Students' experiences of the climate of their medical education environment are related w their achievements, sangaction and success. Measures of educational climate are reviewed and the possibilities of new climate measures for medical education are discussed. These should take account of current trends in medical education and curricula. Measures of the climate may subdivide it inw dzfferent components giving, for example, separate assessment of so-called Faculty Press, Student Press, Administration Press and Physical or Material Environmental Press. Climate measures can be used in different modes with the same stakeholders. For example, students may be asked to report, first, their perceptions of the actual environment they have experienced and, second, w report on their ideal or preferred environment. The same climate index can be used with different stakeholders giving, for example, staff and student comparisons. The climate is important for staff as well as for students. The organizational climate that teaching staff experience in the work environment that they inhabit is important for their well-being, and that of their students. The medical school is a learning organization evolving and changing in the illuminative evaluation it makes of its environment and its curriculum through the action research studies of its climate. Consderations of climate in the medical school along the lines of continuous quality improvement and innovation are likely to further the medical school as a learning organization with the attendant benefits. Unless medical schools become such learning organizations their quality of health and their longevity may be threatened.     

The foundations of measurement and assessment in medical education

As a medical educator, you may be directly or indirectly involved in the quality of assessments. Measurement has a substantial role in developing the quality of assessment questions and student learning. The information provided by psychometric data can improve pedagogical issues in medical education. Through measurement we are able to assess the learning experiences of students. Standard setting plays an important role in assessing the performance quality of students as doctors in the future. Presentation of performance data for standard setters may contribute towards developing a credible and defensible pass mark. Validity and reliability of test scores are the most important factors for developing quality assessment questions. Analysis of the answers to individual questions provides useful feedback for assessment leads to improve the quality of each question, and hence make students’ marks fair in terms of diversity and ethnicity. Item Characteristic Curves (ICC) can send signals to assessment leads to improve the quality of individual questions.     

Unexpected pathological findings in skills training and assessing skills

This article draws attention to unexpected pathological findings encountered by students and teachers when examining one another and/or simulated patients in skills training and assessment sessions.Although no literature on the subject was found, it appears to be not uncommon for students and teachers to find abnormalities during skills training and assessment. It is important that both students and teachers should be aware of this possibility and students should be encouraged to disclose any unexpected findings. Teachers should be given guidelines on the appropriate course of action during teacher-training sessions and students should be informed of the possibility of finding abnormal pathology prior to their participation in skills training and assessment.     

Peer review improves psychometric characteristics of multiple choice questions

Abstract

Purpose: For new and emerging medical schools, developing a system to peer-review and evaluate the assessment processes through faculty development programs can be a challenge. This study evaluates the impact of peer-review practices on item analysis, reliability, and the standard error of measurement of multiple-choice questions for summative final examinations.

Methods: This study used a retrospective cohort design of two consecutive academic years in 2012 and in 2013. Psychometric analyses of multiple-choice questions of three summative final examinations in Medicine, Pediatrics, and Surgery for sixth year medical students at the College of Medicine Taif University were used. Formal peer review of multiple-choice questions began in 2013, using guidelines from the National Board of Medical Examiners. Psychometric analyses of multiple-choice questions included item analysis (item difficulty and item discrimination) and calculation of internal-consistency reliability and the standard error of measurement. Data analyses were conducted using Stata.

Results: Results showed significant improvement in psychometric indices, particularly item discrimination and reliability by .14 and .12 points, respectively, following the implementation of the peer review process across the three exams. Item difficulty remained unchanged for Pediatrics and Surgery.

Conclusion: Peer-review practices of multiple-choice questions using guidelines can lead to improved psychometric characteristics of items; these findings have implications for faculty development programs in improving item quality, particularly for medical schools in early stages of transforming assessment practices.     

AMEE Medical Education Guide No. 23 (Part 1): Curriculum,environment, climate,quality and change in medical education-a unifying perspective

This paper looks at five focal terms in education - curriculum, environment, climate, quality and change - and the interrelationships and dynamics between and among them. It emphasizes the power and utility of the concept of climate as an operationalization or manifestation of the curriculum and the other three concepts. Ideas pertaining to the theory of climate and its measurement can provide a greater understanding of the medical curriculum. The learning environment is an important determinant of behaviour. Environment is perceived by students and it is perceptions of environment that are related to behaviour. The environment, as perceived, may be designated as climate. It is argued that the climate is the soul and spirit of the medical school environment and curriculum. Students' experiences of the climate of their medical education environment are related to their achievements, satisfaction and success. Measures of educational climate are reviewed and climate measures for medical education are discussed. These should take account of current trends in medical education and curricula. Measures of the climate may subdivide it into different components giving, for example, a separate assessment of so-called Faculty Press, Student Press, Administration Press and Physical or Material Environmental Press. Climate measures can be used in different modes with the same stakeholders. For example, students may be asked to report, first, their perceptions of the actual environment they have experienced and, second, to report on their ideal or preferred environment. The same climate index can be used with different stakeholders giving, for example, staff and student comparisons. In addition to the educational climate of the environment that students inhabit, it is important to consider the organizational climate of the work environment that staff inhabit. This organizational climate is very significant, not only for staff, but for their students, too. The medical school is a learning organization evolving and changing in the illuminative evaluation it makes of its environment and its curriculum through the action research studies of its climate. Considerations of climate in the medical school, along the lines of continuous quality improvement and innovation, are likely to further the medical school as a learning organization with the attendant benefits. Unless medical schools become such learning organizations, their quality of health and their longevity may be threatened.     

Measurement practices: methods for developing content-valid student examinations

Measurement experts generally agree that a systematic approach to test construction will probably result in an instrument with sound psychometric properties. One fundamental method is called the blueprint approach to test construction. A test blueprint is a tool used in the process for generating content-valid exams by linking the subject matter delivered during instruction and the items appearing on the test. Unfortunately, this procedure as well as other educational measurement practices is often overlooked A survey of curriculum administrators at 144 United States and international medical schools was conducted to assess the importance and prevalence of test blueprinting in their school. Although most found test blueprinting to be very important, few require the practice. The purpose of this paper is to review the fundamental principals associated with achieving a high level of content validity when developing tests for students. The short-term efforts necessary to develop and integrate measurement theory into practice will lead to long-term gains for students, faculty and academic institutions.     

Does a summative portfolio foster the development of capabilities such as reflective practice and understanding ethics? An evaluation from two medical schools

Portfolios need to be evaluated to determine whether they encourage students to develop in capabilities such as reflective practice and ethical judgment. The aims of this study were (i) to determine whether preparing a portfolio helps promote students' development in a range of capabilities including understanding ethical and legal principles, reflective practice and effective communication, and (ii) to determine to what extent the format of the portfolio affected the outcome by comparing the experiences of students at two different medical schools. A questionnaire was designed to evaluate undergraduate medical students' experiences of completing a portfolio at two medical schools. A total of 526 (45% response rate) students answered the on-line questionnaire. Students from both medical schools gave the highest ranking for the portfolio as a trigger for reflective practice. 63% of students agreed their portfolio helped them develop reflective practice skills (p?相似文献   

The multiple mini-interviews as a predictor of peer evaluations during clinical training in medical school*

Background: Multiple Mini-Interviews (MMI) is an admissions tool being used for more than a decade by medical schools worldwide to assess non-cognitive skills. In Israel, in addition to the MMI tool, two questionnaires are given to the candidates. It has been necessary to find a relevant criterion to examine the predictive validity of the MMI test. The measure developed in this study was a peer assessment tool which was filled out during clerkship.

Aims: The study aims at evaluating the predictive validity of the MMI and two questionnaires when compared with the peer assessment tool.

Method: Ninety nine fourth-year medical students were included in this study. All of them went through the MMI test before the beginning of their studies. The students participated in two clinical rotations and were divided into 10 groups, each consisting of 9–12 students. This study is based on assessments given to 99 students. Every member of a group evaluated his fellow members on a number of attribute. The questionnaire included 10 statements, related to a particular attributes. In addition two concluding questions were included. Pearson correlations were used to test the relationships between pre-admission variables (MMI and two questionnaires) and the peer assessments.

Results: Statistically significant, moderate correlations between the general MOR and MIRKAM scores and the sum of the questionnaire assessments were 0.39 and 0.37, respectively (P-value <0.05).

Conclusions: The study provides important evidence for the validity of the MMI. Additional studies are required to reevaluate the predicted validity of the MMI.     

Weak self-directed learning skills hamper performance in cumulative assessment

Background: Self-regulated learning is an important determinant of academic performance. Previous research has shown that cumulative assessment encourages students to work harder and improve their results. However, not all students seem to respond as intended. We investigated the influence of students’ behavioral traits on their responsiveness to a cumulative assessment strategy.

Method: The cumulative test results of a third-year integrated ten-week course unit were analyzed. The test was divided into three parts delivered at 4, 8 and 10 weeks. Low starters (below median) with low or high improvement (below or above the median) were identified and compared regarding their behavioral traits (assessed with the Temperament and Character Inventory questionnaire).

Results: A total of 295 students filled out the questionnaire. A percentage of 70% of the students below the median on the first two test parts improved during the final part. Students who were less responsive to improve their test results, scored low only on the TCI scale “self directedness” (t?=?2.49; p?=?0.011).

Conclusion: Behavioral traits appear to influence student reactions to feedback on test results, with students with low self-directedness scores being particularly at risk. They can thus be identified and should receive special attention from student counselors.     

Portfolio as a method for continuous assessment in an undergraduate health education programme

A portfolio assessment system has been introduced into a biomedical science programme to promote both continuous learning and deep approaches to learning. Attention has been focused on creating harmony between the assessment system and the PBL curriculum of the programme. Biomedicine and laboratory work are central in the curriculum. The portfolio included evidence of laboratory work, personal reflections and certificates from the PBL tutor. The portfolio was assessed on three occasions over 20 weeks. The grades were 'pass' or 'fail'. The tutor certificate appeared to be a crucial part of the portfolio since a 'fail' in this part usually led to an overall 'fail'. Both students and teachers were concerned about ensuring that enough factual knowledge, as measured by a traditional test, had been achieved. The agreement was good enough for the pass or fail level but some expected differences were found at the detailed level. The course, including the portfolio, was evaluated orally during weekly whole-group meetings and using a questionnaire at the end. The students felt comfortable with the portfolio system and preferred it to a traditional test. The teachers felt that they needed to develop their teacher-student discussion skills and to improve their feedback on the reflections. Peer assessment between students is proposed as a line of action to enhance the credibility of the crucial tutor certificate. The portfolio might be an efficient tool for the students to concentrate their efforts on the most central concepts of medical laboratory work. The model will be developed through further discussions and better consensus among faculty.     

Evaluation and assessment of social accountability in medical schools

Social accountability as an outcome must be measured at institutional (evaluation) and individual (assessment) level. The definitions used in the measurement will be dependent on the social setting of the medical school being scrutinised. A formal framework is needed so that comparisons can be made and progress measured. The World Health Organisation suggests that there are four principles that delineate social accountability--relevance, quality, cost-effectiveness and equity. Medical schools are evaluated according to their planning, doing and impact in relation to these principles. Boelen and Woollard have clarified the ideas of planning, doing and impact into Conceptualisation, Production and Usability. THEnet group of medical schools use a shortened version of Boelen and Woollard's framework with 20 criteria to evaluate their programmes. At the individual level, there is considerable overlap between the concepts of 'social accountability' and 'professionalism'. Attempts are being made to define and measure professionalism, however, if the behaviour and attitudes of individual graduates is a significant component in defining social responsibility new methods of assessment need to be developed. Factors such as the available resources and the structure of the health services have a very large effect on doctors' attitudes and behaviour. As a result, the task of evaluating and assessing the extent to which medical schools are socially accountable is complex. Judgments on how well a school is meeting the standards will have to make allowance for the local political and economic environment.