Plus ça change,plus c’est pareil: Making a continued case for the use of MCQs in medical education

Despite the increased emphasis on the use of workplace-based assessment in competency-based education models, there is still an important role for the use of multiple choice questions (MCQs) in the assessment of health professionals. The challenge, however, is to ensure that MCQs are developed in a way to allow educators to derive meaningful information about examinees’ abilities. As educators’ needs for high-quality test items have evolved so has our approach to developing MCQs. This evolution has been reflected in a number of ways including: the use of different stimulus formats; the creation of novel response formats; the development of new approaches to problem conceptualization; and the incorporation of technology. The purpose of this narrative review is to provide the reader with an overview of how our understanding of the use of MCQs in the assessment of health professionals has evolved to better measure clinical reasoning and to improve both efficiency and item quality.     

Effect of curriculum reform on graduating student performance

With the aim of evaluating the efficacy of a new curriculum implemented in the Faculty of Medicine of Ribeir?o Preto (University of S?o Paulo, Brazil), a yearly objective assessment of both cognitive and practical skills of undergraduate (sixth year) students was performed. All graduating student underwent a multiple-choice questions (MCQs) test and groups of 18-20 students were randomly assigned to OSCEs for either clinical (real and standardized patients) or procedure (manikins) skills. The average MCQs score for the students graduating in the new curriculum (63.1, SD = 8.9, n = 261) was significantly higher (p < 0.001) than for the previous curriculum (55.3, SD = 8.1, n = 222). Results for practical exams showed that the new curriculum was associated with significantly improved performance in five out of the six stations for procedure skills, but in only two of the 10 clinical skills stations. Final evaluation of graduating students revealed deficiencies in the local curriculum and may serve as a guide to take measures to correct them.     

Crowdsourcing for assessment items to support adaptive learning

Abstract

Purpose: Adaptive learning requires frequent and valid assessments for learners to track progress against their goals. This study determined if multiple-choice questions (MCQs) “crowdsourced” from medical learners could meet the standards of many large-scale testing programs.

Methods: Users of a medical education app (Osmosis.org, Baltimore, MD) volunteered to submit case-based MCQs. Eleven volunteers were selected to submit MCQs targeted to second year medical students. Two hundred MCQs were subjected to duplicate review by a panel of internal medicine faculty who rated each item for relevance, content accuracy, and quality of response option explanations. A sample of 121 items was pretested on clinical subject exams completed by a national sample of U.S. medical students.

Results: Seventy-eight percent of the 200 MCQs met faculty reviewer standards based on relevance, accuracy, and quality of explanations. Of the 121 pretested MCQs, 50% met acceptable statistical criteria. The most common reasons for exclusion were that the item was too easy or had a low discrimination index.

Conclusions: Crowdsourcing can efficiently yield high-quality assessment items that meet rigorous judgmental and statistical criteria. Similar models may be adopted by students and educators to augment item pools that support adaptive learning.     

Multiple choice questions revisited

MCQs of the multiple true/false (MTF) variety were widely used in summative assessment 25 years ago. They could test a number of skills in addition to recall of factual knowledge, and were reliable, discriminatory, reproducible and cost-effective. However, there are now considerable doubts about their construct validity, mainly because of the varying responses of examinees to negative countermarking and the 'don't know' option, and the strategies they use when sitting examinations. Extended matching and one-from-five questions are now preferable, and negative countermarking is outmoded. MTF questions are still valuable in formative assessment and revision but are not recommended for summative examinations.     

Innovative learning: employing medical students to write formative assessments

Peninsula Medical School, UK, employed six students to write MCQ items for a formative applied medical knowledge item bank. The students successfully generated 260 quality MCQs in their six-week contracted period. Informal feedback from students and two staff mentors suggests that the exercise provided a very effective learning environment and that students felt they were 'being paid to learn'. Further research is under way to track the progress of the students involved in the exercise, and to formally evaluate the impact on learning.     

The importance of design thinking in medical education

Design thinking provides a creative and innovate approach to solve a complex problem. The discover, define, develop and delivery phases of design thinking lead to the most effective solution and this approach can be widely applied in medical education, from technology intervention projects to curriculum development. Participants in design thinking acquire essential transferable life-long learning skills in dealing with uncertainty and collaborative team working.     

Use of flawed multiple-choice items by the New England Journal of Medicine for continuing medical education

Physicians in the United States are required to complete a minimum number of continuing medical education (CME) credits annually. The goal of CME is to ensure that physicians maintain their knowledge and skills throughout their medical career. The New England Journal of Medicine (NEJM) provides its readers with the opportunity to obtain weekly CME credits. Deviation from established item-writing principles may result in a decrease in validity evidence for tests. This study evaluated the quality of 40 NEJM MCQs using the standard evidence-based principles of effective item writing. Each multiple-choice item reviewed had at least three item flaws, with a mean of 5.1 and a range of 3 to 7. The results of this study demonstrate that the NEJM uses flawed MCQs in its weekly CME program.     

Using concept maps and goal-setting to support the development of self-regulated learning in a problem-based learning curriculum

Problem-based learning (PBL) in medical education focuses on preparing independent learners for continuing, self-directed, professional development beyond the classroom. Skills in self-regulated learning (SRL) are important for success in PBL and ongoing professional practice. However, the development of SRL skills is often left to chance. This study presents the investigated outcomes for students when support for the development of SRL was embedded in a PBL medical curriculum. This investigation involved design, delivery and testing of SRL support, embedded into the first phase of a four-year, graduate-entry MBBS degree. The intervention included concept mapping and goal-setting activities through iterative processes of planning, monitoring and reflecting on learning. A mixed-methods approach was used to collect data from seven students to develop case studies of engagement with, and outcomes from, the SRL support. The findings indicate that students who actively engaged with support for SRL demonstrated increases in cognitive and metacognitive functioning. Students also reported a greater sense of confidence in and control over their approaches to learning in PBL. This study advances understanding about how the development of SRL can be integrated into PBL.     

Improving the fairness of multiple-choice questions: a literature review

The ubiquity of multiple-choice questions (MCQs) results from their efficiency and hence reliability. Cognitive knowledge assessed by MCQ predicts and correlates well with overall competence and performance but examinees and examiners alike frequently perceive MCQ-based testing as 'unfair'. Fairness is akin to defensibility and is an increasingly important concept in testing. It is dependent on psychometric adequacy, diligence of construction, attention to consequential validity and appropriate standard setting. There is a wealth of evidence that extended matching questions are the fairest format but MCQs should always be combined with practical assessments, as written testing emphasizes learning from written sources.     

A modified electronic key feature examination for undergraduate medical students: validation threats and opportunities

The purpose of our study was the development and validation of a modified electronic key feature exam of clinical decision-making skills for undergraduate medical students. Therefore, the reliability of the test (15 items), the item difficulty level, the item-total correlations and correlations to other measures of knowledge (40 item MC-test and 580 items of German MC-National Licensing Exam, Part II) were calculated. Based on the guidelines provided by the Medical Council of Canada, a modified electronic key feature exam for internal medicine consisting of 15 key features (KFs) was developed for fifth year German medical students. Long menu (LM) and short menu (SM) question formats were used. Acceptance was assessed through a questionnaire. Thirty-seven students from four medical schools voluntarily participated in the study. The reliability of the key feature exam was 0.65 (Cronbach's alpha). The items' difficulty level scores were between 0.3 and 0.8 and the item-total correlations between 0.0 and 0.4. Correlations between the results of the KF exam and the other measures of knowledge were intermediate (r between 0.44 and 0.47) as well as the learners' level of acceptance. The modified electronic KF examination is a feasible and reliable evaluation tool that may be implemented for the assessment of clinical undergraduate training.     

LectureKeepr: A novel approach to studying in the adaptive curriculum

Abstract

As medical students search for ways to squeeze more hours of learning into their day, many have turned to technology for quick and efficient ways to study. Most commonly this includes based apps for purchase that involve visual tasks, mental review and repetition. Tasks involve flashcard creation, answering multiple choice questions (MCQs) and schedule planning. They typically require students to visually interact with a computer or smartphone screen. Alternatively, auditory apps may not only offer the benefit of enhanced learning through review and repetition, but also address convenience, portability, and metacognitive development without physical and temporal requirements of visual presentation. Auditory learning is eyes and “hands-free” allowing the learner to contiguously engage in other activities such as physical exercise, traveling, eating, etc. In this randomized, case-comparison study, half of the students were exposed to audio lecture content before class using a mobile application called LectureKeepr (LK) and half were exposed to lecture content before class via written material prepared by the professor. Pre- and post-MCQ quizzes were used to measure differences between groups in knowledge improvement. In addition, student perceptions of the auditory learning experience were assessed by an on-line survey. ANOVA with repeated measures revealed that students in the LK intervention group performed better on the postquiz than the students in the written materials group.     

How we encourage faculty development

Most clinicians enjoy teaching medical students, but many have had little training as clinical teachers. The General Medical Council (GMC) in 'Good Medical Practice' states 'if you are involved in teaching you must develop the skills, attitudes and practices of a competent teacher' (GMC 2006). Mclean et?al.'s (2008) AMEE guide on faculty development outlines practice points for those responsible for developing their faculty's educational skills. In this article, we look at one health region, Tayside in East Scotland, where the University of Dundee, NHS Education for Scotland (NES) and NHS Tayside are collaborating to implement these practice points. This combined approach has proved to be effective in progressing staff development and recruiting additional clinical colleagues to develop their teaching role.     

Continuing medical education: AMEE Education Guide No 35

This guide is designed to provide a foundation for developing effective continuing medical education (CME) for practicing physicians. For the purposes of this work, continuing medical education is defined as any activity which serves to maintain, develop, or increase the knowledge, skills and professional performance and relationships that a physician uses to provide services for patients, the public, or the profession (American Medical Association 2007; Accreditation Council for CME 2007). The term continuing professional development (CPD) is broader and has become more popular in many areas of the world. As defined by Stanton and Grant, CPD includes educational methods beyond the didactic, embodies concepts of self-directed learning and personal development and considers organizational and systemic factors (Stanton & Grant 1997). In fact, this guide describes many modalities that may be defined as CME or CPD. In the interest of simplicity, we will use the term continuing medical education (CME) throughout, with the understanding that the same strategies may be applied to non-clinical continuing professional education. For those who do not work exclusively in CME, many terms and processes may be unfamiliar. This guide is intended to provide a broad overview of the discipline of CME as well as a pragmatic approach to the practice of CME. The format provides an overview of CME including history and rationale for the discipline, followed by a practical approach to developing CME activities, the management of the overall CME programme and finally, future trends. At the end of the guide you will find resources including readings, websites and professional associations to assist in the development and management of CME programmes.     

Designing effective on-line continuing medical education

The Internet, and new information and communication technologies available through the Internet, provides medical educators with an opportunity to develop unique on-line learning environments with real potential to improve physicians' knowledge and effect change in their clinical practice. There are approximately 100 websites offering on-line CME courses in the USA alone. However, few of these CME courses appear to be based on sound educational principles or CME research and may have little chance of achieving the broader goals of CME. The majority of these courses closely resemble their traditional counterparts (e.g. paper-based books are now electronic books) and appear to be mere substitutions for old-technology CME resources. Whilst some CME providers add unique features of the Internet to enrich their websites, they do not employ strategies to optimize the learning opportunities afforded by this new technology. The adoption of adult learning principles, reflective practice and problem-based approaches can be used as a foundation for sound CME course design. In addition, knowledge of Internet technology and the learning opportunities it affords, together with strategies to maintain participation and new assessment paradigms, are all needed for developing online CME. We argue for an evidence-based and strategic approach to the development of on-line CME courses designed to enhance physician learning and facilitate change in clinical behaviour.     

Sure,or unsure? Measuring students’ confidence and the potential impact on patient safety in multiple-choice questions

Background: Multiple-choice questions (MCQs) provide useful information about correct and incorrect answers, but they do not offer information about students’ confidence.

Methods: Ninety and another 81 medical students participated each in a curricular neurology multiple-choice exam and indicated their confidence for every single MCQ. Each MCQ had a defined level of potential clinical impact on patient safety (uncritical, risky, harmful). Our first objective was to detect informed (IF), guessed (GU), misinformed (MI), and uninformed (UI) answers. Further, we evaluated whether there were significant differences for confidence at correct and incorrect answers. Then, we explored if clinical impact had a significant influence on students’ confidence.

Results: There were 1818 IF, 635 GU, 71?MI, and 176 UI answers in exam I and 1453 IF, 613 GU, 92?MI, and 191 UI answers in exam II. Students’ confidence was significantly higher for correct than for incorrect answers at both exams (p?p?=?0.01). At exam II, students’ confidence was significantly higher for incorrect harmful than for incorrect benign (p?p?=?0.01).

Conclusions: We were pleased to see that there were more informed than guessed, more uninformed than misinformed answers and higher students’ confidence for correct than for incorrect answers. Our expectation that students state higher confidence in correct and harmful and lower confidence in incorrect and harmful MCQs could not be confirmed.     

Defining leadership competencies for pediatric critical care fellows: Results of a national needs assessment

Introduction: Physicians in training, including those in Pediatric Critical Care Medicine, must develop clinical leadership skills in preparation to lead multidisciplinary teams during their careers. This study seeks to identify multidisciplinary perceptions of leadership skills important for Pediatric Critical Care Medicine fellows to attain prior to fellowship completion.

Methods: We performed a multi-institutional survey of Pediatric Critical Care Medicine attendings, fellows, and nurses. Subjects were asked to rate importance of 59 leadership skills, behaviors, and attitudes for Pediatric Critical Care practitioners and to identify whether these skills should be achieved before completing fellowship. Skills with the highest ratings by respondents were deemed essential.

Results: Five hundred and eighteen subjects completed the survey. Of 59 items, only one item (“displays honesty and integrity”) was considered essential by all respondents. When analyzed by discipline, nurses identified 21 behaviors essential, fellows 3, and attendings 1 (p?p?Conclusions: Despite significant variability among Pediatric Critical Care attendings, fellows, and nurses in identifying which clinical leadership competencies are important for graduating Pediatric Critical Care fellows, they place the highest importance on skills in self-management and self-awareness. Leadership skills identified as most important may guide the development of interventions to improve trainee education and interprofessional care.     

Teaching smoking cessation knowledge to second-year undergraduates

Positive smoking cessation knowledge changes can be achieved through teaching. It is not known if, or how, increases in knowledge will contribute to actual interventions in practice. It is suggested that interventions to support medical undergraduates to develop the knowledge and skills required to support patients effectively in stopping smoking need to be part of an integrated programme of teaching clinical knowledge, communication and clinical skills, coupled with opportunities to practise in simulated situations.     

Student learning strategies, mental models and learning outcomes in problem-based and traditional curricula in medicine

Kelson and Distlehorst (2000) state the PBL outcomes should be discussed on a broad basis including a useable knowledge base, skills in problem solving, self-directed learning and collaboration. The present research compares students in a PBL programme and in a traditional program on learning strategies, mental models and outcomes. Learning strategies and metal models of learning were measured for two student groups in 2001 using an adaptation of Vermunt's (1994) "Inventory of learning styles". Learning outcomes were measured for the same groups in 2002 using a test of medical knowledge. PBL-students showed significantly more self-regulated learning and more constructive conceptions of learning. No significant differences in learning outcomes were found between the two groups. Students in the PBL-programme perceived students as more active contributors to group learning process and made use of a broader range of resources than students in the traditional programme. The findings confirm effects of educational programmes on student learning strategies (Vermetten 1999) and also confirm lack of significant differences in medical knowledge (Albanese 2000, Colliver 2000, Norman & Schmidt 2000, Wiers-Jensen & Aasland 2004). The curricular influence on students' learning strategies challenges educators to design approaches that promote lifelong learning skills as well as disciplinary knowledge.     

AMEE Medical Education Guide No. 15: Problem-based learning: a practical guide

This practical guide for health professions teachers provides a perspective of one of the most important educational developments in the past 30 years.Problem-based learning (PBL) is a continuum of approaches rather than one immutable process. It is a teaching method that can be included in the teacher's tool-kit along with other teaching methods rather than used as the sole educational strategy.PBL reverses the traditional approach to teaching and learning. It starts with individual examples or problem scenarios which stimulate student learning. In so doing, students arrive at general principles and concepts which they then generalize to other situations. PBL has many advantages. It facilitates the acquisition of generic competences, encourages a deep approach to learning and prepares students for the adult learning approach they need for a lifetime of learning in the health care professions. It is also fun. PBL helps in curriculum planning by defining core, ensuring relevance of content, integrating student learning and providing prototype cases. There are also drawbacks associated with PBL. Students may fail to develop an organized framework for their knowledge. The PBL process may inhibit good teachers sharing their enthusiasm for their topic with students and student identification with good teachers.Teachers may not have the skills to facilitate PBL.The problem scenario is of crucial significance. It should engage the students' interest and be skilfully written. While the medium selected for presentation of the scenario is usually print, other media may be used.The clinical tasks carried out by the student may replace the problem scenario as the focus for learning.Students are supported during the PBL process by tutors and/or study guides.The amount of support required is inversely related to the students' prior learning and understanding of the PBL process. A range of additional learning resources and opportunities may be made available to the students, including textbooks, videotapes, computer-based material, lectures and clinical sessions. Tutors require group facilitation skills, an understanding of the PBL process and knowledge of the course and of the curriculum in general.They need special personal qualities and it is preferable if they have expertise in the content area.While special assessment processes have been developed to assess students learning by the PBL method, the general principles of assessment apply to PBL courses and a mixed menu of assessment methods needs to be employed. Curriculum design involves a skilful blend of educational strategies designed to help students achieve the curriculum outcomes. PBL may make a valuable contribution to this blend but attention needs to be paid to how it is implemented.