Impact of online toxicology training on health professionals: the Global Educational Toxicology Uniting Project (GETUP)

Objective: The Global Educational Toxicology Uniting Project (GETUP), supported by the American College of Medical Toxicology, links countries with and without toxicology services via distance education with the aim to improve education. Due to the lack of toxicology services in some countries there is a knowledge gap in the management of poisonings. We describe our experience with the worldwide delivery of an online introductory toxicology curriculum to emergency doctors and other health professionals treating poisoned patients.

Methods: We delivered a 15-module introductory Internet-based toxicology curriculum to emergency doctors and health professionals, conducted from August to December 2016. This Internet-based curriculum was adapted from one used to teach emergency residents toxicology in the United States. Modules covered themes such as pharmaceutical (n?=?8), toxidromes (n?=?2) and agrochemicals (n?=?5) poisoning. Participants completed pre-test and post-test multiple choice questions (MCQs) before and after completing the online module, respectively, throughout the course. We collected information on participant demographics, education and training, and perception of relevance of the curriculum. Participants gave feedback on the course and how it affected their practice.

Results: One hundred and thirty-six health professionals from 33 countries participated in the course: 98 emergency doctors/medical officers, 25 physicians, eight pharmacists/poisons information specialists, two toxicologists, two medical students and one nurse. Median age of participants was 34 years. Median number of years postgraduate was seven. Ninety (65%) had access to either a poisons information centre over the phone or toxicologist and 48 (35%) did not. All participants expected the course to help improve their knowledge. Overall median pre-module MCQ scores were 56% (95%CI: 38, 75%) compared to post-module MCQ scores median 89% (95% CI: 67, 100%) (p?Conclusions: Our participants demonstrated an increase in medical knowledge based on performance on MCQs. An online toxicology curriculum is an effective way to deliver education to health professionals treating poisoned patients and can help to bridge the knowledge gap and change practice in developed and developing countries.     

Do medical students studying in the United Kingdom have an adequate factual knowledge of basic life support?

BACKGROUND: Healthcare professionals have a duty to maintain basic life support (BLS) skills. This study aims to evaluate medical students’ factual knowledge of BLS and the training they receive. METHODS: A cross-sectional, closed-response questionnaire was distributed to the first- and fourth-year students studying at institutions in the United Kingdom. The paper questionnaire sought to quantify respondent’s previous BLS training, factual knowledge of the BLS algorithm using five multiple choice questions (MCQs), and valuate their desire for further BLS training. Students received 1 point for each correctly identified answer to the 5 MCQ’s. RESULTS: A total of 3,732 complete responses were received from 21 medical schools. Eighty percent (n=2,999) of students completed a BLS course as part of their undergraduate medical studies. There was a significant difference (P<0.001) in the percentage of the fourth-year students selecting the correct answer in all the MCQ’s compared to the first-year students except in identifying the correct depth of compressions required during CPR (P=0.095). Overall 10.3% (95% CI 9.9% to 10.7%) of respondents correctly identified the answer to 5 MCQ’s on BLS: 9% of the first-year students (n=194) and 12% of the fourth-year students (n=190). On an institutional level the proportion of students answering all MCQ’s correctly ranged from 2% to 54% at different universities. Eighty-one percent of students (n=3,031) wished for more BLS training in their curriculum. CONCLUSION: Factual knowledge of BLS is poor among medical students in the UK. There is a disparity in standards of knowledge across institutions and respondents indicating that they would like more training.     

How do medical students learn: Medical student learning styles and factors that affect these learning styles

Medical students have individual learning styles, but these have been shown to change and develop as students mature and progress through their medical education. The purpose of this research project was to study the factors that influence the learning styles and activities of undergraduate medical students. We hypothesized that undergraduate medical students use different learning approaches and that their learning approaches depend on individual student characteristics: sex, age, and previous premedical education. In particular, we hypothesized that final‐year medical students who had global assessments of their performance would have higher deep learning scores and lower surface learning scores than would 3rd‐year medical students in their preclerkship year who must write a comprehensive, multiple‐choice question (MCQ) final examination. The Study Process Questionnaire was included and produced scores for surface, deep, and achieving learning approaches. The students who were surveyed had predominantly deep learning approaches; scores for surface learning were second highest, and scores for achieving learning were the lowest. There were correlations between the learning approaches and (a) student motives to “do well”; in medical school, (b) student study methods, and (c) their academic performance. Student approaches to learning were influenced by their age, premedical university education, and career choices. There were no differences between the 3rd‐ and 4th‐year students. Students also indicated that their study methods were determined by the assessment methods they encountered. Students indicated that in the preclinical years, much of their study was rote memorization. They indicated their study for MCQ examinations fostered surface learning; whereas, studying for free‐response written questions and clinical examinations encouraged deep learning.     

Assessing Advanced Communication Skills via Objective Structured Clinical Examination: A Comparison of Faculty Versus Self,Peer, and Standardized Patient Assessors

Abstract

Construct: The construct addressed in this study is assessment of advanced communication skills among senior medical students. Background: The question of who should assess participants during objective structured clinical examinations (OSCEs) has been debated, and options discussed in the literature have included peer, self, standardized patient, and faculty assessment models. What is not known is whether same-level peer assisted learning can be utilized for formative assessment of advanced communication skills when no faculty, standardized patients, or other trained assessors are involved in providing feedback. If successful, such an educational model would optimize resource utilization and broaden the scope of topics that could be covered in formative OSCEs. Approach: The investigators developed a 4-station formative OSCE focused on advanced communication skills for senior medical students, and evaluated the concordance of assessment done by same-level peers, self, standardized patients, and faculty for 45 students. After each station, examinees completed a self-assessment checklist and received checklist-based assessment and verbal feedback from same-level peers only. Standardized patients completed checklist-based assessments outside the room, and faculty did so after the OSCE via video review; neither group provided direct feedback to examinees. The investigators assessed inter-rater agreement and mean difference scores on the checklists using faculty score as the gold standard. Findings: There was fair to good overall agreement among self, same-level peer, standardized patient, and faculty-assessment of advanced communication skills. Relative to faculty, peer and standardized patient assessors overestimated advanced communication skills, while self-assessments underestimated skills. Conclusions: Self and same-level peer-assessment may be a viable alternative to faculty assessment for a formative OSCE on advanced communication skills for senior medical students.     

Teaching medical students to resuscitate children: An innovative two‐part programme

Objective: The objective of the present study was to design and evaluate a novel, ‘blended learning’ approach to the teaching of paediatric resuscitation to medical students. Methods: Participants were recruited from the Graduate Medical Program at the University of Sydney. The course incorporated an initial e‐learning module and a subsequent practical component. The e‐learning module taught basic and advanced life support. Students then attended a 90 min practical session, which focussed on team work and the psychomotor components of resuscitation. Improvement in knowledge was measured by a multiple choice question (MCQ) test. The MCQ was completed prior to beginning the whole course, after completion of the e‐learning module and again at follow up 8 months later. Students also completed an evaluation survey. Results: Twenty‐one students participated. There was a significant objective increase in knowledge from pre‐course to post e‐learning scores, median scores (interquartile range) from 12/23 (10.5 to 13.5) to 21/23 (20 to 22.5), P < 0.001. This significant increase in knowledge was still apparent at follow up 8 months later. Median MCQ score at follow up was 17/23 (14 to 18.5), P < 0.002. Students self‐rated significant improvements in their knowledge, confidence and ability to perform basic and advanced life support for the whole course and between individual components (P < 0.001). Conclusions: A novel paediatric resuscitation course for medical students was developed and evaluated. This demonstrated significant objective improvements in student knowledge throughout the course, at course completion and at 8 month follow up. There were also significant subjective improvements in knowledge, confidence and ability to perform paediatric resuscitation.     

Using Automatic Item Generation to Improve the Quality of MCQ Distractors

Construct: Automatic item generation (AIG) is an alternative method for producing large numbers of test items that integrate cognitive modeling with computer technology to systematically generate multiple-choice questions (MCQs). The purpose of our study is to describe and validate a method of generating plausible but incorrect distractors. Initial applications of AIG demonstrated its effectiveness in producing test items. However, expert review of the initial items identified a key limitation where the generation of implausible incorrect options, or distractors, might limit the applicability of items in real testing situations. Background: Medical educators require development of test items in large quantities to facilitate the continual assessment of student knowledge. Traditional item development processes are time-consuming and resource intensive. Studies have validated the quality of generated items through content expert review. However, no study has yet documented how generated items perform in a test administration. Moreover, no study has yet to validate AIG through student responses to generated test items. Approach: To validate our refined AIG method in generating plausible distractors, we collected psychometric evidence from a field test of the generated test items. A three-step process was used to generate test items in the area of jaundice. At least 455 Canadian and international medical graduates responded to each of the 13 generated items embedded in a high-stake exam administration. Item difficulty, discrimination, and index of discrimination estimates were calculated for the correct option as well as each distractor. Results: Item analysis results for the correct options suggest that the generated items measured candidate performances across a range of ability levels while providing a consistent level of discrimination for each item. Results for the distractors reveal that the generated items differentiated the low- from the high-performing candidates. Conclusions: Previous research on AIG highlighted how this item development method can be used to produce high-quality stems and correct options for MCQ exams. The purpose of the current study was to describe, illustrate, and evaluate a method for modeling plausible but incorrect options. Evidence provided in this study demonstrates that AIG can produce psychometrically sound test items. More important, by adapting the distractors to match the unique features presented in the stem and correct option, the generation of MCQs using automated procedure has the potential to produce plausible distractors and yield large numbers of high-quality items for medical education.     

Association between EMS Question Bank Completion and Passing Rates on the EMS Certification Examination

Introduction: A board review question bank was created to assist candidates in their preparation for the 2015 EMS certification examination. We aimed to describe the development of this question bank and evaluate its successes in preparing candidates to obtain EMS subspecialty board certification. Methods: An online question bank was developed by 13 subject matter experts who participated as item writers, representing eight different EMS fellowship programs. The online question bank consisted of four practice tests, with each of the tests comprised of 100 questions. The number of candidates who participated in and completed the question bank was calculated. The passing rate among candidates who completed the question bank was calculated and compared to the publicly reported statistics for all candidates. The relationship between candidates' performance on the question bank and subspecialty exam pass rates was determined. Results: A total of 252 candidates took at least one practice test and, of those, 225 candidates completed all four 100-question practice tests. The pass rate on the 2015 EMS certification exam was 79% (95%CI 74–85%) among candidates who completed the question bank, which is 12% higher than the overall pass rate (p = 0.003). Candidates' performance on the question bank was positively associated with overall success on the exam (X² = 75.8, p < 0.0001). Achieving a score of ≥ 70% on the question bank was associated with a higher likelihood of passing the exam (OR = 17.8; 95% CI: 8.0–39.6). Conclusion: Completing the question bank program was associated with improved pass rates on the EMS certification exam. Strong performance on the question bank correlated with success on the exam.     

Computer Versus Paper—Does It Make Any Difference in Test Performance?

Construct: In this study, we examine the differences in test performance between the paper-based and the computer-based version of the Berlin formative Progress Test. In this context it is the first study that allows controlling for students’ prior performance. Background: Computer-based tests make possible a more efficient examination procedure for test administration and review. Although university staff will benefit largely from computer-based tests, the question arises if computer-based tests influence students’ test performance. Approach: A total of 266 German students from the 9th and 10th semester of medicine (comparable with the 4th-year North American medical school schedule) participated in the study (paper = 132, computer = 134). The allocation of the test format was conducted as a randomized matched-pair design in which students were first sorted according to their prior test results. The organizational procedure, the examination conditions, the room, and seating arrangements, as well as the order of questions and answers, were identical in both groups. Results: The sociodemographic variables and pretest scores of both groups were comparable. The test results from the paper and computer versions did not differ. The groups remained within the allotted time, but students using the computer version (particularly the high performers) needed significantly less time to complete the test. In addition, we found significant differences in guessing behavior. Low performers using the computer version guess significantly more than low-performing students in the paper-pencil version. Conclusions: Participants in computer-based tests are not at a disadvantage in terms of their test results. The computer-based test required less processing time. The reason for the longer processing time when using the paper-pencil version might be due to the time needed to write the answer down, controlling for transferring the answer correctly. It is still not known why students using the computer version (particularly low-performing students) guess at a higher rate. Further studies are necessary to understand this finding.     

High-Stakes Collaborative Testing: Why Not?

Phenomenon: Studies of high-stakes collaborative testing remain sparse, especially in medical education. We explored high-stakes collaborative testing in medical education, looking specifically at the experiences of students in established and newly formed teams. Approach: Third-year psychiatry students at 5 medical schools across 6 sites participated, with 4 participating as established team sites and 2 as comparison team sites. For the collaborative test, we used the National Board of Medical Examiners Psychiatry subject test, administering it via a 2-stage process. Students at all sites were randomly selected to participate in a focus group, with 8–10 students per site (N = 49). We also examined quantitative data for additional triangulation. Findings: Students described a range of heightened emotions around the collaborative test yet perceived it as valuable regardless if they were in established or newly formed teams. Students described learning about the subject matter, themselves, others, and interpersonal dynamics during collaborative testing. Triangulation of these results via quantitative data supported these themes. Insights: Despite student concerns, high-stakes collaborative tests may be both valuable and feasible. The data suggest that high-stakes tests (tests of learning or summative evaluation) could also become tests for learning or formative evaluation. The paucity of research into this methodology in medical education suggests more research is needed.     

Development and reliability of the Motivation for Change Questionnaire

Purpose.?The purpose of this study was to describe the development of the Motivation for Change Questionnaire (MCQ) and to test its intra-patient reliability on musculoskeletal pain patients in interdisciplinary rehabilitation as a basis for use in rehabilitation planning.

Method.?The MCQ questionnaire was developed from a literature search in the Medline, Cinahl and Psychlit databases concerning motivating factors for change in the life and work situation. Questions covering these factors were developed (item generation). Factor analysis of the questions implied a reduction of the number of questions (item reduction). Inter-item correlation was assessed on the baseline administration of the questionnaire by calculating Cronbach's α. When testing the structure of the scales, it was shown that the MCQ questionnaire could be described in two scales, one scale relating to motivation for change in the life situation and the other focusing on motivation for change in the work situation and in total 49 questions.

Results.?The test?–?retest reliability was calculated using the intra-class correlation coefficient (ICC) and 95% confidence intervals were calculated. One question was excluded due to the threshold limit of >?0.5. Seven scales relating to the life situation were accepted by the analysis: social support, mastery in life, challenges in life, control in life, values, self-efficacy and self-confidence. Six scales relating to the work situation were also accepted: co-worker support, supervisory support, challenges in work, job control, interaction and job-satisfaction.

Conclusions.?The MCQ questionnaire with 48 questions is reliable for use on musculoskeletal pain patients in interdisciplinary rehabilitation. It can be used to identify each individual's motivating factors for change in life and work situation as a basis for motivational work within rehabilitation and/or to measure within-subject changes in motivation over time. The validity and the responsiveness of the MCQ, need to be studied.     

The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments

Multiple-choice questions are a common assessment method in nursing examinations. Few nurse educators, however, have formal preparation in constructing multiple-choice questions. Consequently, questions used in baccalaureate nursing assessments often contain item-writing flaws, or violations to accepted item-writing guidelines. In one nursing department, 2770 MCQs were collected from tests and examinations administered over a five-year period from 2001 to 2005. Questions were evaluated for 19 frequently occurring item-writing flaws, for cognitive level, for question source, and for the distribution of correct answers. Results show that almost half (46.2%) of the questions contained violations of item-writing guidelines and over 90% were written at low cognitive levels. Only a small proportion of questions were teacher generated (14.1%), while 36.2% were taken from testbanks and almost half (49.4%) had no source identified. MCQs written at a lower cognitive level were significantly more likely to contain item-writing flaws. While there was no relationship between the source of the question and item-writing flaws, teachergenerated questions were more likely to be written at higher cognitive levels (p < 0.001). Correct answers were evenly distributed across all four options and no bias was noted in the placement of correct options. Further training in item-writing is recommended for all faculty members who are responsible for developing tests. Pre-test review and quality assessment is also recommended to reduce the occurrence of item-writing flaws and to improve the quality of test questions.     

Development and reliability of the Motivation for Change Questionnaire

PURPOSE: The purpose of this study was to describe the development of the Motivation for Change Questionnaire (MCQ) and to test its intra-patient reliability on musculoskeletal pain patients in interdisciplinary rehabilitation as a basis for use in rehabilitation planning. METHOD: The MCQ questionnaire was developed from a literature search in the Medline, Cinahl and Psychlit databases concerning motivating factors for change in the life and work situation. Questions covering these factors were developed (item generation). Factor analysis of the questions implied a reduction of the number of questions (item reduction). Inter-item correlation was assessed on the baseline administration of the questionnaire by calculating Cronbach's alpha. When testing the structure of the scales, it was shown that the MCQ questionnaire could be described in two scales, one scale relating to motivation for change in the life situation and the other focusing on motivation for change in the work situation and in total 49 questions. RESULTS: The test -- retest reliability was calculated using the intra-class correlation coefficient (ICC) and 95% confidence intervals were calculated. One question was excluded due to the threshold limit of > 0.5. Seven scales relating to the life situation were accepted by the analysis: social support, mastery in life, challenges in life, control in life, values, self-efficacy and self-confidence. Six scales relating to the work situation were also accepted: co-worker support, supervisory support, challenges in work, job control, interaction and job-satisfaction. CONCLUSIONS: The MCQ questionnaire with 48 questions is reliable for use on musculoskeletal pain patients in interdisciplinary rehabilitation. It can be used to identify each individual's motivating factors for change in life and work situation as a basis for motivational work within rehabilitation and/or to measure within-subject changes in motivation over time. The validity and the responsiveness of the MCQ, need to be studied.     

Impact of an In-Person Versus Web-Based Practice Standardized Patient Examination on Student Performance on a Subsequent High-Stakes Standardized Patient Examination

Background: Optimal methods of preparing students for high-stakes standardized patient (SP) examinations are unknown. Purposes: The purpose is to compare the impact of two formats of a formative SP examination (Web-based vs. in-person) on scores on a subsequent high-stakes SP examination and to compare students' satisfaction with each formative examination format. Methods: Clustered randomized trial comparing a Web-based module versus in-person formative SP examination. We compared scores on a subsequent high-stakes SP examination and satisfaction. Results: Scores on the subsequent high-stakes SP examination did not differ between the two formative formats but were higher after the formative assessment than without (p < .001). Satisfaction was higher with the in-person than Web-based formative assessment format (4.00 vs. 3.62 on a 5-point scale, p = .01). Conclusions: Two formats of a formative SP examination led to equivalent improvement in scores on a subsequent high-stakes examination. Students preferred an in-person formative examination to online but were satisfied with both.     

Development,Evaluation, and Delivery of an Innovative National Undergraduate Surgical Workshop: Recognition and Management of the Acutely Unwell Surgical Patient

Problem: Recognition and management of acutely unwell surgical patients is an important skill to which medical students have little exposure. Intervention: We present the evaluation of a novel national surgical workshop that consisted of high-fidelity simulations, lectures, case demonstrations, case discussions, and a basic surgical skills tutorial. The high-fidelity simulations re-created genuine patient encounters and were used to facilitate the acquisition of knowledge and skill in the early recognition and management of acutely unwell surgical patients. Context: The optional workshop was designed for senior medical students and delivered by surgical trainees. Students were asked to complete a 12-item evaluation questionnaire and a 26-item multiple-choice question (MCQ) quiz, which assessed their confidence; self-perceived competence; and knowledge prior to, immediately following, and 8 weeks after the workshop. Pre- and postdata were compared using student's two-tailed t test. Outcome: A total of 66 medical students from 6 UK universities attended, the majority of whom enjoyed the workshop (98.3%, n = 59). Participants' confidence rating (scale = 1–5) in assessing an unwell surgical patient improved from a mean of 2.5 (n = 47) to 4.4 (n = 60). Confidence in commencing initial management improved from a mean of 2.7 (n = 47) to 4.1 (n = 59). Confidence and self-perceived competence across 12 domains improved significantly following the workshop, two-tailed unpaired t test, t(22) = 8.64, p <.0001, d = 3.68. MCQ scores immediately following the workshop were a statistically significant improvement on the preworkshop MCQ scores (n = 44), paired two-tailed t test, t(43) = 7.76, p <.0001, d = 2.37, and the improvement was sustained 8 weeks following the workshop (n = 18), paired two-tailed t test, t(17) = 3.34, p =.0039, d = 1.62. Lessons Learned: Feedback from students was very positive and clearly demonstrated that a workshop taught by surgical trainees improved medical students’ confidence, self-perceived competence, and knowledge in the assessment and management of acutely unwell surgical patients.