First experience with multiple mini interview for medical school admission in Brazil: Does it work in a different cultural scenario?

Purpose: Evaluation of non-cognitive skills never has been used in Brazil. This study aims to evaluate Multiple Mini Interviews (MMI) in the admission process of a School of Medicine in São Paulo, Brazil.

Methods: The population of the study comprised 240 applicants summoned for the interviews, and 96 raters. MMI contributed to 25% of the applicants’ final grade. Eight scenarios were created with the aim of evaluating different non-cognitive skills, each one had two raters. At the end of the interviews, the applicants and raters described their impressions about MMI. The reliability of the MMI was analyzed using the Theory of Generalization and Many-Facet Rasch Model (MFRM).

Results: The G-study showed that the general reliability of the process was satisfactory (coefficient G?=?0.743). The MMI grades were not affected by the raters’ profile, time of interview (p?=?0.715), and randomization group (p?=?0.353). The Rasch analysis showed that there was no misfitting effects or inconsistent stations or raters. A significant majority of the applicants (98%) and all the raters believed MMIs were important in selecting students with a more adequate profile to study medicine.

Conclusions: The general reliability of the selection process was excellent, and it was fully accepted by the applicants and raters.     

Narrative information obtained during student selection predicts problematic study behavior

Introduction: Up to now, student selection for medical schools is merely used to decide which applicants will be admitted. We investigated whether narrative information obtained during multiple mini-interviews (MMIs) can also be used to predict problematic study behavior.

Methods: A retrospective exploratory study was performed on students who were selected into a four-year research master’s program Physician-Clinical Investigator in 2007 and 2008 (n?=?60). First, counselors were asked for the most prevalent non-cognitive problems among their students. Second, MMI notes were analyzed to identify potential indicators for these problems. Third, a case-control study was performed to investigate the association between students exhibiting the non-cognitive problems and the presence of indicators for these problems in their MMI notes.

Results: The most prevalent non-cognitive problems concerned planning and self-reflection. Potential indicators for these problems were identified in randomly chosen MMI notes. The case-control analysis demonstrated a significant association between indicators in the notes and actual planning problems (odds ratio: 9.33, p?=?0.003). No such evidence was found for self-reflection-related problems (odds ratio: 1.39, p?=?0.68).

Conclusions: Narrative information obtained during MMIs contains predictive indicators for planning-related problems during study. This information would be useful for early identification of students-at-risk, which would enable focused counseling and interventions to improve their academic achievement.     

Construct validity of multiple mini interviews – Investigating the role of stations,skills, and raters using Bayesian G-theory

Abstract

Background: One popular procedure in the medical student selection process are multiple mini-interviews (MMIs), which are designed to assess social skills (e.g., empathy) by means of brief interview and role-play stations. However, it remains unclear whether MMIs reliably measure desired social skills or rather general performance differences that do not depend on specific social skills. Here, we provide a detailed investigation into the construct validity of MMIs, including the identification and quantification of performance facets (social skill-specific performance, station-specific performance, general performance) and their relations with other selection measures.

Methods: We used data from three MMI samples (N?=?376 applicants, 144 raters) that included six interview and role-play stations and multiple assessed social skills.

Results: Bayesian generalizability analyses show that, the largest amount of reliable MMI variance was accounted for by station-specific and general performance differences between applicants. Furthermore, there were low or no correlations with other selection measures.

Discussion: Our findings suggest that MMI ratings are less social skill-specific than originally conceptualized and are due more to general performance differences (across and within-stations). Future research should focus on the development of skill-specific MMI stations and on behavioral analyses on the extents to which performance differences are based on desirable skills versus undesired aspects.     

COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape

Abstract

Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented “business as usual” admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.

Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.

Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.

Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.     

The multiple mini-interviews as a predictor of peer evaluations during clinical training in medical school*

Background: Multiple Mini-Interviews (MMI) is an admissions tool being used for more than a decade by medical schools worldwide to assess non-cognitive skills. In Israel, in addition to the MMI tool, two questionnaires are given to the candidates. It has been necessary to find a relevant criterion to examine the predictive validity of the MMI test. The measure developed in this study was a peer assessment tool which was filled out during clerkship.

Aims: The study aims at evaluating the predictive validity of the MMI and two questionnaires when compared with the peer assessment tool.

Method: Ninety nine fourth-year medical students were included in this study. All of them went through the MMI test before the beginning of their studies. The students participated in two clinical rotations and were divided into 10 groups, each consisting of 9–12 students. This study is based on assessments given to 99 students. Every member of a group evaluated his fellow members on a number of attribute. The questionnaire included 10 statements, related to a particular attributes. In addition two concluding questions were included. Pearson correlations were used to test the relationships between pre-admission variables (MMI and two questionnaires) and the peer assessments.

Results: Statistically significant, moderate correlations between the general MOR and MIRKAM scores and the sum of the questionnaire assessments were 0.39 and 0.37, respectively (P-value <0.05).

Conclusions: The study provides important evidence for the validity of the MMI. Additional studies are required to reevaluate the predicted validity of the MMI.     

The OSCE progress test – Measuring clinical skill development over residency training

Purpose: The purpose of this study was to explore the use of an objective structured clinical examination for Internal Medicine residents (IM-OSCE) as a progress test for clinical skills.

Methods: Data from eight administrations of an IM-OSCE were analyzed retrospectively. Data were scaled to a mean of 500 and standard deviation (SD) of 100. A time-based comparison, treating post-graduate year (PGY) as a repeated-measures factor, was used to determine how residents’ performance progressed over time.

Results: Residents’ total IM-OSCE scores (n?=?244) increased over training from a mean of 445 (SD?=?84) in PGY-1 to 534 (SD?=?71) in PGY-3 (p?<?0.001). In an analysis of sub-scores, including only those who participated in the IM OSCE for all three years of training (n?=?46), mean structured oral scores increased from 464 (SD?=?92) to 533 (SD?=?83) (p?<?0.001), physical examination scores increased from 464 (SD?=?82) to 520 (SD?=?75) (p?<?0.001), and procedural skills increased from 495 (SD?=?99) to 555 (SD?=?67) (p?=?0.033). There was no significant change in communication scores (p?=?0.97).

Conclusions: The IM-OSCE can be used to demonstrate progression of clinical skills throughout residency training. Although most of the clinical skills assessed improved as residents progressed through their training, communication skills did not appear to change.     

An analysis of reflective writing early in the medical curriculum: The relationship between reflective capacity and academic achievement

Objective: To examine the relationship between reflection, gender, residency choice, word count, and academic achievement among medical students.

Methods: A modified version of the Reflection Evaluation for Learners’ Enhanced Competencies Tool (REFLECT) was developed and used for this study (Cronbach’s alpha of 0.86 with an intraclass correlation coefficient [ICC] of 0.68). This was applied to writing samples about professionalism in gross anatomy from first-year medical students between 2005 and 2011. Four analysts reviewed and scored written reflections independently. Composite reflection scores were compared with gender, residency choice, length of written reflection, NBME® Gross Anatomy and Embryology Subject Examination scores, and final gross anatomy course.

Results: Total of 319 written reflections were evaluated. Female students who pursued medicine specialties had the highest composite reflection scores (87 [27.2%]). Word count frequently correlated with reflection score (p?<?0.0001). Students who performed well on the NBME® Gross Anatomy and Embryology Subject Examination tended to achieve high anatomy course grades (p?<?0.0001). There was no statistically significant relationship between composite reflection scores and NBME® Gross Anatomy and Embryology Subject Examination scores (p?=?0.16) or anatomy course grades (p?=?0.90).

Conclusions: This study suggests there are likely no correlations between reflective capacity and academic performance on tests of medical knowledge administered early in the medical curriculum.     

Cost-consciousness among Iranian internal medicine residents

Abstract

Background: Study aimed at assessing residents' cost awareness and their attitude about health care costs.

Methods: Internal medicine residents at teaching hospitals of Tehran University of Medical Sciences were surveyed during August–December 2016 using a researcher-made questionnaire comprising attitude statements and cost estimation of diagnostic and treatment items.

Results: Eighty-nine residents completed the survey (response rate?=?56.6%). The results indicate that less than one quarter (23.69%) of cost estimates were in the range of correct answers. The mean (SD) for correct estimation of medications (out of 8 scores), lab tests (out of 20 scores), and total (out of 35 scores) were 1.25 (0.96), 4.92 (0.27), and 7.97 (0.34), respectively. An analysis of variance showed that the level of residency was positively correlated with residents’ correct cost estimation (F (3, 77)=9.98, p?=?0.029). There was a significant positive correlation between age of residents with the correct estimate of medication prices (p?=?0.018, r?=?0.261).

Conclusions: The internal medicine residents of Tehran University of Medical Sciences have poor knowledge of health care costs, including medications, diagnostic tests, and hospitalization costs. The results of this study explain the necessity of developing a training program for the transfer of cost information to physicians.     

Changes in medical student implicit attitudes following a health equity curricular intervention

Purpose: This study assessed the: (1) effect of an LGBTQI?+?health equity curriculum (eQuality) on implicit attitudes among first (M1) and second year (M2) medical students and (2) utility of dedicated time to explore implicit bias.

Method: Implicit biases were assessed at baseline using implicit association tests (IAT) for all M2s and a random sample of first years (M1A). These students were then debriefed on strategies to mitigate bias. Following eQuality, all M1 and M2s completed post-intervention IATs. The remaining first years (M1B) were then debriefed. Paired sample t-tests assessed differences between pre/post. Independent sample t-tests assessed differences in post-IATs between M1 groups.

Results: IATs indicated preferences for “Straight,” “White,” and “Thin” at both pre and post. M2s demonstrated statistically significant improvements pre to post for sexuality (p?=?0.01) and race (p?=?0.03). There were significant differences in post-intervention IAT scores between M1As who received the IAT and debriefing prior to eQuality and M1Bs for sexuality (p?=?0.002) and race (p?=?0.046). There were no significant changes for weight.

Conclusion: eQuality reduced implicit preference for “Straight” and “White.” Differences in M1 post-intervention IAT scores between groups suggest dedicating time to debrief implicit attitudes enhances bias mitigation.     

A new admission method may select applicants with a distinct personality profile

Background: There is little data to suggest that a specific admission method can select students with a distinct personality profile. We have recently introduced a new admission process that combines a computerized personality test, with a single interview. The purpose of the current study was to find whether the new method selects applicants with a different personality profile and attitudes compared with the previous method.

Method: Using a validated personality questionnaire (HEXACO) and attitudes questionnair, that were filled anonymously between November 2014 and May 2015, the authors compared two groups of students: group A comprising students accepted with the new method (first and second year) with group B comprising students accepted with the previous method (third to sixth year).

Results: In group A, 157 responded out of 250 (63%), while in group B 194 out of 352 (55%). Group A students ranked significantly higher in honesty-humility, extraversion, agreeableness and openness to experience, and lower in emotionality. Physicians’ role in society was perceived to be more meaningful among Group A students (M?=?4.19, SD?=?0.50, N?=?152) compared to Group B students (M?=?3.86, SD?=?0.57, N?=?184).

Conclusions: The new method may select applicants with a distinct personality profile and different attitudes toward the physicians' role in the society.     

Mixed reality as a time-efficient alternative to cadaveric dissection

Abstract

Objectives: The extent of medical knowledge increases yearly, but the time available for students to learn is limited, leading to administrative pressures to revise and reconfigure medical school curricula. The goal of the present study is to determine whether the mixed reality platform HoloAnatomy represents an effective and time-efficient modality to learn anatomy when compared to traditional cadaveric dissection.

Methods: This was a prospective, longitudinal study of medical students completing a musculoskeletal anatomy course at Case Western Reserve University School of Medicine. Participants were divided into two groups based on learning platform (HoloAnatomy versus traditional cadaveric dissection) and content area (upper limb versus lower limb anatomy). Time spent in lab and end of course practical exam scores were compared between groups.

Results: The average study time of 48 medical students who completed study requirements was 4.564?h using HoloAnatomy and 7.318?h in the cadaver lab (p?=?0.001). No significant difference was found between exam scores for HoloAnatomy and cadaver learners (p?=?0.185).

Conclusions: Our results indicate that HoloAnatomy may decrease the time necessary for anatomy didactics without sacrificing student understanding of the material.     

Comparison of faculty assessment and students’ self-assessment of performance during clinical case discussions in a pharmacotherapy capstone course

Objectives: The primary objective of this study was to compare faculty assessment and third year students' self-assessment of performance in clinical case discussions. The secondary objective was to evaluate if student characteristics influence self-assessments.

Methods: This retrospective analysis compared faculty and student self-assessment scores for two clinical case discussions using Spearman’s correlation and Wilcoxon’s signed ranks test. Chi-squared test was used to compare frequency of faculty and student self-assessments indicating the highest possible rating for the pooled score and for each individual component. The pooled score included three individual components: level of engagement, quality of contribution, and professionalism.

Results: Pooled faculty and student self-assessments correlated for both the first (r?=?0.41, p?r?=?0.35; p?p?=?0.25) and second (58.6% vs. 47.4%, p?=?0.05) clinical case discussions. Student characteristics (age, gender, and grade point average at graduation) did not influence self-assessments.

Conclusions: Students’ self-assessment correlated with faculty assessment of performance during clinical case discussions. Increased use of self-assessments for professional development in pharmacy and other healthcare professional curricula should be considered.     

Undergraduate medical students’ perspectives of skills,uses and preferences of information technology in medical education: A cross-sectional study in a Saudi Medical College

Abstract

Background: Information technology (IT) is widely used in medical education. However, there are not enough studies about IT uses and preferences among traditional and problem-based learning (PBL) medical students.

Aim: To compare IT skills, uses and preferences for education between traditional and PBL medical students’.

Method: A cross-sectional study; a modified Educause Center for Analysis and Research online survey was sent to traditional curriculum 5th and PBL 4th year medical students of King Saud University.

Results: Most of the responding 176 students prefer mobile devices and moderate amount of IT in education. Fourth and fifth year students perceived high academic value of Google (94.2 vs. 86.7%, p?=?0.34), YouTube (90.7 vs. 92.2%, p?=?0.83) and PubMed (83.7 vs. 86.7%, p?=?0.06). More 4th year than 5th year students rated themselves as skilled in learning management system (54.7 vs. 21.1%, p?=?0.0001) and Smartboard use (40.7 vs. 23.3%, p?=?0.04). Most students rated faculty IT skills as effective. Students agreed that technology helps working faster (95.5%) and make learning creative (85.9%).

Conclusions: More integration of information literacy and IT training in medical curricula is needed to enhance better utilization of full features of IT resources available for learning and problem solving. National multi-institutional studies are recommended.     

Risk factors for a delay in medical education: Results of an online survey among four German medical schools

Background: Delayed study progress in medical school is a challenging issue for the tax paying community, the faculty and the medical students themselves. Reasons for a delay might be different from known risk factors for academic difficulties.

Methods: An online survey regarding delays in the study progress and including a personality test (BFI-10) was presented to medical students from four German medical schools after completion of their 3rd year of study.

Results: Of 617 students, 51.2% reported a mean delay of 2.1?±?1.5 semesters. Frequent risk factors were secondary employment (69.5%, odds ratio (OR) 1.7, p?=?0.004), female gender (69.8%, OR 1.6, p?=?0.007), work or study abroad (35.9%, OR 1.5, p?=?0.02), a late graduation (5.9%, OR 2.4, p?=?0.02), as well as support through scholarship or mentoring (19.9%, OR 1.8, p?=?0.004). “Working on doctoral thesis” (11.3%, OR 1.9, p?=?0.03) and structural curricular issues (36.6%, OR 0.9, p?=?0.7) were frequently identified as obstacles. “Support by friends/family” was considered helpful by 24.1% (OR 1.4, p?=?0.09), as well as a high intrinsic motivation (19.1%, OR 0.5, p?=?0.01). In the BFI-10, students with study delay were more prone to openness and agreeableness.

Conclusions: Risk factors for delay are not identical to those for academic difficulties. To decrease the risk for delays, firm curricular structures should be identified and alleviated. Intrinsic motivation is a strong impetus of study progress and additionally might be strengthened by curricular changes.     

Using a self-regulated learning-enhanced video feedback educational intervention to improve junior doctor prescribing

Abstract

Introduction: Medical school graduates in the UK consistently report feeling underprepared for the task of prescribing when embarking on practice. The effective application of self-regulated learning (SRL) approaches and feedback on complex tasks are associated with improved outcomes in practice-based clinical skills.

Aims: This study aimed to investigate the effectiveness of an educational intervention using SRL-enhanced video feedback for improving the prescribing competency of junior doctors.

Methods: A prospective cohort study was designed to compare intervention and control cohorts of junior doctors undertaking simulated clinical encounters at the beginning and end of their 4-month rotation through renal medicine.

Results: The improvement in prescribing competency for the intervention cohort was significant (p?<?0.001) with large effect size (d?=?1.42). Self-efficacy improved in both cohorts with large (control cohort p = 0.026, r= 0.64) and medium (intervention cohort p = 0.083, d = 0.55) effect sizes. Goal setting and self-monitoring skills improved in the intervention cohort only with medium effect size (p?=?0.096, d?=?0.53).

Conclusions: SRL-enhanced video feedback is effective for improving prescribing competency and developing SRL processes such as goal setting and self-monitoring skills in simulated clinical encounters. Further research is required to evaluate transferability to other clinical sub-speciality contexts and investigate the effectiveness of the intervention for improving prescribing in non-simulated settings.     

A unique flipped classroom approach shows promising results in physician assistant education

Abstract

Aims: The flipped classroom (FC) model utilizes pre-class foundational learning with in-class higher-order application exercises. FC approaches have reported positive perceptions and mixed academic performance outcomes in medical education; however, little evidence exists on their impact in the 56-week didactic curriculum of Physician Assistant (PA) education.

Methods: A 4-week FC curriculum was piloted in a PA behavioral medicine course (n?=?34). Before class, students completed online video modules. During class, students completed a pre-class quiz and answered clinical case-based questions in a think-pair-share format.

Results: Students rated in-class time higher than a lecture-based segment of the same course (p?=?0.028). Assessment of achievement emotions demonstrated increased enjoyment (p?=?0.028) and decreased boredom (p?<?0.001) in the FC curriculum with no difference in anxiety. The FC curriculum produced a 57.2?min increase in pre-class preparation time with no change in post-class studying time. Compared to historical lecture-based controls (n?=?35), students in the FC curriculum scored significantly higher on clinical vignette questions (p?=?0.019) with equivalent performance on other question formats.

Conclusions: This study offers the first positive outcomes of an FC approach in PA education and provides a framework for improving academic performance while scaling back in-class time.     

The impact of informed self-assessment on the development of medical students’ learning goals

Purpose: This study investigates the contributions of self-assessment (SA) and external feedback on the development of learning goals (LG) and the influence on LG recall and implementation in medical students.

Methods: Following a standardized patient (SP) assessment, 168 pre-clinical medical students completed a SA, received SP feedback and created a LG. LG were categorized by source. Two weeks later, students recalled LG and described implementation. Chi-squared analyses were used to test the associations.

Results: SA influenced LG for 82.8% of students whereas SP feedback influenced LG for 45.9%. Students rarely generated LG based on SA when they received discordant feedback (5.4%), but sometimes incorporated feedback discordant from their SA into LG (14.9%). Students who created LG based on SP feedback were more likely to recall LG than those who created LG based on SA, 89.7 versus 67.6%, p?2(1)?=?5.3, p?=?0.017. Students who reported receiving effective feedback were more likely to implement their LG than those reporting adequate feedback, 60.9 versus 37.9%, χ²(1)?=?8.0, p?=?0.01.

Conclusions: SA is an essential part of goal setting and subsequent action. Perception of feedback plays a crucial role in LG implementation.     

The role of direct single monetary grants in residents’ choice of field of residency

Background: In an attempt to address severe medical manpower shortages in several medical disciplines, the Israeli Ministry of Health offered grants to residents who chose one of these fields.

Methods: A total of 220 residents from various disciplines were surveyed on demographic, academic, and professional data, and asked to rank considerations in the choice of their field of residency.

Results: Residents in targeted fields attributed significantly more importance to the grant in their decision-making process (U?=?3704.5, p?p?p?=?0.031) and attribution of significance to the working conditions compared to other residency fields (OR 1.69, 95%CI 1.23–2.32, p?=?0.001) were significantly associated with receptivity toward the grant in a multivariate analysis.

Discussion: Receptivity toward the offered grants correlated with real-life data shows a rise in physician in these fields, and the weak association between such receptivity and most variables tested may suggest that the grants were perceived as a property of the specific choice rather than a special bonus.

Conclusions: Grants may be useful in diverting medical manpower. Further analysis and modeling are required to determine causal relationship and budgetary feasibility.     

Speed and quality goals in procedural skills learning: A randomized experiment

Abstract

Purpose: Compare time (speed) and product quality goals in a surgical procedural task.

Methods: Secondary school students participating in a medical simulation-based training activity participated in a randomized experiment. Each participant completed eight repetitions of a blood vessel ligation. Once, between repetitions four and five, each participant received a randomly-assigned speed goal or quality goal. Outcomes included time and leak-free ligatures.

Results: 80 students participated. The speed-goal group performed 18% faster on the final repetition than the quality-goal group, with adjusted fold change (FC) 0.82 (95% confidence interval [CI], 0.71, 0.94; p?=?0.01). Conversely, the speed-goal group had fewer high-quality (leak-free) ligatures (odds ratio [OR] 0.36 [95% CI, 0.22, 0.58; p?<?0.001]). For the quality-goal group, leaky ligatures took longer post-intervention than leak-free ligatures (FC 1.09 [95% CI, 1.02, 1.17; p?=?0.01]), whereas average times for leaky and leak-free ligatures were similar for the speed-goal group (FC 0.97 [95% CI, 0.91, 1.04; p?=?0.38]). For a given performance time, the speed-goal group had more leaks post-intervention than the quality-goal group (OR 3.35 [95% CI, 1.58, 7.10; p?=?0.002]).

Conclusions: Speed and quality goals promote different learning processes and outcomes among novices. Use of both speed and quality goals may facilitate more effective and efficient learning.