The multiple mini-interviews as a predictor of peer evaluations during clinical training in medical school*

Background: Multiple Mini-Interviews (MMI) is an admissions tool being used for more than a decade by medical schools worldwide to assess non-cognitive skills. In Israel, in addition to the MMI tool, two questionnaires are given to the candidates. It has been necessary to find a relevant criterion to examine the predictive validity of the MMI test. The measure developed in this study was a peer assessment tool which was filled out during clerkship.

Aims: The study aims at evaluating the predictive validity of the MMI and two questionnaires when compared with the peer assessment tool.

Method: Ninety nine fourth-year medical students were included in this study. All of them went through the MMI test before the beginning of their studies. The students participated in two clinical rotations and were divided into 10 groups, each consisting of 9–12 students. This study is based on assessments given to 99 students. Every member of a group evaluated his fellow members on a number of attribute. The questionnaire included 10 statements, related to a particular attributes. In addition two concluding questions were included. Pearson correlations were used to test the relationships between pre-admission variables (MMI and two questionnaires) and the peer assessments.

Results: Statistically significant, moderate correlations between the general MOR and MIRKAM scores and the sum of the questionnaire assessments were 0.39 and 0.37, respectively (P-value <0.05).

Conclusions: The study provides important evidence for the validity of the MMI. Additional studies are required to reevaluate the predicted validity of the MMI.     

Open book tests: assessment of academic learning in clerkships

An examination of an open-book testing approach in a family medicine clerkship seeks to determine whether this method more closely mirrors the discipline of family medicine, where practitioners refer daily to written resource materials in order to make clinical decisions without compromising the learning and assessment process. Student scores on the multiple-choice test were analysed by year, by quarter and by site using ANOVA. Students in the experimental site were interviewed to determine preparation style, use of text during test, as well as attitudes toward open-book testing. Analysis of variance showed that the interaction of site and year was significant at p = 0.03. The mean score of 88.2 for Maine students in 2002 was significantly different from the other three mean scores. The desired qualitative outcomes of the intervention were confirmed: reducing the anxiety of students, wider reading of the textbook, knowing the structure of the textbook as a learning resource, and deeper understanding of concepts and principles rather than time spent on memorization. While the difference in scores did reach statistical significance, it is important to note that the difference in mean score was only four points on a 100-point scale. Given the percentage of the total grade represented by the test score, it is unlikely that this difference represents any real difference in grade for students in Maine compared with Vermont. The students appeared to approach the textbook and therefore, perhaps, the body of knowledge as a whole with the orientation of a generalist. The MMC Clerkship Director recommended the implementation of the open-book approach to the Family Practice clerkship at all sites and the University of Vermont Medical School accepted the proposal. This recommendation supports advising students on the preparation for an open-book test and on tactics for the best use of the textbook during the test.     

A comparison of clinical teaching evaluations by resident and peer physicians

The purpose of this study was to compare the reliability of inpatient teaching evaluations by resident and peer physicians on Mayo internal medicine hospital services. Three resident and three peer evaluators observed 10 consecutively chosen attending physicians on the Mayo hospital services. Evaluations by resident and peer physicians were compared in terms of mean scores. Kendall's coefficient of concordance (KCC) was used to summarize inter-rater reliabilities and Cronbach's coefficient alpha was used to determine internal consistencies of evaluations by residents and peers. Results of this study revealed that mean scores of the 13 evaluation items were generally higher for resident than peer physicians. None of the items completed by residents had KCC scores >0.5, whereas 10 of the items completed by peers had KCC scores >0.5. Likewise, none of the residents' items had KCC p-values <0.05, whereas nine of the peers' items had KCC p-values <0.05. The overall internal consistency was higher for peers (alpha = 0.76) than for residents (alpha = 0.71). In conclusion, resident physicians uniformly rate faculty highly. Furthermore, peer evaluations yield higher inter-rater and internal reliabilities than resident evaluations, indicating that peer physicians are more reliable than residents for assessing bedside teaching.     

Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions

Background: It remains unclear which item format would best suit the assessment of clinical reasoning: context-rich single best answer questions (crSBAs) or key-feature problems (KFPs). This study compared KFPs and crSBAs with respect to students’ acceptance, their educational impact, and psychometric characteristics when used in a summative end-of-clinical-clerkship pediatric exam.

Methods: Fifth-year medical students (n?=?377) took a computer-based exam that included 6–9 KFPs and 9–20 crSBAs which assessed their clinical reasoning skills, in addition to an objective structured clinical exam (OSCE) that assessed their clinical skills. Each KFP consisted of a case vignette and three key features using a “long-menu” question format. We explored students’ perceptions of the KFPs and crSBAs in eight focus groups and analyzed statistical data of 11 exams.

Results: Compared to crSBAs, KFPs were perceived as more realistic and difficult, providing a greater stimulus for the intense study of clinical reasoning, and were generally well accepted. The statistical analysis revealed no difference in difficulty, but KFPs resulted more reliable and efficient than crSBAs. The correlation between the two formats was high, while KFPs correlated more closely with the OSCE score.

Conclusions: KFPs with long-menu exams seem to bring about a positive educational effect without psychometric drawbacks.     

Use of interaction cards to evaluate clinical performance

Currently in-training evaluation in Kuwait depends on the use of the global rating scale at the end of clinical rotation clerkships. Such a scale is inconsistent, subjective, and suffers from deficiencies such as positive skewness of the distribution of ratings and poor reliability. The aim of the study was to assess the inter-rater variation and the reliability of the recently introduced Interaction Card (IC) method for evaluating clinical performance and to measure the agreement between trainees' overall performance evaluation by the currently used global rating scale and the IC summative evaluation. In the study, 370 evaluators encountered 50 trainees during their basic clinical training rotations (internal medicine, surgery, obstetrics and gynecology, and pediatrics) at six hospitals. A total of 9146 encounters were conducted focusing on six clinical performance domains: clinical skills (taking history, case sheet, and physical examination), professional behaviour, case presentation, diagnosis, therapy and handling of emergencies. The method demonstrated significant inter-rater variation in the overall IC ratings according to specialty, rank of evaluator and hospital (p < 0.001). The Interaction Card was found to be reliable, as shown by the internal consistency between the six domains (Cronbach's alpha = 0.914). There was low correlation (Spearman rank correlation coefficient, rs = 0.337), and low agreement (Kappa = 0.131) between the global rating scale and Interaction Cards summative evaluations. The IC method provided instantaneous formative feedback and summative evaluation for clinical performance to trainees. The method can be generalized to encompass training and examinations programmes for all categories of trainees in most clinical specialties.     

Applying sport psychology to improve clinical performance*

Preparedness for practice has become an international theme within Medical Education: for healthcare systems to maintain their highest clinical standards, junior doctors must “hit the ground running” on beginning work. Despite demonstrating logical, structured assessment and management plans during their undergraduate examinations, many newly qualified doctors report difficulty in translating this theoretical knowledge into the real clinical environment. “Preparedness” must constitute more than the knowledge and skills acquired during medical school. Complexities of the clinical environment overwhelm some junior doctors, who acknowledge that they lack strategies to manage their anxieties, under-confidence and low self-efficacy. If uncontrolled, such negative emotions and behaviors may impede the delivery of time-critical treatment for acutely unwell patients and compound junior doctors’ self-doubt, thus impacting future patient encounters. Medical Education often seeks inspiration from other industries for potential solutions to challenges. To address “preparedness for practice,” this AMEE Guide highlights sport psychology: elite sportspeople train both physically and psychologically for their discipline. The latter promotes management of negative emotions, distractions and under-confidence, thus optimizing performance despite immense pressures of career-defining moments. Similar techniques might allow junior doctors to optimize patient care, especially within stressful situations. This AMEE Guide introduces the novel conceptual model, PERFORM, which targets the challenges faced by junior doctors on graduation. The model applies pre-performance routines from sport psychology with the self-regulatory processes of metacognition to the clinical context. This model could potentially equip junior doctors, and other healthcare professionals facing similar challenges, with strategies to optimize clinical care under the most difficult circumstances.     

Diffusion of innovation and longitudinal integrated clerkships: Results of the clerkship directors in internal medicine annual survey

Purpose: Longitudinal integrated clerkships (LICs) are innovative educational models that emphasize medical student continuity with patients, preceptors, peers, and health systems. We characterize LIC growth in the US and interpret the growth using Rogers’ Diffusion of Innovation Theory.

Methods: In 2015, we surveyed 123?US allopathic medical schools affiliated with Clerkship Directors in Internal Medicine (CDIM). The organization’s annual survey was supplemented with questions aimed to quantify the number of current and planned LICs and to determine the intended purpose of starting LICs.

Results: Of the 94 (out of 123 possible) schools which were responding, 35 (37%) have at least one LIC of six months or greater; of these 20 are year-long. Nineteen schools are engaged in planning a new LIC or increasing the number of students in an LIC. At least 45 (48%) responding schools will have LICs in future years. Respondents report implementing LICs to foster continuity of care, support patient-centeredness, advance inter-professional education, and address workforce shortages.

Conclusions: The number of LICs is increasing across the US. We considered the data through the lens of Diffusion of Innovation Theory, speculated that LIC growth has reached “critical mass,” and considered why the LIC innovation may be self-sustaining.     

Students' perceptions of time spent during clinical rotations

Some rotations during clinical education are characterized by a high number of hours spent per week in the hospital because students panicipate in a hospital on-call system, i.e. hours beyond usual working hours, e.g. at night. However, students complain about spending too many hours in the hospital on non-instructive activities. This study was undertaken to investigate differences among rotations in time spent in hospital, in on-call hours, in self-study and in non-instructive activities and to investigate the relationship between time spent on the various activities and the overall effectiveness as perceived by students. A questionnaire was administered to students at the end of various clinical rotations. Rotations differ considerably in time spent in hospital, on call (i.e. beyond usual working hours), on self-study and on non-instructive activities. In some rotations students report spending on average 18-20% of their time in the hospital on non-instructive activities. Furthermore, the numbers of hours spent in hospital do not correlate with the overall effectiveness as perceived by students, unless the numbers of hours spent in non-instructive activities are taken into account. The effectiveness of a rotation does not automatically improve if students spend more hours in hospital. The number of hours spent on non-instructive activities should be reduced in some rotations and should be kept as low as possible. Further research is needed to find out which activities are perceived as non-instructive.