The effect of three commercial coaching courses on Step One USMLE performance

BACKGROUND: A substantial industry exists to provide formal review courses for Step 1 of the United States Medical Licensing Examination (USMLE). There are limited data on the usefulness of these courses. AIM: To determine whether or not student participation in a commercial coaching course improves performance on Step 1 of the USMLE. METHODS: Scores achieved by 468 students on the National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination (CBSE) were used to predict a score on Step 1 of the USMLE. The NBME is the organisation that prepares and administers the USMLE. Predicted USMLE scores were then regressed against the actual scores achieved by the students on Step 1. The students were divided into 2 groups: those who took a 3-4-week live commercial coaching course and those who studied on their own. RESULTS: The regression lines for the 369 students who studied on their own and the 99 students who took a commercial coaching course were statistically indistinguishable. The analysis was powerful enough to have picked up a difference of 1% on average (P = 0.05) or 2 questions out of the 350 constituting Step 1 of the USMLE. Neither the students who performed above average nor those who performed below average on the CBSE improved their performance on Step 1 as a result of the coaching courses. CONCLUSIONS: Students who take a live, 3-4-week commercial coaching course to improve performance on Step 1 of the USMLE do not achieve higher scores than students who study on their own. Students should strongly consider whether or not a substantial investment in time and money for a commercial coaching course is justified in the light of such meagre returns.     

Validation of a performance assessment instrument in problem-based learning tutorials using two cohorts of medical students

Although problem-based learning (PBL) has been widely used in medical schools, few studies have attended to the assessment of PBL processes using validated instruments. This study examined reliability and validity for an instrument assessing PBL performance in four domains: Problem Solving, Use of Information, Group Process, and Professionalism. Two cohorts of medical students (N = 310) participated in the study, with 2 years of PBL evaluation data extracted from archive rated by a total of 158 faculty raters. Analyses based on generalizability theory were conducted for reliability examination. Validity was examined through following the Standards for Educational and Psychological Testing to evaluate content validity, response processes, construct validity, predictive validity, and the relationship to the variable of training. For construct validity, correlations of PBL scores with six other outcome measures were examined, including Medical College Admission Test, United States Medical Licensing Examination (USMLE) Step 1, National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination, NBME Comprehensive Clinical Science Examination, Clinical Performance Examination, and USMLE Step 2 Clinical Knowledge. Predictive validity was examined by using PBL scores to predict five medical school outcomes. The highest percentage of PBL total score variance was associated with students (60 %), indicating students in the study differed in their PBL performance. The generalizability and dependability coefficients were moderately high (Ep² = .68, ? = .60), showing the instrument is reliable for ranking students and identifying competent PBL performers. The patterns of correlations between PBL domain scores and the outcome measures partially support construct validity. PBL performance ratings as a whole significantly (p < .01) predicted all the major medical school achievements. The second year PBL scores were significantly higher than those of the first year, indicating a training effect. Psychometric findings provided support for reliability and many aspects of validity of PBL performance assessment using the instrument.     

Relationships between preclinical course grades and standardized exam performance

Success in residency matching is largely contingent upon standardized exam scores. Identifying predictors of standardized exam performance could promote primary intervention and lead to design insights for preclinical courses. We hypothesized that clinically relevant courses with an emphasis on higher-order cognitive understanding are most strongly associated with performance on United States Medical Licensing Examination Step exams and National Board of Medical Examiners clinical subject exams. Academic data from students between 2007 and 2012 were collected. Preclinical course scores and standardized exam scores were used for statistical modeling with multiple linear regression. Preclinical courses were categorized as having either a basic science or a clinical knowledge focus. Medical College Admissions Test scores were included as an additional predictive variable. The study sample comprised 795 graduating medical students. Median score on Step 1 was 234 (interquartile range 219–245.5), and 10.2 % (81/795) scored lower than one standard deviation below the national average (205). Pathology course score was the strongest predictor of performance on all clinical subject exams and Step exams, outperforming the Medical College Admissions Test in strength of association. Using Pathology score <75 as a screening metric for Step 1 score <205 results in sensitivity and specificity of 37 and 97 %, respectively, and a likelihood ratio of 11.9. Performance in Pathology, a clinically relevant course with case-based learning, is significantly related to subsequent performance on standardized exams. Multiple linear regression is useful for identifying courses that have potential as risk stratifiers.     

Correlation of United States Medical Licensing Examination and Internal Medicine In-Training Examination performance

The Internal Medicine In-Training Examination (ITE) is administered during residency training in the United States as a self-assessment and program assessment tool. Performance on this exam correlates with outcome on the American Board of Internal Medicine Certifying examination. Internal Medicine Program Directors use the United States Medical Licensing Examination (USMLE) to make decisions in recruitment of potential applicants. This study was done to determine a correlation of USMLE Steps 1, 2 and 3 results with ITE scores in each level of Internal Medicine training. A retrospective review of all residents graduating from an Internal Medicine program from 1999 to 2006 was done. Subjects included had data for all USMLE Steps and ITE during all years of training. Thirty-one subjects were included in the study. Correlations of USMLE Steps 1, 2 and 3 were done with ITE scores (percent correct) in each year of training. Pearson’s correlation coefficient (r) was determined for each pairing and a t test to determine statistical significance of the correlation was done. Statistical significance was defined as P value <0.05. The r values for USMLE Step 1 and ITE percent correct in PGY I, II and III were 0.46, 0.55 and 0.51 respectively. Corresponding r values for USMLE Step 2 and ITE percent correct were 0.79, 0.70 and 0.72; for USMLE Step 3 these values were 0.51, 0.37 and 0.51 respectively for each training year. USMLE scores are correlated with ITE scores. This correlation was strongest for USMLE Step 2.     

Problem-based learning versus lecture-based learning in a course of basic pharmacology: a controlled, randomized study

Since its first implementation in a medical programme at McMaster University, Canada, problem-based learning (PBL) has become a well-established means of teaching and learning medicine. Extensive research has been conducted and a number of strengths of the method are well supported. Several items, however, remain unclear although there is evidence that no relevant difference exists in factual knowledge among students from PBL and traditional curricula, a controlled, randomized study has not been conducted to address this issue. The Medical Faculty of the University of Cologne is in the process of integrating elements of PBL into its curriculum. In the spring term of 1997, after seven semesters of experience with PBL supplementing the traditional course of basic pharmacology, we did for the first time use PBL instead of the lecture-based course (LBL) and conducted a controlled prospective study to determine the effects of this intervention. One-hundred and twenty-three students were randomly assigned to either PBL (n = 63), with tutorial groups of up to nine students, or to the traditional, lecture-based course (n = 60). Analysis of the results of both groups in the examination of basic pharmacology, consisting of multiple-choice and short-essay questions, revealed similar scores with a tendency favouring PBL students in the category of short-essay questions. Hence, it seems clear that PBL does not imply a disadvantage in terms of factual knowledge. Students considered PBL to be an effective learning method and favoured it over the lecture format. Furthermore, students reported positive effects of PBL in terms of use of additional learning resources, interdisciplinarity, team work and learning fun.     

What do students want? The types of learning activities preferred by final year medical students

CONTEXT: A well-aligned curriculum has consistent focus on curricular goals, teaching/learning activities and assessment. Poor alignment affects the way students budget learning time and may interfere with achievement of curricular goals. We noticed students' poor attendance in clinical clerkships prior to final examinations and hypothesised that they were responding to curricular misalignment. OBJECTIVES: To quantify the extent to which students and tutors perceived a mismatch between activities pertaining to their current clerkship and those they thought necessary to prepare for final examinations and internship. SUBJECTS: Medical students in final (sixth) year clerkships in medicine, surgery and critical care and clinical tutors. METHOD: A self-report questionnaire on time spent in clerkship activities was administered 3 months before the final examinations. RESULTS: The amounts of time spent on current activities were fairly evenly spread over teaching, study and self-directed patient contacts, and observing patient care (ward rounds, operating theatre, accident and emergency, outpatient department and clinical meetings). Less time was available for recreation. Students and tutors concurred, independently, that good examination and internship preparation required a shifting of the balance. Examination preparation redistributed time from observing patient care and recreation to study and self-directed patient contacts. Internship preparation redistributed time from teaching and recreation to observing patient care. CONCLUSION: Students and tutors perceived that current activities were not well aligned with assessment requirements but were better aligned with the requirements of internship. If we want students to direct their attention towards curricular goals, we need to bring goals, teaching/learning activities and assessment into alignment.     

Non-cognitive characteristics of medical students: entry to problem-based and lecture-based curricula

The effectiveness of problem-based learning (PBL) versus lecture-based learning (LBL) continues to be debated all over the world. These arguments have often been based on students' cognitive measures of performance. Little emphasis has been placed on non-cognitive factors that may directly or indirectly affect the medical school performance of students in either curriculum. The purpose of this study was to (1) document possible differences in student cognitive and non-cognitive characteristics at entry between the two curricula and (2) to explore the relationships that exist between cognitive and noncognitive factors. Data were obtained from three medical school classes (   n = 281  ). The results indicate that students who entered the PBL curriculum at this medical school had higher total Medical College Admission Test and undergraduate grade point average than students who entered the LBL curriculum. Students who entered the PBL curriculum were also more self-sufficient and were more likely to do well in individualistic and less structured settings. There were no strong correlations between cognitive and non-cognitive variables. Before conclusions can be drawn about the effectiveness of either PBL or LBL curricula, we need to document patterns in entry characteristics to control for a priori differences that affect student performance.     

Introducing a problem-based unit into a lifespan nutrition class using a randomized design produces equivocal outcomes

Problem-based learning (PBL) is a tutorial, student-centered, problem-driven educational strategy adopted by medical and allied health educators to positively influence self-directed learning, critical thinking, and learning behavior. PBL was examined in dietetics education through random assignment of 32 undergraduate dietetics students for two weeks to either a problem-based or a lecture-based case format for the infant and elderly units of a 16-week lifespan nutrition course. Random assignment followed stratification for gender and earlier course examination performance. Expert validation of PBL curricular components and noted differences in discussion structure and information resources verified curriculum distinctiveness. Main outcome measures were pre- and post-Cognitive Behavior Survey scores for memorization, reflection, and positive learning experience scales, unit and course evaluations and unit examination scores. Students in problem-based modules demonstrated greater gains in reflective thinking with stable memorization, suggesting improved critical thinking skills. Tenets that problem-based learning promotes knowledge retention and provides a more positive learning experience were not upheld. Knowledge acquisition was not hindered by a problem-based approach. We conclude that gains in reflective thinking and evidence of increased self-directed learning argue for inclusion of PBL in dietetics curricula and that a problem-based education will help dietitians successfully respond to professional development needs.     

Problem-based learning of psychopathology in a traditional curriculum using multiple conceptual models

A course in basic psychopathology is described in which problem-based learning is implemented in small groups in a traditional medical school curriculum. Simulated patients are utilized to provide the problem data and to allow for practice in medical interviewing. The problem-based portion of the course focuses on the explanation of various psychiatric disorders using an integration of four conceptual models: the medical/biological; the behavioural/learning; the sociocultural; and the psychodynamic. Reception of the course, based on student evaluations as well as teachers' enthusiasm and participation, has been positive for the 6 years the course has existed in this format, lending support to the conclusion that problem-based methods can be successfully integrated into a traditional lecture-based curriculum.     

Empathy in medical students as related to academic performance,clinical competence and gender

CONTEXT: Empathy is a major component of a satisfactory doctor-patient relationship and the cultivation of empathy is a learning objective proposed by the Association of American Medical Colleges (AAMC) for all American medical schools. Therefore, it is important to address the measurement of empathy, its development and its correlates in medical schools. OBJECTIVES: We designed this study to test two hypotheses: firstly, that medical students with higher empathy scores would obtain higher ratings of clinical competence in core clinical clerkships; and secondly, that women would obtain higher empathy scores than men. MATERIALS AND SUBJECTS: A 20-item empathy scale developed by the authors (Jefferson Scale of Physician Empathy) was completed by 371 third-year medical students (198 men, 173 women). METHODS: Associations between empathy scores and ratings of clinical competence in six core clerkships, gender, and performance on objective examinations were studied by using t-test, analysis of variance, chi-square and correlation coefficients. RESULTS: Both research hypotheses were confirmed. Empathy scores were associated with ratings of clinical competence and gender, but not with performance in objective examinations such as the Medical College Admission Test (MCAT), and Steps 1 and 2 of the US Medical Licensing Examinations (USMLE). CONCLUSIONS: Empathy scores are associated with ratings of clinical competence and gender. The operational measure of empathy used in this study provides opportunities to further examine educational and clinical correlates of empathy, as well as stability and changes in empathy at different stages of undergraduate and graduate medical education.