An analysis of the transfusion medicine content of the test of the National Board of Medical Examiners

A two-year (1984-1985) review of 3,764 questions appearing on Part I and Part II of the National Board of Medical Examiners (NBME) indicated 140 (4%) were transfusion medicine (TM)-related. Fewer questions on TM appeared on Part II than Part I (45 vs. 95, P less than 0.001). Unexpectedly, the lowest proportion of questions (excluding psychiatry) in the Part II examination was in the surgery section. Part I NBME appears to adequately reflect and assess the TM content of the medical school curricula, but the number and distribution of questions in Part II of the NBME does not reflect recent changes in the clinical transfusion medicine curricula of many United States medical schools.     

Comparing the transfusion medicine content of the NBME's examinations in 1984-1985 and 1989-1990.

Recognition of the seriousness of transfusion-transmitted diseases has been demonstrated by U.S. medical schools through the integration of transfusion medicine (TM) content into their curricula. To evaluate the degree to which these changes in curricula have been reflected in the National Board of Medical Examiners' (NBME) examinations, a study conducted in 1991 evaluated the proportions of TM-related items on Parts I and II of the NBME examinations for 1984-1985 versus 1989-1990. Both Part I (basic sciences) and Part II (clinical sciences) demonstrated significant gains in TM items between the comparison periods (p less than .001), with Part II having the higher gain. An analysis of students' knowledge revealed that students in 1989-1990 tended to perform better on TM items than on examination items generally. The increases in TM content and student performance on TM items on the 1989-1990 examinations suggest that the national effort to expand and improve teaching of TM in U.S. medical schools has been effective.     

The strategic impact of a changing curriculum and learning environment on medical students' academic performance

In recent years, the administration of Meharry Medical College, School of Medicine (SOM), Nashville, Tennessee, recognized the need to modify the curriculum to help improve student academic performance especially on the National Board of Medical Examiners (NBME) US Medical Licensing Examination (USMLE) steps 1 and 2. Thus, a number of changes occurred with respect to the traditional curriculum in the SOM, resulting in an integrated organ system-based curriculum design. The change in the learning environment was studied to determine the impact on performance after the introduction of the integrated organ system-based curriculum as compared to that of the traditional curriculum. With the utilization of a cadre of variables, it was believed that the strategic impact anticipated would provide a predictive validity profile to assist in the identification of students "at risk" of failure so that proactive intervention methodology could be made available to facilitate the students' successful progression during matriculation in the SOM. The purpose of this study was to analyze whether students trained with the integrated organ systems curriculum perform better than students trained with the traditional medical school curriculum on the medical education preclinical subject board examinations, and the NBME USMLE steps 1 and 2 examinations. From the 584 students studied in the control group (graduation classes for years 2005, 2006, and 2007) and the intervention group (graduation classes for years 2008, 2009, and 2010), significant improvement in performance on the NBME USMLE steps 1 and 2 examinations was noted following the introduction of the integrated organ system-based curriculum particularly among "at-risk" students. Data access availability from the School of Medicine of Meharry Medical College automatically gave reason for a preferential comparative relationship and study of the resulting strategic impact on cohorts graduating in years 2005-2010. Thus, this longitudinal retrospective review was to determine whether or not students' academic performance profiles might provide some valid predictive information to help identify "at-risk" students early in their pursuit of a health professions career.     

Clerkship timing and disparity in performance of racial-ethnic minorities in the medicine clerkship

PURPOSE: To determine the effect of the timing of the medicine clerkship on academic performance in different racial-ethnic student groups. METHOD: Performance was measured by the average assessment of clinical preceptors, an OSCE (objective structured clinical examination), and the NBME (National Board of Medical Examiners) medicine subject examination. Outcomes were analyzed by student racial-ethnicity and clerkship sequence. RESULTS: Of the 650 students who took the clerkship over four years, 6.9% were African American, 34.6% were Asian-Pacific Islander, 9.1% were Hispanic and 49.4% were white. African American and Hispanic students were in the earliest clerkship sequence 46.7% and 30.5% of the time, respectively, compared to 20% of Asian-Pacific Islanders and 27.4% of white students. Academic performance improved with time and varied among the racial-ethnic groups. All groups achieved higher scores in the NBME medicine examination later in the year but scores of African American and Hispanic students increased to a greater degree than other students. CONCLUSION: Sometimes, a "few points" on the NBME medicine examination can affect students' final grades and alter their attractiveness to competitive residency training programs. All students, but African American and Hispanic students, in particular, can significantly improve their scores in the NBME medicine examination by taking the clerkship later in the year. Students should be counseled regarding the timing effect and methods to neutralize the disparity should be considered.     

Early clinical experience enhances third-year pediatrics clerkship performance.

PURPOSE: To determine whether clinical experiences in the preclinical years improve medical students' performances in a third-year clerkship. METHOD: A cohort study reviewed the pediatrics clerkship performances of 400 Eastern Virginia Medical School students in the graduating classes of 1996 through 1999. The first two classes completed a traditional preclinical curriculum with limited clinical experience. The final two classes participated in a mentorship program that provided 18 months of early clinical experience, consisting of one-on-one half-day sessions every other week with a generalist community faculty. The authors compared the clinical clerkship performances of the groups using clinical skills (CS) scores, history and physical examination (H&P) scores, and scores on the NBME pediatrics shelf examination. They also looked at the mean MCAT and USMLE scores for each group. The authors also looked at the scores within each class, comparing students who completed one of the first two pediatrics clerkship rotations with their classmates who completed clerkships later in the academic year. RESULTS: The students' NBME scores rose significantly (p < .05, r2 = 0.95) over the four-year study, paralleling a significant rise in MCAT scores (p < .03, r2 = 0.73). The CS and H&P scores did not rise. Students who had the traditional preclinical curriculum and who completed their clerkships early in the year had significantly lower CS and H&P scores than did their classmates. In contrast, the scores of students who had the early clinical experiences did not differ significantly according to the timing of their rotation. CONCLUSION: Students who had participated in a mentorship program that provided early clinical experience demonstrated significantly improved clinical skills in the pediatrics clerkship early in the academic year.     

Participation in a premedical summer program for underrepresented-minority students as a predictor of academic performance in the first three years of medical school: two studies.

PURPOSE: To determine whether the performances of underrepresented minority students in the Medical Education and Development Program (MEDP) at the University of North Carolina Chapel Hill (UNC-CH) School of Medicine (a structured, nine-week summer premedical program that simulates the medical school's first year curriculum) predicted their academic performances in the first three years of medical school. METHOD: These two studies looked separately at the predictive value of students' rankings (top quarter or top half) within their MEDP cohorts. The first study involved 165 students who had participated in the MEDP from 1981 to 1990 and who then matriculated at UNC-CH. Using backward elimination logistic regression models, the author determined whether those rankings and other, more traditional academic performance predictors predicted three types of academic difficulty during the first two years of medical school: (1) required participation in summer review; (2) deceleration; and (3) dismissal. The predictive validity of each regression model was assessed by calculating the sensitivity and the positive predictive value. The second study involved 135 students who had participated in the MEDP from 1981 to 1990 and who had finished their third-year clinical clerkships at UNC-CH. Forty-four of those students had taken the National Board of Medical Examiners (NBME) Part II examination. Using Spearman correlations, the Student t test, and the chi-square test, the author determined the simple relationships among MEDP ranking, traditional premedical academic predictors, and third-year clinical performance (as measured by clerkship grades and NBME scores). The author determined the best predictors of third-year clinical performance using stepwise backward-elimination linear regression models. RESULTS: In both studies, a student's ranking within his or her MEDP cohort was the strongest, if not the only, predictor of medical school academic performance. CONCLUSION: These studies suggest that structured summer premedical programs such as UNC-CH's MEDP are quite good at determining whether participants will be able to handle the academic rigors of medical school.     

Assessing medical students' clinical sciences knowledge in France: a collaboration between the NBME and a consortium of French medical schools.

PURPOSE: The French government, as part of medical education reforms, has affirmed that an examination program for national residency selection will be implemented by 2004. The purpose of this study was to develop a French multiple-choice (MC) examination using the National Board of Medical Examiners' (NBME) expertise and materials. METHOD: The Evaluation Standardisée du Second Cycle (ESSC), a four-hour clinical sciences examination, was administered in January 2002 to 285 medical students at four university test sites in France. The ESSC had 200 translated and adapted MC items selected from the Comprehensive Clinical Sciences Examination (CCSE), an NBME subject test. RESULTS: Less than 10% of the ESSC items were rejected as inappropriate to French practice. Also, the distributions of ESSC item characteristics were similar to those reported with the CCSE. The ESSC also appeared to be very well targeted to examinees' proficiencies and yielded a reliability coefficient of.91. However, because of a higher word count, the ESSC did show evidence of speededness. Regarding overall performance, the mean proficiency estimate for French examinees was about 0.4 SD below that of a CCSE population. CONCLUSIONS: This study provides strong evidence for the usefulness of the model adopted in this first collaborative effort between the NBME and a consortium of French medical schools. Overall, the performance of French students was comparable to that of CCSE students, which was encouraging given the differences in motivation and the speeded nature of the French test. A second phase with the participation of larger numbers of French medical schools and students is being planned.     

Using a standardized-patient examination to establish the predictive validity of the MCAT and undergraduate GPA as admissions criteria

Performance of senior medical students on an objectively scored examination of clinical competence based on standardized-patient cases was used to assess the predictive validity of the two most commonly used admission measures, the Medical College Admissions Test and the undergraduate grade point average. The students were in the classes of 1986 and 1987 at Southern Illinois University School of Medicine. The correlations of the admissions measures with clinical performances were quite weak, and none of the admissions measures consistently showed a clear advantage as a predictor of clinical performance. Correlations of the admissions measures with scores on National Board of Medical Examiners (NBME) Part I and Part II examinations were small to moderate, although somewhat larger than the correlations with clinical performances. Correlations were corrected for attenuation due to differential unreliabilities of the clinical examination results and the scores on NBME examinations, and for restriction of range due to the stringent medical school selection process. Corrected correlations were small to moderate and showed the same pattern as the uncorrected ones. The study documents that traditional admissions measures are useful for selecting students who will perform effectively in clinical as well as basic science settings.     

Comparison of medical school performances and career plans of students with broad and with science-focused premedical preparation.

This paper reports (1) a method for classifying students according to the breadth of their premedical preparation and (2) a comparison of the medical school performances and career plans of the students thus classified. The method was developed in 1987, in part by using input from a small but representative sample of admission officers. Students were grouped according to undergraduate major, ratio of nonscience-to-science course hours, and extracurricular involvement. After tentatively classifying all individuals who had entered U.S. medical schools in 1981 as having either broad or science-focused preparation, the author compared the two most distinct groups selected from a random sample of the individuals in each classification: 59 individuals constituted the final broadly prepared group, and 73, the science-focused group. The science-focused group attained higher mean scores (p less than .05) on three science sections of the National Board of Medical Examiners (NBME) Part I examination, and the broadly prepared group scored higher on the Behavioral Sciences section (p less than .05). No other significant difference was evident between the groups' mean scores on the NBME Parts I, II, or III, or in the groups' rates of experiences of academic difficulty, specialty choice distributions, or percentages of individuals deciding to pursue research careers. The author concludes that this method of classifying students is useful and that the students with less premedical focus in the sciences were able to perform well.     

Evaluation of computer-assisted instruction in histology: effect of interaction on learning outcome

The purpose of this study was to determine better strategies for the design and use of computer-assisted instruction (CAI) in health science subjects that require visual learning. Evaluation of current use of CAI was focused on three CD-based modules developed to teach histological images to beginning medical students at multiple sites. For internal control, students' learning outcomes and perceived effectiveness were analyzed with their demographic characteristics, computer attitude, computer experience, and learning behaviors being considered. Results indicated that students who used at least two different CAI programs scored significantly higher on the final examination than those who used only the CAI tool designed by their site's instructor. Further investigation indicated that students might have benefited from the interactive features of a specific CAI tool. Such scaffolds could have successfully supported encoding processes while students were restructuring their mental models. In addition, students perceived the CAI programs to be more effective when the tools were fully integrated into the curriculum. Perceived module effectiveness was significantly correlated with examination performance, suggesting a well-designed and appropriately used CAI tool may help students achieve not only learning efficiency, but also better learning outcome.     

Evaluation of procedural skills of internal medicine residents

In 1983 the American Board of Internal Medicine mandated that training programs assess residents in procedural skills prior to board certification. Beginning in 1984, the University of Hawaii developed an assessment and testing program that consists of a three-phase evaluation process for qualifying internal medicine residents in basic procedural skills: an observation period spanning the three-year residency, a 100-question multiple-choice examination, and a slide-identification examination involving simple identification of body fluid elements. From 1984 through 1987 the mean scores of both examinations were analyzed for each level of residency and for each year of testing to assess whether curriculum changes were effective. There was a statistically significant improvement (p less than .005) in performance on both the multiple-choice and the slide-identification examinations as the residents progressed through the program. No difference in performance was found for the multiple-choice examination between calendar years, but a significant difference (p less than .01) was found for the slide-identification examination between calendar years. Despite perceptions by the faculty that their increased emphasis on the learning of procedural skills has been successful, the multiple-choice examination results over time do not support this belief. These findings suggest that residents need more structured curriculum guidelines to aid their learning and faculty members require better direction in the educational goals for procedural skills training.     

The quality of in-house medical school examinations.

PURPOSE: Most medical schools test their students throughout the curriculum using in-house examinations written by the faculty who teach the courses. The authors assessed the quality of in-house examinations used in three U.S. medical schools. METHOD: In 1998, nine basic science examinations from the three schools were gathered and each question was subjected to quality assessment by three expert biomedical test developers, each of whom has had extensive experience in reviewing and evaluating questions for the United States Medical Licensing Examination (USMLE) Steps 1 and 2. Questions were rated on a five-point scale: 1 = tested recall only and was technically flawed to 5 = used a clinical or laboratory vignette, required reasoning to answer, and was free of technical flaws. Each rater made independent assessments, and the mean score for each question was calculated. Mean quality scores for National Board of Medical Examiners (NBME) who were trained question writers were compared with the mean scores for question writers without NBME training. The raters' quality assessments were made without knowledge of the test writers' training background or the study's hypothesis. RESULTS: A total of 555 questions were analyzed. The mean score for all questions was 2.39 +/- 1.21. The 92 questions written by NBME-trained question writers had a mean score of 4.24 +/- 0.85, and the 463 questions written by faculty without formal NBME training had a mean score of 2.03 +/- 0.90 (p <.01). CONCLUSIONS: The in-house examinations were of relatively low quality. The quality of examination questions can be significantly improved by providing question writers with formal training.     

Effects of an expanded medical curriculum on the number of graduates practicing in a rural state

In 1973 the University of North Dakota School of Medicine (UNDSM), following the national trend toward four-year medical programs, expanded its previous two-year medical school curriculum to include all four years of medical education. It was hoped that this change, along with a renewed emphasis on primary care-oriented residency training within the state, would encourage medical students to establish practices within the state. In 1985 the UNDSM's Center for Rural Health mailed questionnaires to the 2,230 living graduates of the UNDSM to document a variety of their personal and practice characteristics. Based on the responses to the 924 completed questionnaires, the authors found that (1) the students from rural North Dakota were more likely than were urban students to practice in rural areas of the state, as were the students with primary care specialty training; and (2) the alumni completing residencies in North Dakota following the curriculum expansion (1976-1985) were more than twice as likely to establish practices in North Dakota. It was concluded that recruiting medical students (preferably in-state "natives") from rural areas, training them in primary care specialty areas, and enabling them to remain in North Dakota for the duration of their medical training (including residency training) combined to exert a considerable "retaining" effect on the UNDSM alumni.     

T-lymphocyte polyclonal proliferation: effects of stress and stress response style on medical students taking national board examinations

This study investigated the effects of stress and stress response style on T-lymphocyte polyclonal proliferation (TLPP). The TLPP levels of 15 medical students taking NBME Part I were compared to those of a matched (on age and sex) sample of students not taking exams and hospital employees. All subjects completed the Impact of Events Scale to measure recent stress levels and style of responding to stress. TLPP data for both experimental and control groups, pre- and post-boards, were analyzed via a 2 X 2 factorial analysis of variance. Results indicated a significant main effect for the NBME stress vs control conditions and a significant interaction between stress response style (avoidance vs intrusion) and NBME stress vs control conditions. TLPP levels were significantly depressed (relative to the controls) for the NBME stress group 1 week after examinations. Furthermore, those students whose stress response style was characterized by intrusion were significantly more immunosuppressed than those characterized by an avoidance stress response style. Four- and six-week follow-up measures of TLPP were taken on 10 and 11 NBME group students, respectively. Follow-up results, analyzed via t tests, indicated that TLPP levels were significantly suppressed for 4 weeks but returned to initial normal levels at 6 weeks. Results are discussed in terms of needed research examining the mechanism of stress intruders' susceptibility to stress-induced immunosuppression and the possible clinical significance of brief periods of stress-induced decrements in TLPP levels.     

The New Mexico experiment: educational innovation and institutional change

Over the past ten years the University of New Mexico School of Medicine has conducted an educational experiment featuring learner-centered, problem-based, community-oriented learning. The experiment was introduced into an established institution by means of an innovative educational track running parallel to the more conventional curriculum. Students in the innovative track, compared with those in the conventional tract, tended to score lower on the National Board of Medical Examiners (NBME) Part I examination (basic sciences) and higher on NBME Part II (clinical sciences), received higher clinical grades on clinical clerkships, and experienced less distress. They were more likely than conventional-track students to retain their initial interest in or switch their preference to careers in family medicine. The parallel-track strategy for introducing curriculum reform succeeded in fostering institutional acceptance of continuing educational innovation. Generic steps in overcoming institutional barriers to change are identified.     

Medical gross anatomy as a predictor of performance on the USMLE Step 1

Traditional predictors of medical school performance, such as Medical College Admission Test (MCAT) scores and grade point averages, are often used during the admissions process to help identify the prospective students who are most likely to complete the basic science portion of the curriculum successfully. Here we analyzed the admissions files and student records of 285 first-year medical students who matriculated at the University of California at Davis School of Medicine between 1999 and 2001 to determine if performance in medical gross anatomy is a similar, if not better, predictor of performance on the United States Medical Licensing Examination (USMLE) Step 1 than traditional predictors used by medical school admissions committees. Though MCAT scores and grade point averages were correlated with scores on the USMLE Step 1, only the score on the biological science section of the MCAT was significantly correlated with passing the licensing examination. In contrast, class rank in medical gross anatomy and the score on a gross anatomy comprehensive final examination were correlated both with scores on the USMLE Step 1 and passing the examination. Our results indicate that medical schools should consider performance in medical gross anatomy just as much, if not more, than traditional predictors of medical school performance when trying to identify students who may need more time or tutoring to pass the licensing examination.     

Natural history of knowledge deficiencies following clerkships.

PURPOSE: To determine the prognosis of failing a clerkship final examination. METHOD: From 1994-99, 48 students at the Uniformed Services University of the Health Sciences (6%) failed the end-of-clerkship National Board of Medical Examiners' (NBME) subject exam, but otherwise had acceptable medicine clerkship performances. All of these students were retested after self-directed study; those who failed the retest were prescribed fourth-year medicine before a second retest. The authors compared median NBME subject exam scores and number of weeks of self-directed study time from the initial exam to the retest for those who passed and those who failed the retest. RESULTS: 40 students (83%) passed and eight students (17%) failed the first retest. There was no difference between the initial median NBME score (303 versus 295, p = ns) or length of self-directed study time (25 weeks versus 18 weeks, p = ns) between those who passed and failed the retest, respectively. After fourth-year medicine, all eight students passed the second retest of the subject exam. Four of the 48 students failed USMLE Step 2. Compared with those who passed Step 2, these four students had a similar median initial NBME subject exam score (293 versus 291, p = ns), but had a lower median retest exam score (354 versus 405, p = ns). CONCLUSIONS: It appears reasonable to allow students with isolated knowledge deficiencies one attempt to retake a failed clerkship exam following a period of self-directed study. Confirmatory studies are needed.     

The Harvard Medical School-Cambridge integrated clerkship: an innovative model of clinical education.

The Harvard Medical School-Cambridge Integrated Clerkship (HMS-CIC) is a redesign of the principal clinical year to foster students' learning from close and continuous contact with cohorts of patients in the disciplines of internal medicine, neurology, obstetrics-gynecology, pediatrics, and psychiatry. With year-long mentoring, students follow their patients through major venues of care. Surgery and radiology also are taught longitudinally, grounded in the clinical experiences of a cohort of patients and in a brief immersion experience working directly with an attending surgeon. Students participate in weekly, case-based tutorials integrating instruction in the basic sciences with training to address the common and important issues in medicine, as identified by national organizations. In addition, they participate in a social science curriculum that focuses on self-reflection, communication skills, ethics, population sciences, and cultural competence. In the pilot year (July 2004 to July 2005), HMS-CIC students performed at least as well as traditional students in tests of content knowledge and skills, as measured by National Board of Medical Examiners (NBME) Subject Exams and the fourth-year Objective Structured Clinical Exam, and they scored higher on a year-end comprehensive clinical skills self-assessment examination, suggesting that they retained content knowledge better. From surveys, HMS-CIC students were much more likely to see patients before diagnosis and after discharge and to receive feedback and mentoring from experienced faculty than were their traditionally educated peers. HMS-CIC students expressed more satisfaction with their curriculum and felt better prepared to cope with the professional challenges of patient care, such as being truly caring, involving patients in decision making, and understanding how the social context affects their patients.     

The effects of acute test anxiety on NBME Part I performance.

This study examined the associated effects of acute test-taking anxiety on the performance of a class of second-year medical students who took Part I of the National Board of Medical Examiners (NBME) Examination. The study is a follow-up of a previous study that examined the related effects of chronic anxiety on NBME Part I performance. The purpose of this study was to observe whether there was a relationship between acute anxiety and NBME performance and, further, if there were distinguishable differences in the effects of acute and chronic test anxiety on NBME performance. The first study indicated a significant relationship between chronic anxiety and NBME performance. The results of the present study indicate a relationship between acute test anxiety and NBME performance, but not to the extent associated with chronic anxiety, thus chronic test anxiety may be a more critical factor affecting test performance on critical examinations such as the NBME. Implications concerning anxiety and test performance are discussed.     

Traditional Admissions Variables as Predictors of Minority Students' Performance in Medical School �� A Cause for Concern

This study explored the relationship between traditional admissions criteria, performance in the first semester of medical school, and performance on the National Board of Medical Examiners'' (NBME) Examination, Part 1 for minority medical students, non-minority medical students, and the two groups combined. Correlational analysis and step-wise multiple regression procedures were used as the analysis techniques. A different pattern of admissions variables related to National Board Part 1 performance for the two groups. The General Information section of the Medical College Admission Test (MCAT) contributed the most variance for the minority student group. MCAT-Science contributed the most variance for the non-minority student group. MCATs accounted for a substantial portion of the variance on the National Board examination.