Using standardized patients as teachers: a concurrent controlled trial.

PURPOSE: To compare two methods of teaching physical assessment, a traditional faculty-taught course and a course with components taught by specially trained standardized patients (SPs), with respect to students' performances and costs. METHOD: Medical students in their second year and without preliminary course work in physical assessment were taught by faculty-led small groups. Students in their first year were taught by faculty-led lecture-demonstrations and exercises led by physical examination teaching associates (PETAs). Both groups of students were tested with a performance-based examination that involved six identical stations. The costs of both courses were calculated using faculty and SP salaries. RESULTS: There was no difference in students' performances on two of the stations, those involving the eye and abdominal examinations. The class that had been taught by PETAs, however, demonstrated a statistically significant performance advantage on the remaining four stations. The cost saving from using the PETAs was conservatively estimated at $24,155. CONCLUSION: Specially trained SPs can effectively teach the normal physical examination to medical students and are a less expensive alternative to traditional faculty small-group teaching methods.     

"Teach 1, do 1 ... better": superior communication skills in senior medical students serving as standardized patient--examiners for their junior peers.

PURPOSE: To see if senior medical students who had served as standardized patients (SP) demonstrated improvement in their own interpersonal communication skills. METHOD: From George Washington School of Medicine's class of 1996, 154 fourth-year students took a clinical practice examination that used professional SPs. Within the preceding six months, 28 of these students had been SP-examiners in similar examinations for first- and second-year medical students. The professional SPs rated the fourth-year examinees using checklists that measured five dimensions of interpersonal communication skills. Four of these five dimensions were identical to those measured on the examinations for which the fourth-year students had served as SPs. Hypothesizing that the SP-experienced seniors would outscore their inexperienced classmates on those four dimensions, but not on the fifth, the authors analyzed the fourth-year students' scores. P values were computed by the F test from a two-way analysis of variance. RESULTS: As predicted, the group with prior SP experience significantly outscored their inexperienced colleagues in each of the four expected dimensions of interpersonal communication skills, with p values ranging from .000 to .023. The score differential in these dimensions ranged from 3.8 to 11.8 percentage points. As further predicted, there was no significant difference between the scores of the two groups on the fifth dimension. CONCLUSION: Compared with their inexperienced peers, senior medical students with prior SP experience consistently demonstrated superior scores when their own communication skills were tested in a similar manner. The U.S. Medical Licensing Examinations propose to incorporate SP clinical examinations; in response, medical schools will use more SP examinations in their own curricula. Such examinations are expensive when using professional standardized patients; the authors believe that an SP program using senior medical students will prove an attractive alternative. Such programs may have the added advantage of making better communicators of senior medical student teachers as well as the students they teach.     

The influence of task formats on the accuracy of medical students' self-assessments.

PURPOSE: Accurate self-assessment is an essential skill for the self-directed learning activities and appropriate patient referral decisions of practicing physicians. However, many questions about the characteristics of self-assessment remain unanswered. One is whether self-assessment is a generalizable skill or dependent on the characteristics of the task. This study examines the self-assessment skills of medical students across two task formats: performance-based and cognitive-based. METHOD: In 1997 and 1998, fourth-year medical students at the University of Michigan assessed their own performances on ten stations of a clinical examination. The examination used two formats: performance tasks (the examination or history taking of standardized patients) and cognitive tasks (interpreting vignettes or test results and then answering paper-and-pencil questions). Three measures of self-assessment accuracy were used: a bias index (average difference between the students' estimates of their performances and their actual scores), a deviation index (average absolute difference between estimate and actual score), and an actual score-estimate-of-performance correlation (the correlation between the estimate and actual scores). RESULTS: The student bias and deviation indices were similar on the cognitive and the performance tasks. The correlations also indicated similarity between the two types of tasks. CONCLUSION: The results indicate that the format of the task does not influence students' abilities to self-assess their performances, and that students' self-assessment abilities are consistent over a range of skills and tasks. The authors also emphasize the importance of sampling tasks while conducting self-assessment research.     

The effect of a domestic violence interclerkship on the knowledge, attitudes, and skills of third-year medical students.

PURPOSE: To determine whether participation in an intensive domestic violence interclerkship (DVI) improved the knowledge, attitudes, and skills of two successive cohorts of students at the University of Massachusetts Medical School. METHOD: The authors measured the knowledge, attitudes, and skills pertaining to domestic violence of third-year students in the classes of 1997 and 1998 using a validated written examination administered before, immediately after, and six months after participation in a 3.5-day or two-day DVI, respectively; they compared the scores using paired t-tests. Nine months after the DVI, the students' domestic violence screening skills were measured by a performance-based assessment (OSCE); using unpaired t-tests, the authors compared the OSCE scores with those of a previous third-year class that had not participated in a DVI. Immediately after the OSCE, the students reported their levels of confidence in domestic violence screening and their satisfaction with the domestic violence curriculum; using chi-square analysis, those self-reports were compared with those of the class with no DVI. RESULTS: The students who participated in the DVIs immediately and significantly improved their knowledge, attitudes, and skills (p < .001), and fully or partially sustained those improvements six months later (p < .001). Nine months after the DVI, the students performed domestic violence screening more effectively (p < .001), expressed greater comfort with domestic violence screening (p < .001), and felt better-prepared by the curriculum to address domestic violence issues (p < .001) than did the students with no DVI. CONCLUSION: Participation in a short, focused DVI curriculum produced sustainable improvements in knowledge, attitudes, and skills that were successfully applied by third-year medical students to effective domestic violence screening. Interclerkships are an effective way to fit into the clinical curriculum those subjects that transcend the traditional biomedical domain and intersect all areas of medical practice.     

A performance-based method for early identification of medical students at risk of developing academic problems

The authors studied three classes at the Albert Einstein College of Medicine and found that students' performances on examinations administered during the third month (November) of medical school were highly predictive of their subsequent performances during the first two years of medical school. The investigation had two components: (1) a retrospective study of the classes of 1988 and 1989, which found that students' November grades from three first-year courses predicted 76% of the variance in the year-one weighted aggregate score and 41% for the year-two score, and (2) a prospective study of the class of 1992, in which three November of year one examination scores of the students in the lowest quarter of the class were highly predictive of their encountering substantial academic problems, with a sensitivity of .77 and a specificity of .99. This performance-based method was found to be more powerful than using the scores on the 1977 version of the Medical College Admission Test or the students' undergraduate grade-point averages, or both, in identifying individual students who were academically at risk.     

Comparing students' feedback about clinical instruction with their performances

Validation of students' feedback as a measure of teaching effectiveness has been problematic for courses teaching clinical skills. This is true in part because establishing a valid and reliable method of assessing students' mastery of clinical skills has been a stumbling block. Reported here is the correlation of students' performances on an objective structured clinical examination (OSCE) with previously and independently collected feedback from students. In 1987-88, 190 second-year medical students at the University of Minnesota Medical School--Minneapolis spent one fourth of a second-year clinical skills course on neurology randomly assigned to one of four teaching sites--hospitals A, B, C, and D. Following their rotations, 180 of the students completed usable feedback forms. The students were consistently and significantly more positive about the teaching at hospital A. At the end of the year, all 190 students were tested using an OSCE having 20 stations, four of which presented neurologic problems. The students who had the neurology course at hospital A performed better on all four neurology problems, and differences were statistically significant for two of the problems. Feedback in this case accurately reflected a more effective teaching program.     

Test security in examinations that use standardized-patient cases at one medical school

The use of performance-based examinations consisting of standardized-patient (SP) cases has increased greatly in recent years. These examinations are typically long and thus require the presentation of the same SP cases to several consecutive examinee groups. Consequently, concerns have arisen about the potential for violations of test security whereby students who were tested early in the examination period pass on information to students tested later. These concerns are addressed using data from the SP-based examinations administered to five classes (1986-1990) of senior medical students at Southern Illinois University School of Medicine. Because of the length of the examinations, each class was randomly divided into five groups and the examination was administered to one group at a time, requiring three days of testing time per group and three weeks of testing time per class. The results showed no consistent, systematic changes in case means across the five groups tested at different times throughout the examination period, and thus provide no evidence of serious, widespread violations of test security.     

The impact of alternating dissection on student performance in a medical anatomy course: Are dissection videos an effective substitute for actual dissection?

The way in which anatomy is taught to first year medical students at the University of North Carolina at Chapel Hill was recently changed, so that first year students are now divided into two groups that dissect alternately. The effect of this change on both written and practical test performance was analyzed by comparing grades from 2004 with those from the previous year (2003), when students performed all the dissections. A statistically significant decrease (P < or = 0.05) from 2003 was noted on three of the four written test scores in 2004, while practical examination scores in 2004 fluctuated from lower to higher than those in 2003, depending on the unit of material being covered. However, the number of students failing each of the examinations (written and practical combined) was statistically greater on only one of the four examinations in 2004. Scores of the two groups dissecting alternately in 2004 were essentially the same on the practical examinations. There was no difference in the number of questions answered incorrectly between these two groups in the two practical examinations where comparisons were made. Furthermore, students who dissected a particular structure did not score significantly better on practical questions concerning that structure than students who had not dissected it. The effect of the availability of step-by-step dissection videos on student practical examination scores is also discussed. We conclude that the change in the curriculum had a significant impact on the students' written examination performance, given the same material in the course. The reasons for this include student course load, increased need for self-study, and a loss of a learning opportunity in the dissection laboratory, all of which affect student comprehension and retention of the material and their ability to use it in problem solving.     

The Harvard Medical School-Cambridge integrated clerkship: an innovative model of clinical education.

The Harvard Medical School-Cambridge Integrated Clerkship (HMS-CIC) is a redesign of the principal clinical year to foster students' learning from close and continuous contact with cohorts of patients in the disciplines of internal medicine, neurology, obstetrics-gynecology, pediatrics, and psychiatry. With year-long mentoring, students follow their patients through major venues of care. Surgery and radiology also are taught longitudinally, grounded in the clinical experiences of a cohort of patients and in a brief immersion experience working directly with an attending surgeon. Students participate in weekly, case-based tutorials integrating instruction in the basic sciences with training to address the common and important issues in medicine, as identified by national organizations. In addition, they participate in a social science curriculum that focuses on self-reflection, communication skills, ethics, population sciences, and cultural competence. In the pilot year (July 2004 to July 2005), HMS-CIC students performed at least as well as traditional students in tests of content knowledge and skills, as measured by National Board of Medical Examiners (NBME) Subject Exams and the fourth-year Objective Structured Clinical Exam, and they scored higher on a year-end comprehensive clinical skills self-assessment examination, suggesting that they retained content knowledge better. From surveys, HMS-CIC students were much more likely to see patients before diagnosis and after discharge and to receive feedback and mentoring from experienced faculty than were their traditionally educated peers. HMS-CIC students expressed more satisfaction with their curriculum and felt better prepared to cope with the professional challenges of patient care, such as being truly caring, involving patients in decision making, and understanding how the social context affects their patients.     

Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study

ABSTRACT: BACKGROUND: Over two-thirds of UK medical schools are augmenting their selection procedures for medical students by using the United Kingdom Clinical Aptitude Test (UKCAT), which employs tests of cognitive and non-cognitive personal qualities, but clear evidence of the tests' predictive validity is lacking. This study explores whether academic performance and professional behaviours that are important in a health professional context can be predicted by these measures, when taken before or very early in the medical course. METHODS: This prospective cohort study follows the progress of the entire student cohort who entered Hull York Medical School in September 2007, having taken the UKCAT cognitive tests in 2006 and the non-cognitive tests a year later. This paper reports on the students' first and second academic years of study. The main outcome measures were regular, repeated tutor assessment of individual students' interpersonal skills and professional behaviour, and annual examination performance in the three domains of recall and application of knowledge, evaluation of data, and communication and practical clinical skills. The relationships between non-cognitive test scores, cognitive test scores, tutor assessments and examination results were explored using the Pearson product--moment correlations for each group of data; the data for students obtaining the top and bottom 20% of the summative examination results were compared using Analysis of Variance. RESULTS: Personal qualities measured by non-cognitive tests showed a number of statistically significant relationships with ratings of behaviour made by tutors, with performance in each year's objective structured clinical examinations (OSCEs), and with themed written summative examination marks in each year. Cognitive ability scores were also significantly related to each year's examination results, but seldom to professional behaviours. The top 20% of examination achievers could be differentiated from the bottom 20% on both non-cognitive and cognitive measures. CONCLUSIONS: This study shows numerous significant relationships between both cognitive and non-cognitive test scores, academic examination scores and indicators of professional behaviours in medical students. This suggests that measurement of non-cognitive personal qualities in applicants to medical school could make a useful contribution to selection and admission decisions. Further research is required in larger representative groups, and with more refined predictor measures and behavioural assessment methods, to establish beyond doubt the incremental validity of such measures over conventional cognitive assessments.     

Early clinical experience enhances third-year pediatrics clerkship performance.

PURPOSE: To determine whether clinical experiences in the preclinical years improve medical students' performances in a third-year clerkship. METHOD: A cohort study reviewed the pediatrics clerkship performances of 400 Eastern Virginia Medical School students in the graduating classes of 1996 through 1999. The first two classes completed a traditional preclinical curriculum with limited clinical experience. The final two classes participated in a mentorship program that provided 18 months of early clinical experience, consisting of one-on-one half-day sessions every other week with a generalist community faculty. The authors compared the clinical clerkship performances of the groups using clinical skills (CS) scores, history and physical examination (H&P) scores, and scores on the NBME pediatrics shelf examination. They also looked at the mean MCAT and USMLE scores for each group. The authors also looked at the scores within each class, comparing students who completed one of the first two pediatrics clerkship rotations with their classmates who completed clerkships later in the academic year. RESULTS: The students' NBME scores rose significantly (p < .05, r2 = 0.95) over the four-year study, paralleling a significant rise in MCAT scores (p < .03, r2 = 0.73). The CS and H&P scores did not rise. Students who had the traditional preclinical curriculum and who completed their clerkships early in the year had significantly lower CS and H&P scores than did their classmates. In contrast, the scores of students who had the early clinical experiences did not differ significantly according to the timing of their rotation. CONCLUSION: Students who had participated in a mentorship program that provided early clinical experience demonstrated significantly improved clinical skills in the pediatrics clerkship early in the academic year.     

The impact of preclinical preceptorships on learning the fundamentals of clinical medicine and physical diagnosis skills.

PURPOSE: To learn whether preclinical primary care preceptorships resulted in demonstrable clinical performance benefits to medical students. METHOD: This was a retrospective cohort study of 267 medical students who elected and 310 students who did not elect to take a four-week primary care preceptorship following the first year of training at the University of Texas Medical School at Houston in 2001-2003. Outcome variables were the students' performances on a written examination testing their integration of basic science and fundamentals of clinical medicine and performances on a comprehensive objective structured clinical examination (OSCE). After adjusting for factors that might have explained differences in the students' performances, logistic regression models were used to assess the association of the outcome variables with participation in a preceptorship. RESULTS: Students who participated in any one of the preceptorships performed better on the OSCE and on the fundamentals of clinical medicine examination than students who did not participate (p < .01). Students who participated in the family medicine or pediatric preceptorship scored higher on an OSCE (odds ratio [OR], 1.67; 95% confidence interval [CI], 1.01-2.78 and OR, 2.26; 95% CI, 1.18-4.36, respectively) than those students who did not participate in a preceptorship. Students who participated in the internal medicine preceptorship scored higher on the fundamentals of clinical medicine examination (OR, 3.18; 95% CI, 1.92-5.23). CONCLUSIONS: Preliminary evidence indicates that a short preclinical primary care preceptorship can help medical students to consolidate and integrate the fundamental cognitive and clinical skills they will apply during the clinical years of medical training.     

The impact of a brief interclerkship about substance abuse on medical students' skills.

PURPOSE: To examine the immediate and delayed impact of an intensive one- or two-day interclerkship on substance abuse (SA) for third-year medical students. The program is a response to the problem of inadequacy of substance abuse education in the standard curriculum. METHOD: Each year since 1997-98 all third-year students at the University of Massachusetts Medical School have participated in a one- or two-day SA interclerkship to enhance their knowledge and competence with SA assessment and brief intervention. Students' knowledge, attitudes, and confidence were assessed immediately before and after the interclerkship. In addition, during 1998-99, each student's clinical skills in SA assessment and intervention were evaluated at the completion of the student's six-week psychiatry clerkship using objective standardized clinical examinations (OSCEs) with two simulated patients, one with and one without active SA issues. Students who took the psychiatry clerkship in the first half of the year had not yet participated in the interclerkship. Students' pooled performances before and after the interclerkship were compared. RESULTS: Students' attitudes toward and knowledge about SA disorders and their confidence about SA assessment and intervention all showed significant positive changes immediately after the interclerkship. The OSCE performance data demonstrated a significant sustained improvement in clinical skills in SA assessment and intervention as measured up to six months following the interclerkship. CONCLUSION: These data suggest that brief intensive training in SA during the clinical years of medical school can have a positive and lasting impact on students' clinical performances.     

Developing students' cognitive skills in a problem-based surgery clerkship.

In 1989-90, 57 students in a new program for the third-year surgery clerkship at the University of Kentucky College of Medicine participated either in a control group (22 students) receiving a traditional method of instruction (Socratic instruction, SI) or in an experimental group (35 students) taking part in problem-based learning (PBL) sessions. The two groups' performances on six evaluative instruments designed to test either their factual knowledge or their knowledge application (i.e., clinical problem-solving skills) were compared. The measures of factual knowledge were associated with higher scores for the SI group on two quizzes; scores were not significantly different on another quiz and on a cumulative final examination. The measures of knowledge application (administered at the end of the clerkship) were associated with higher scores for the PBL group: scores were significantly higher on a modified essay examination and approached significance on a standardized-patient examination. The authors conclude that their results (1) have important similarities to those of previous research suggesting that a PBL format is essentially equivalent to a traditional curricular format in improving students' factual knowledge and (2) support the hypothesis that PBL is superior in improving clinical problem-solving skills.     

Results of the second Clinical Skills Assessment Examination of the ECFMG

To test the hypothesis that the clinical skills of many foreign medical graduates entering residencies in the United States are deficient, the Educational Commission for Foreign Medical Graduates (ECFMG) administered a second, more extensive pilot examination called the Clinical Skills Assessment Examination. The test was constructed after identifying those clinical skills considered to be suitable for testing in a large-scale examination conducted at multiple locations. The test was given to 635 foreign medical graduates (including 53 U.S. citizens) who had been certified by the ECFMG and to a reference sample of 123 graduates of U.S. medical schools. All of the graduates took the test prior to their first postgraduate year of training. The results did not differ significantly from those of a previous test. The clinical skills of 28% of the graduates of foreign medical schools who took the examination were found to be inadequate when compared with those of graduates of U.S. schools. The conclusion reached was that a clinical skills examination is feasible for inclusion in the ECFMG certification process.     

The effect on grades of the timing and site of third-year internal medicine clerkships

Two important factors affecting the performances of third-year medical students on their basic internal medicine clerkships were investigated: (1) the effect on their grades of when in the academic year they took the internal medicine clerkship, and (2) the effect on their grades of the site of the clerkship. During the academic years 1983-84, 1984-85, and 1985-86, the Department of Internal Medicine of the University of Illinois College of Medicine at Chicago conducted 12-week junior clerkships at six hospital sites. The study analyzed the 535 students' subjective grades, clerkship examination scores, final grades, and National Board of Medical Examiners Part I (NBME-I) scores. Although students' performances as characterized by subjective evaluations did not improve as the academic year progressed, mean scores on clerkship examinations improved steadily during the same period. The site of the clerkship, whether a community-affiliated hospital or a medical center hospital, affected neither subjective nor objective grades.     

Enhancing clinical skills education: University of Virginia School of Medicine's Clerkship Clinical Skills Workshop Program.

In 1993, the University of Virginia School of Medicine began a clinical skills workshop program in an effort to improve the preparation of all clerkship students to participate in clinical care. This program involved the teaching of selected basic clinical skills by interested faculty to small groups of third-year medical students. Over the past 14 years, the number of workshops has increased from 11 to 31, and they now involve clerkship faculty from family medicine, internal medicine, and pediatrics. Workshops include a variety of common skills from the communication, physical examination, and clinical test and procedure domains such as pediatric phone triage, shoulder examination, ECG interpretation, and suturing. Workshop sessions allow students to practice skills on each other, with standardized patients, or with models, with the goal of improving competence and confidence in the performance of basic clinical skills. Students receive direct feedback from faculty on their skill performance. The style and content of these workshops are guided by an explicit set of educational criteria.A formal evaluation process ensures that faculty receive regular feedback from student evaluation comments so that adherence to workshop criteria is continuously reinforced. Student evaluations confirm that these workshops meet their skill-learning needs. Preliminary outcome measures suggest that workshop teaching can be linked to student assessment data and may improve students' skill performance. This program represents a work-in-progress toward the goal of providing a more comprehensive and developmental clinical skills curriculum in the school of medicine.     

Direct observation of students during clerkship rotations: a multiyear descriptive study.

PURPOSE: To determine how often students report that they are observed while performing physical examinations and taking histories during clerkship rotations. METHOD: From 1999-2001, 397 students at the University of Virginia School of Medicine were asked at the end of their third year to report the number of times they had been observed by a resident or faculty member while taking histories and performing physical examinations on six rotations. RESULTS: Three hundred and forty-five students (87%) returned the survey instrument; of these, 322 (81%) returned instruments with complete information. On average, the majority reported that they had never been observed by a faculty member while taking a history (51%), performing a focused physical examination (54%), or a complete physical examination (81%). The majority (60%) reported that they had never been observed by a resident while performing a complete physical examination. Faculty observations occurred most frequently during the four-week family medicine rotation and least frequently during the 12-week surgery rotation. The length of the clerkship rotation was inversely related to the number of reported observations, chi(2) (5, n = 295) = 127.85, p <.000. CONCLUSIONS: Although alternative assessments of clinical skills are becoming more common in medical education, faculty ratings based on direct observation are still prominent. The data in this study reflect that these observations may actually be occurring quite infrequently, if at all. Decreasing the evaluative weight of faculty and resident ratings during the clerkship rotation may be necessary. Otherwise, efforts should be made to increase the validity of these ratings.