Implementation of the mini-CEX to evaluate medical students' clinical skills.

OBJECTIVE: The majority of medicine clerkships use faculty and resident summative ratings to assess medical students' clinical skills. Still, many medical students complete training without ever being observed performing a clinical skill. The mini-CEX is method of clinical skills assessment developed by the American Board of Internal Medicine for graduate medical education. The brief, focused encounters are feasible and produce scores with adequate reproducibility if enough observations are made.(1) The mini-CEX has been used in the medicine core clerkship, being performed once to augment feedback by faculty evaluators in the inpatient setting.(2) However, additional study is needed to address at least two feasibility issues if the mini-CEX is to be used as a measurement tool: (1) multiple settings (inpatient and outpatient) and (2) resident-completed evaluations. Our objective was to determine the feasibility of having students receive multiple mini-CEX's in both the inpatient and outpatient settings from resident and faculty evaluators. DESCRIPTION: We introduced the mini-CEX into our nine-week medicine clerkship (six weeks inpatient and three weeks outpatient) in July 2001. The clerkship uses four inpatient clinical sites and 16 outpatient practices. Inpatient faculty rotate on two-week blocks and residents on four-week blocks. At our clerkship orientation, each student (n = 39) received a booklet of ten adapted mini-CEX forms. In the mini-CEX, students are observed conducting a focused history and physical examination and then receive immediate feedback. Students are rated in seven competencies (interviewing, physical examination, professionalism, clinical judgment, counseling, organization, and overall clinical competence) using a nine-point rating scale (1 = unsatisfactory and 9 = superior). Our students were instructed to collect nine evaluations: three from faculty (one every two weeks), three from residents (one every two weeks), and three from their out-patient attendings (one per week). Students and evaluators were asked to rate their satisfaction with the exercise using a nine-point scale (1 = low and 9 = high). Students were asked to turn in their booklets the day of the exam. Prior to implementation, we reviewed the mini-CEX forms and rationale for use with residents and inpatient faculty. Similar information was mailed to outpatient faculty preceptors. DISCUSSION: Booklets were received from 32 students. The mean number of evaluations completed per student was 7.3 (range 2-9), for a total of 232 evaluations. Faculty completed 58% of the evaluations; 68% were from the inpatient setting. The observation and feedback took an average of 21 minutes and 8 minutes, respectively. Satisfaction with the exercise was rated by faculty/residents as 7.2 and by students as 6.8. We believe these findings support the feasibility of collecting multiple mini-CEX assessments from both inpatient and outpatient sites using faculty and resident evaluators. The feasibility of collecting multiple assessments is important if the mini-CEX is to be a reproducible assessment of clinical skills. Having established feasibility, we plan to look at the reproducibility and validity of mini-CEX scores in order to determine if it can be used as a formal means of clinical skills assessment. We also plan to evaluate the impact on the quality and specificity of end-of-clerkship summative ratings.     

Global ratings of videotaped performance versus global ratings of actions recorded on checklists: a criterion for performance assessment with standardized patients.

PURPOSE: To test whether global ratings of checklists are a viable alternative to global ratings of actual clinical performance for use as a criterion for standardized-patient (SP) assessment. METHOD: Five faculty physicians independently observed and rated videotaped performances of 44 medical students on the seven SP cases that comprise the fourth-year assessment administered at The Morchand Center of Mount Sinai School of Medicine to students in the eight member schools in the New York City Consortium. A year later, the same panel of raters reviewed and rated checklists for the same 44 students on five of the same SP cases. RESULTS: The mean global ratings of clinical competence were higher with videotapes than checklists, whereas the mean global ratings of interpersonal and communication skills were lower with videotapes. The correlations for global ratings of clinical competence showed only moderate agreement between the videotape and checklist ratings; and for interpersonal and communication skills, the correlations were somewhat weaker. CONCLUSION: The results raise serious questions about the viability of global ratings of checklists as an alternative to ratings of observed clinical performance as a criterion for SP assessment.     

A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator.

PURPOSE: To determine the correlation between global ratings and criterion-based checklist scores, and inter-rater reliability of global ratings and criterion-based checklist scores, in a performance assessment using an anesthesia simulator. METHOD: All final-year medical students at the University of Toronto were invited to work through a 15-minute faculty-facilitated scenario using an anesthesia simulator. Students' performances were videotaped and analyzed by two faculty using a 25-point criterion-based checklist and a five-point global rating of competency (1 = clear failure, 5 = superior performance). Correlations between global ratings and checklist scores, as well as specific performance competencies (knowledge, technical skills, and judgment), were determined. Checklist and global scores were converted to percentages; means of the two marks were compared. Mean reliability of a single rater for both checklist and global ratings was determined. RESULTS: The correlation between checklist and global ratings was.74. Mean ratings of both checklist and global scores were low (58.67, SD = 14.96, and 57.08, SD = 24.27, respectively); these differences were not statistically significant. For a single rater, the mean reliability score across rater pairs for checklist scores was.77 (range.58-.93). Mean reliability score across rater pairs for global ratings was.62 (.40-.77). Global ratings correlated more highly with technical skills and judgment (r =.51 and r =.53, respectively) than with knowledge. (r =.24) CONCLUSION: Inter-rater reliability was higher for checklist scores than for global ratings; however, global ratings demonstrated acceptable inter-rater reliability and may be useful for competency assessment in performance assessments using simulators.     

Measuring outcomes of undergraduate medical education: residency directors' ratings of first-year residents.

PURPOSE: There is a growing recognition of the need to show the relationship between undergraduate medical education (UME) and achievements during residency. This study provides reliability and validity evidence for a residency rating scale as well as a method for gathering comparison information about first-year residents. METHOD: A 25-item rating scale measuring important areas of physician functioning was mailed to residency directors of 485 graduates of the 1998-2000 classes of the University of Kansas School of Medicine. The same rating scale was sent to residency directors for a comparison sample of 251 graduates of other U.S. medical schools who were residents at the University of Kansas Medical Center. Each item on the rating scale was rated on a five-point Likert scale. Principal-components analysis, correlational analyses, internal consistency reliability analysis, and mean comparisons were used to provide evidence of reliability and validity. RESULTS: A total of 382 (82%) usable rating scales were returned. A principal-components analysis extracted five factors that accounted for 86% of the variance. The final factors were (1) interpersonal communication, (2) clinical skills, (3) population-based health care, (4) record-keeping skills, and (5) critical appraisal skills. The internal consistency of the entire scale was.98, with coefficients for the five factors ranging from.92 to.97. The correlations between the five factors and measures of undergraduate performance ranged from.21 to.49. Group analyses revealed that residents with high GPAs and USMLE Step 1 and Step 2 scores tended to be rated higher than those with lower scores. CONCLUSIONS: The rating scale demonstrated adequate reliability and validity and showed that residency directors' ratings are a useful outcome measure for UME performance.     

Using standardized patients to assess the geriatrics medicine skills of medical students, internal medicine residents, and geriatrics medicine fellows.

PURPOSE: Medical schools and residency programs are placing additional emphasis on including clinical geriatrics competencies within their curricula. An eight-station, Geriatric Medicine Standardized Patient Examination (GSPX) was studied as a method to assess bedside geriatrics clinical skills over the continuum of medical education from medical school through residency and fellowship training. METHOD: The GSPX was administered to 39 medical students, 49 internal medicine residents, and 11 geriatrics medicine fellows in 2001-02. Reliability of standardized patient (SP) checklists and rating scales used to assess examinees' performance was measured by Cronbach's alpha. Validity was measured by surveying the examinees' assessment of fairness, individual case length, difficulty, and believability, and by faculty standard setting for each level of trainee. RESULTS: Reliability was high (alpha =.89). All levels of examinees found the SPs to be believable, station lengths to be adequate, and rated the GSPX as a fair assessment. Students rated the cases as more difficult. Previous experience with similar real patients increased significantly with level of training (Pearson's r =.48, p <.0001). Faculty set passing scores that increased from students to residents to fellows. However, GSPX scores decreased with level of training (r = -.25, p =.01). CONCLUSION: The GSPX is a reliable measure of geriatrics medicine skills with adequate face validity for examinees at all levels. However, GSPX scores did not increase with level of training, suggesting that a single form of the examination cannot be used across the continuum of training. Potential modifications to the GSPX that might provide more discrimination between levels of training are currently being explored.     

Structured assessment of minor surgical skills (SAMSS) for family medicine residents.

PURPOSE: To develop a valid and reliable examination to assess the technical proficiency of family medicine residents' performance of minor surgical office procedures. METHOD: A multi-station OSCE-style examination using bench-model simulations of minor surgical procedures was developed. Participants were a randomly selected group of 33 family medicine residents (PGY-1 = 16, PGY-2 = 17) and 14 senior surgical residents who functioned as a validation group. Examiners were qualified surgeons and family physicians who used both checklists and global rating scales to score the participants' performances. RESULTS: When family medicine residents were evaluated by family physicians, interstation reliabilities were .29 for checklists and .42 for global ratings. When family medicine residents were evaluated by surgeons, the reliabilities were .53 for checklists and .75 for global ratings. Interrater reliability, measured as a correlation for total examination scores, was .97. Mean scores on the examination were 60%, 64%, and 87% for PGY-1 family medicine, PGY-2 family medicine, and surgery residents, respectively. The difference in scores between family medicine and surgery residents was significant (p < .001), providing evidence of construct validity. CONCLUSION: A new examination developed for assessing family medicine residents' skills with minor surgical office procedures is reliable and has evidence for construct validity. The examination has low reliability when family physicians serve as examiners, but moderate reliability when surgeons are the evaluators.     

Effectiveness of a brief workshop designed to improve teaching performance at the University of Alberta.

PURPOSE: To determine whether a two-day teaching enhancement workshop at the University of Alberta improved participants' teaching performance as rated by students. METHOD: Workshop participants (academic staff or residents) were asked to assess the value of the workshop. In addition, students were asked to rate instructors' teaching abilities before and after the instructors participated in the workshops, by completing a five-statement questionnaire routinely used to assess instruction at the University of Alberta. For control purposes, ratings were also obtained for a group of instructors who had not taken the workshop, over a similar time period. The authors used data from 1993-2002. RESULTS: The participants uniformly regarded the workshops as helpful. Both faculty and residents regarded the short teaching exercise as the most important component of the program. Of the instructional sections, the presentations on objectives and on structure (set, body, closure) were rated most highly by both groups. The students' mean ratings for the instructors after the workshop were significantly increased, while ratings for those who had not taken the workshop were unchanged CONCLUSION: Short teaching-enhancement workshops are regarded by the participants as helpful in improving their instructional skills. This view is supported by a significant increase in students' ratings of the instructors after they had taken the workshop.     

A national collaboration to disseminate skills for outpatient teaching in internal medicine: program description and preliminary evaluation.

The shift of clinical care and teaching to outpatient settings has challenged ambulatory and community-based teachers. To address this challenge, U.S. internal medicine organizations devised "Faculty Development for General Internal Medicine: Generalist Faculty Teaching in Ambulatory Settings," a national program to train leaders to create local faculty development projects. In 1999, teams from all 386 internal medicine training institutions were invited to apply. Participation required an acceptable plan for a local project and inclusion of an institutional leader, residency or clerkship director, and a community-based faculty member on the project team. Team members attended one of three national training conferences held in 1999 and 2000 that included plenary sessions, workshops, and team meetings. Participants were invited to a wrap-up conference to present their accomplishments. One hundred ten teams from 57 university and 53 non-university hospitals attended the training conferences; 412 (93%) participants returned conference evaluations. All sessions were rated highly. Participants preferred workshops and team meetings to plenary sessions. Two hundred thirty-five (57%) would have recommended the training conference to colleagues as an outstanding experience; 148 (36%) as a good experience; and 25 (6%) as a satisfactory experience. Forty-nine teams (122 participants) returned for the wrap up conference where 35 teams presented their local faculty development projects. Cost per team trained was US$11,818. This program demonstrated a national desire for training in teaching skills, reached a broad audience of ambulatory-based clinical teachers, provided highly rated faculty development conferences in teaching skills, and facilitated development of a variety of local projects at modest expense. Partnerships were forged between academic leaders and community-based teachers.     

Underuse of coronary revascularization procedures in patients considered appropriate candidates for revascularization

BACKGROUND: Ratings by an expert panel of the appropriateness of treatments may offer better guidance for clinical practice than the variable decisions of individual clinicians, yet there have been no prospective studies of clinical outcomes. We compared the clinical outcomes of patients treated medically after angiography with those of patients who underwent revascularization, within groups defined by ratings of the degree of appropriateness of revascularization in varying clinical circumstances. METHODS: This was a prospective study of consecutive patients undergoing coronary angiography at three London hospitals. Before patients were recruited, a nine-member expert panel rated the appropriateness of percutaneous transluminal coronary angioplasty (PTCA) and coronary-artery bypass grafting (CABG) on a nine-point scale (with 1 denoting highly inappropriate and 9 denoting highly appropriate) for specific clinical indications. These ratings were then applied to a population of patients with coronary artery disease. However, the patients were treated without regard to the ratings. A total of 2552 patients were followed for a median of 30 months after angiography. RESULTS: Of 908 patients with indications for which PTCA was rated appropriate (score, 7 to 9), 34 percent were treated medically; these patients were more likely to have angina at follow-up than those who underwent PTCA (odds ratio, 1.97; 95 percent confidence interval, 1.29 to 3.00). Of 1353 patients with indications for which CABG was considered appropriate, 26 percent were treated medically; they were more likely than those who underwent CABG to die or have a nonfatal myocardial infarction--the composite primary outcome (hazard ratio, 4.08; 95 percent confidence interval, 2.82 to 5.93)--and to have angina (odds ratio, 3.03; 95 percent confidence interval, 2.08 to 4.42). Furthermore, there was a graded relation between rating and outcome over the entire scale of appropriateness (P for linear trend=0.002). CONCLUSIONS: On the basis of the ratings of the expert panel, we identified substantial underuse of coronary revascularization among patients who were considered appropriate candidates for these procedures. Underuse was associated with adverse clinical outcomes.     

Relationship of tutors' group-dynamics skills to their performance ratings in problem-based learning.

PURPOSE: A tutor's performance is often investigated as a stable teacher characteristic, isolated from the context in which the tutor functions. This study investigated the influence of a tutor's group-dynamics skills in differently functioning tutorial groups on performance ratings. METHOD: Students' ratings of tutors' performances collected from 75 tutorial groups at one school over four different academic years were compared with the independent variables, groups' cohesion scores and sponging scores (the degree to which some students let others do the work), and the intervening variable, tutors' group-dynamics skills. RESULTS: Tutors with strong group-dynamics skills were assigned mean tutor's-performance scores of 7.4 (SD = 0.8) and 8.2 (SD = 0.5), respectively, by groups scoring low and high on cohesion. Tutors with low group-dynamics skills were rated 7.0 (SD = 1.1) and 7.4 (SD = 1.1) by the same groups. The same pattern held for sponging scores. Tutors who had strong group-dynamics skills received mean tutor's-performance scores of 8.1 (SD = 0.6) and 7.4 (SD = 0.8), respectively, from groups with low and high sponging scores. Tutors with weak group-dynamics skills were scored 7.6 (SD = 1.0) and 6.9 (SD = 1.1) by the same groups. CONCLUSIONS: Tutors possessing group-dynamics skills were rated higher by students than were tutors who lacked these skills, irrespective of the quality of a tutorial group's performance. A tutor who evaluates tutorial-group function on a regular basis and makes appointments with students based on these evaluations is seen as performing better than a tutor who does not.     

What do medicine clerkship preceptors do best?

PURPOSE: Students' ratings of preceptors are widely used in medical education for feedback and evaluation purposes. The present study investigated students' ratings of the clinical teaching skills of inpatient attending physicians, inpatient residents, and outpatient attending physicians to assess differences among types of preceptors and relative strengths and weaknesses. METHOD: A total of 268 students from three academic years (1997-2000) at one medical school rated preceptors on an end-of-clerkship evaluation, for a total of 1,680 ratings. When the ratings were aggregated by preceptors' names and types, there were 691 mean ratings of preceptors. Relative strengths and weaknesses were identified. Differences in mean ratings by preceptor type (inpatient attending physician, inpatient resident, and outpatient attending physician) were evaluated, and strengths and weaknesses were identified by rank ordering the items' means. RESULTS: Students tended to rate outpatient attending physicians higher than inpatient attending physicians or residents. Areas where ratings suggested relative strengths included showing an interest in teaching, respecting students' opinions, and being available to students. Areas of relative weakness included increasing physical examination and interviewing skills. CONCLUSIONS: Students' ratings are useful for identifying strengths and weakness for groups of preceptors and, as such, are important sources of information for setting priorities for faculty development efforts.     

An investigation of construct validity of humanistic clinical skills on a medical licensure examination

Objective

To test construct validity of humanistic clinical skills measured by a medical licensure performance examination using multitrait-multimethod confirmatory factor analysis.

Methods

Two hundred and twenty-seven third- and fourth-year undergraduate osteopathic medical students in 2007-2008 were randomly sampled. The Global Patient Assessment Tool (GPAT), designed to assess professionalism, interpersonal relationship, and doctor-patient communication was tested under two measurement methods: standard examination ratings and peer performance ratings of the same examinee. Two concurrent validity factors (data gathering and written patient notes) were included.

Results

Convergent validity was supported under the two methods of scoring. Moderate to strong correlations among trait factors indicated weak discriminate validity. Method effects were indicated. The relationship between GPAT and two concurrent validity factors indicate measures of relatively different constructs.

Conclusion

Evidence of construct validity for the GPAT indicate scores should be interpreted as measuring a humanistic clinical skills construct consisting of homogeneous measures of professionalism, interpersonal relationship, and physician-patient communication. Findings are consistent with the interpretation and use of the GPAT as an important part of medical licensure examinations.

Practical implications

The implications to medical licensure standardized-patient examinations are discussed in terms of the GPAT as a valid measure of medical students’ humanistic clinical skills.     

Outcome measurement in postgraduate year one of graduates from a medical school with a pass/fail grading system.

PURPOSE: To measure the performances of first-year residents who had graduated from a medical school with a pass/fail grading system and to compare the preparedness of these graduates with that of their peers. METHOD: All 169 graduates of Stanford University School of Medicine's classes of 1993 and 1994 were included in this study. First-year program directors rated the performance of each Stanford graduate in 11 areas, compared the graduate's clinical preparedness with that of his or her peer group, and rated the accuracy of the dean's letter in presenting the graduate's capabilities. RESULTS: Responses were obtained for 144 of the 169 graduates (85%). The program directors rated the overall clinical competencies of most of the graduates as "superior" (76%) or "good" (22%); they rated very few as "unsatisfactory" (2%). When the Stanford graduates were compared with their peers, their clinical preparedness was judged "outstanding" (33%), "excellent" (44%), and "good" (20%); very few were judged "poor" (3%). Stratification of programs by either hospital or medical specialty did not reveal significant differences in overall clinical competence. Ninety-one percent of the responses reported that the dean's letters had accurately presented the capabilities of the graduates. CONCLUSION: Graduates from a medical school with a two-interval, pass/fail system successfully matched with strong, highly-sought-after postgraduate training programs, performed in a satisfactory to superior manner, and compared favorably with their peer group.     

The predictive validity of selection for entry into postgraduate training in general practice: evidence from three longitudinal studies

Background

The selection methodology for UK general practice is designed to accommodate several thousand applicants per year and targets six core attributes identified in a multi-method job-analysis study

Aim

To evaluate the predictive validity of selection methods for entry into postgraduate training, comprising a clinical problem-solving test, a situational judgement test, and a selection centre.

Design and setting

A three-part longitudinal predictive validity study of selection into training for UK general practice.

Method

In sample 1, participants were junior doctors applying for training in general practice (n = 6824). In sample 2, participants were GP registrars 1 year into training (n = 196). In sample 3, participants were GP registrars sitting the licensing examination after 3 years, at the end of training (n = 2292). The outcome measures include: assessor ratings of performance in a selection centre comprising job simulation exercises (sample 1); supervisor ratings of trainee job performance 1 year into training (sample 2); and licensing examination results, including an applied knowledge examination and a 12-station clinical skills objective structured clinical examination (OSCE; sample 3).

Results

Performance ratings at selection predicted subsequent supervisor ratings of job performance 1 year later. Selection results also significantly predicted performance on both the clinical skills OSCE and applied knowledge examination for licensing at the end of training.

Conclusion

In combination, these longitudinal findings provide good evidence of the predictive validity of the selection methods, and are the first reported for entry into postgraduate training. Results show that the best predictor of work performance and training outcomes is a combination of a clinical problem-solving test, a situational judgement test, and a selection centre. Implications for selection methods for all postgraduate specialties are considered.     

Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine.

By the end of 1990-91, the Southern Illinois University School of Medicine had had six years of experience with comprehensive, performance-based examinations of senior medical students' levels of clinical competence; this report assesses the psychometric aspects of the six examinations given during that period. The examinations were aimed at determining the students' readiness for postgraduate training. Compared with other clinical performance-based assessments that use standardized patients (SPs), these examinations had two important and unique features: (1) the examinations assessed a comprehensive range of clinical skills and reasoning; and (2) they approximated the challenges of real clinical practice wherein a practitioner's skills need to be orchestrated and prioritized in order to meet the challenges of the case encountered. Each year, the performance-based assessment given was an intensive clinical examination requiring each student to work up 13 to 18 SP problems over a three-day period. To administer an examination to an entire class of students took three weeks. Because all students after the first year of administration (1986) were required to pass these examinations, the fairness of test design and scoring and the setting of performance standards for the examinations became important issues for the faculty. The results, accumulated over six years and based on a total of 6,804 student-patient encounters involving 405 students, indicate that this kind of clinical performance-based examination can discriminate a wide range of students' clinical performances. The results provide evidence for the examinations' test security, content validity, construct validity, and reliability.     

The use of questionnaires to assess achievement of course goals in medical students' longitudinal community-based clinical experiences.

PURPOSE: To examine the usefulness of questionnaires for assessing achievement of course goals in medical students' longitudinal community-based clinical experiences. METHOD: In 1997, the authors surveyed 114 first-year students and their preceptors in a longitudinal community-based program at The Joan and Sanford I. Weill Medical College of Cornell University. The questionnaire used a Likert scale to assess students' and preceptors' pre-course expectations for achieving specific course goals and their post-course perceptions of having met those goals. The students also rated global learning and satisfaction during each office preceptor session, and faculty assessed the students' physical examination skills at the end of the course. RESULTS: For all goals assessed, the preceptors scored their students' achievement of course goals significantly higher than did the students themselves (p < .01). The students invariably scored their post-course perceptions of having achieved the goals lower than they did their pre-course expectations (p < .001). Before the course, the preceptors were confident in their ability to teach the curricular material; this confidence remained after the course. Global learning and satisfaction scores were high and all students performed satisfactorily in the demonstration examination. CONCLUSIONS: Students and preceptors may not agree on students' achievement of course goals. Furthermore, despite students' high ratings of global satisfaction and learning, and despite their satisfactory performance of physical examination skills, their ratings of post-course performance may be affected by pre-course expectations. The authors suggest that questionnaires assessing students' and preceptors' perceptions of students' achievement of specific goals should be independently verified before making decisions to modify objectives and activities in these kinds of courses.     

Self-reported competency ratings of graduates of a problem-based medical curriculum.

PURPOSE: To study the self-reports of professional competencies by graduates of a problem-based medical curriculum. METHOD: All graduates from a medical school and a faculty of health sciences with a problem-based curriculum were sent a questionnaire asking them to compare their own performances in 19 domains with those of colleagues trained at schools with conventional curricula. RESULTS: Overall, alumni of the medical school rated themselves as better than colleagues who were trained at schools with conventional curricula for cooperation skills, problem-solving skills, skills relevant to running meetings, and the ability to work independently. There was no difference for possession of general academic knowledge and writing reports or articles. The self-reported ratings of better competencies were maintained after correcting the data for self-overestimation. CONCLUSION: The problem-based medical curriculum appears to contribute to the development of professional competencies. Further study is needed, however, to control for the effect of selection bias and respondents' emotional commitment to their alma mater.     

Direct observation of students during clerkship rotations: a multiyear descriptive study.

PURPOSE: To determine how often students report that they are observed while performing physical examinations and taking histories during clerkship rotations. METHOD: From 1999-2001, 397 students at the University of Virginia School of Medicine were asked at the end of their third year to report the number of times they had been observed by a resident or faculty member while taking histories and performing physical examinations on six rotations. RESULTS: Three hundred and forty-five students (87%) returned the survey instrument; of these, 322 (81%) returned instruments with complete information. On average, the majority reported that they had never been observed by a faculty member while taking a history (51%), performing a focused physical examination (54%), or a complete physical examination (81%). The majority (60%) reported that they had never been observed by a resident while performing a complete physical examination. Faculty observations occurred most frequently during the four-week family medicine rotation and least frequently during the 12-week surgery rotation. The length of the clerkship rotation was inversely related to the number of reported observations, chi(2) (5, n = 295) = 127.85, p <.000. CONCLUSIONS: Although alternative assessments of clinical skills are becoming more common in medical education, faculty ratings based on direct observation are still prominent. The data in this study reflect that these observations may actually be occurring quite infrequently, if at all. Decreasing the evaluative weight of faculty and resident ratings during the clerkship rotation may be necessary. Otherwise, efforts should be made to increase the validity of these ratings.