Validity and generalizability of global ratings in an objective structured clinical examination

The performance of foreign medical graduates on multistation standardized patient-based tests and used to determine the validity and generalizability of global ratings of their clinical competence made by expert examiners. Data were derived from the entrance examinations of the 1989 and 1990 applicants to the Ontario Pre-Internship Program and the exit examination of 24 participants from the 1989 cohort. For each candidate, the examiners completed a detailed checklist and two five-point global ratings dealing with the candidate's approach to the patients' problem and attitude toward the patient. Generalizability coefficients for both ratings were satisfactory and stable across cohorts. Construct validity of the global ratings was demonstrated by comparing entry and exit ratings and by evidence of significant and positive correlations between the global ratings and total test scores. Tentative evidence of criterion validity of the global ratings was demonstrated. These findings suggest that global ratings by expert examiners can be used as an effective form of assessment in multistation standardized patient examinations.     

Outcome measurement in postgraduate year one of graduates from a medical school with a pass/fail grading system.

PURPOSE: To measure the performances of first-year residents who had graduated from a medical school with a pass/fail grading system and to compare the preparedness of these graduates with that of their peers. METHOD: All 169 graduates of Stanford University School of Medicine's classes of 1993 and 1994 were included in this study. First-year program directors rated the performance of each Stanford graduate in 11 areas, compared the graduate's clinical preparedness with that of his or her peer group, and rated the accuracy of the dean's letter in presenting the graduate's capabilities. RESULTS: Responses were obtained for 144 of the 169 graduates (85%). The program directors rated the overall clinical competencies of most of the graduates as "superior" (76%) or "good" (22%); they rated very few as "unsatisfactory" (2%). When the Stanford graduates were compared with their peers, their clinical preparedness was judged "outstanding" (33%), "excellent" (44%), and "good" (20%); very few were judged "poor" (3%). Stratification of programs by either hospital or medical specialty did not reveal significant differences in overall clinical competence. Ninety-one percent of the responses reported that the dean's letters had accurately presented the capabilities of the graduates. CONCLUSION: Graduates from a medical school with a two-interval, pass/fail system successfully matched with strong, highly-sought-after postgraduate training programs, performed in a satisfactory to superior manner, and compared favorably with their peer group.     

Results of the second Clinical Skills Assessment Examination of the ECFMG

To test the hypothesis that the clinical skills of many foreign medical graduates entering residencies in the United States are deficient, the Educational Commission for Foreign Medical Graduates (ECFMG) administered a second, more extensive pilot examination called the Clinical Skills Assessment Examination. The test was constructed after identifying those clinical skills considered to be suitable for testing in a large-scale examination conducted at multiple locations. The test was given to 635 foreign medical graduates (including 53 U.S. citizens) who had been certified by the ECFMG and to a reference sample of 123 graduates of U.S. medical schools. All of the graduates took the test prior to their first postgraduate year of training. The results did not differ significantly from those of a previous test. The clinical skills of 28% of the graduates of foreign medical schools who took the examination were found to be inadequate when compared with those of graduates of U.S. schools. The conclusion reached was that a clinical skills examination is feasible for inclusion in the ECFMG certification process.     

Residency directors' assessments of which selection criteria best predict the performances of foreign-born foreign medical graduates during internal medicine residencies.

In the summer of 1990, 102 directors of internal medicine residency programs from nine areas of the country with the largest numbers of foreign-born foreign medical graduates (FFMGs) were surveyed by mail to determine what criteria used to select FFMGs for residency positions best predicted performance. The directors felt that the most important predictors were performance on the Foreign Medical Graduate Examination in the Medical Sciences or the National Board of Medical Examiners examinations; performance during the interview; and postgraduate clinical experience in the United States. Recent graduation from medical school was felt to be a better predictor than postgraduate clinical experience in a foreign country. Seventy-three percent of the directors found letters of recommendation from a foreign country to be useless. The author suggests the results of this study may be useful to residency programs in evaluating FFMG applicants and to FFMGs in assessing their own credentials.     

Performances of U.S. osteopathic and Canadian medical school graduates on the American Board of Internal Medicine Certifying Examinations, 1984-1988

The performances of Canadian medical school graduates and U.S. osteopathic medical school graduates who first took the American Board of Internal Medicine Certifying Examination between 1984 and 1988 were compared with the performances, during the same period, of U.S. and foreign medical school graduates. Approximately 100 Canadian graduates took the examination each year; by 1988 the number of osteopathic graduates taking it was 102, double the number participating in 1984. Nearly all the Canadian graduates attended university or university-affiliated residencies, whereas half of the osteopathic graduates attended non-university-affiliated programs. For their overall clinical competence and for the eight components of clinical competence, the Canadian graduates were rated highest, followed by the U.S., osteopathic, and foreign graduates. The Canadians' average examination scores were also highest. The authors discuss the relatively low level of performance of the osteopathic graduates, but conclude that these graduates appear to be an untapped source of talented physicians for internal medicine residencies. The limitations of studying self-selected groups of candidates are also discussed.     

Medical students trained abroad and medical manpower. Recent trends and predictions

Physicians trained in foreign medical schools, including U.S. citizens, are once again playing a larger part in the provision of medical are in this country. After a decrease in the numbers of such physicians from 1977 through 1980, the number of foreign nationals entering the National Resident Matching Program increased by 312 per cent, and the number of U.S. citizens from foreign schools increased by 273 per cent. U.S. graduates of foreign medical schools participating in Fifth Pathway programs appear to benefit from their extra year of clinical training by being more successful in the National Resident Matching Program and having a higher pass rate on state licensing examinations. These increases in physicians trained in foreign medical schools, together with an increasing number of students graduating from U.S. medical schools, have resulted in an insufficient total number of first-year postgraduate positions, regardless of the specialty, to accommodate all physicians seeking a first-year residency. Since the number of residency positions will probably not expand to meet applicant demand, an increase in the pool of physicians with neither residency training nor licenses to practice medicine is likely. Alien foreign medical-school graduates and U.S. students who go abroad to study medicine can no longer take for granted residency training and practice in the United Stages.     

Vertically integrated medical education and the readiness for practice of graduates

Background

Medical curricula become more and more vertically integrated (VI) to prepare graduates better for clinical practice. VI curricula show early clinical education, integration of biomedical sciences and focus on increasing clinical responsibility levels for trainees. Results of earlier questionnaire-based studies indicate that the type of the curriculum can affect the perceived preparedness for work as perceived by students or supervisors. The aim of the present study is to determine difference in actual performance of graduates from VI and non-VI curricula.

Methods

We developed and implemented an authentic performance assessment based on different facets of competence for medical near-graduates in the role of beginning residents on a very busy day. Fifty nine candidates participated: 30 VI (Utrecht, The Netherlands) and 29 non-VI (Hamburg, Germany). Two physicians, one nurse and five standardized patients independently assessed each candidate on different facets of competence. Afterwards, the physicians indicated how much supervision they estimated each candidate would require on nine so called “Entrustable Professional Activities (EPAs)” unrelated to the observed scenarios.

Results

Graduates from a VI curriculum received significantly higher scores by the physicians for the facet of competence “active professional development”, with features like ‘reflection’ and ‘asking for feedback’. In addition, VI graduates scored better on the EPA “solving a management problem”, while the non-VI graduates got higher scores for the EPA “breaking bad news”.

Conclusions

This study gives an impression of the actual performance of medical graduates from VI and non-VI curricula. Even though not many differences were found, VI graduates got higher scores for features of professional development, which is important for postgraduate training and continuing education.

     

The association between students' research involvement in medical school and their postgraduate medical activities

The authors examined the impact of students' research involvement during medical school on their postresidency medical activities. The three medical schools involved--The Pennsylvania State University College of Medicine (PSU), The University of Connecticut School of Medicine (UCONN), and The University of Massachusetts Medical School (UMASS)--have nearly indistinguishable applicant, matriculant, and curriculum profiles. However, at PSU a research project is a curriculum requirement for students who did not do medical research prior to entering medical school. Questionnaires were sent to all graduates from the classes of 1980, 1981, and 1982. A total of 567 graduates completed the questionnaires, an overall response rate of approximately 76%. Medical school research experience was reported by 83% (183) of the PSU graduates, 34% (52) of the UCONN graduates, and 28% (54) of the UMASS graduates. When compared on a school-by-school basis, the graduates from the three schools did not differ with respect to residency specialty training, fellowship training, academic appointments, career practice choices, or postgraduate research involvement. However, when all the graduates studied were examined as a single group, medical school research experience was found to be strongly associated with postgraduate research involvement.     

Global ratings of videotaped performance versus global ratings of actions recorded on checklists: a criterion for performance assessment with standardized patients.

PURPOSE: To test whether global ratings of checklists are a viable alternative to global ratings of actual clinical performance for use as a criterion for standardized-patient (SP) assessment. METHOD: Five faculty physicians independently observed and rated videotaped performances of 44 medical students on the seven SP cases that comprise the fourth-year assessment administered at The Morchand Center of Mount Sinai School of Medicine to students in the eight member schools in the New York City Consortium. A year later, the same panel of raters reviewed and rated checklists for the same 44 students on five of the same SP cases. RESULTS: The mean global ratings of clinical competence were higher with videotapes than checklists, whereas the mean global ratings of interpersonal and communication skills were lower with videotapes. The correlations for global ratings of clinical competence showed only moderate agreement between the videotape and checklist ratings; and for interpersonal and communication skills, the correlations were somewhat weaker. CONCLUSION: The results raise serious questions about the viability of global ratings of checklists as an alternative to ratings of observed clinical performance as a criterion for SP assessment.     

Licensure, competence, and manpower distribution. A follow-up study of foreign medical graduates.

Medical statistics report United States medical graduates licensed at higher rates than foreign medical graduates. This difference is often interpreted to show greater medical competence of United States graduates. This study questions this interpretation by analyzing 1971 licensure rates for both groups who had been interns and residents in 1963. We found that factors unrelated to competence--namely, visa-citizenship status and state of examination--are associated with holding a license. Moreover, quality of medical education is not an accurate predictor of licensure. It follows that the use of licensure rates as measures of medical competence distorts understanding of the quality of medical care in the United States. More probably, the difficulties in obtaining medical licensure experienced by foreign graduates result from the use of such graduates to relieve specific medical-manpower shortages.     

Laboratory medicine in the compulsory postgraduate clinical training system--newly revised program in Nihon University School of Medicine

In 1999, the 4th report of the discussion group on 21st century medicine and health care proposed that it was necessary to establish in medical education a core curriculum emphasizing carefully selected basic content and to greatly expand the number of subjects that students might take electively. Following this report, a model core curriculum was made by the research and development project committee for medical educational programs in 2001. Similarly, the necessity of establishing a standardized postgraduate training curriculum and system has been discussed for many years. The Departments of Education and Technology and Welfare and Labor proposed a new curriculum named "Objectives in postgraduate clinical training" in order to ensure the quality of the medical doctors licensed to practice medicine in postgraduate clinical training in 2002. Following this new curriculum, Nihon University School of Medicine revised and enlarged its postgraduate program. All graduates are required to enter a full rotation program including general disciplines for two years. In addition to this basic curriculum, an elective curriculum will be added, providing attractive training programs based on medical trainees' needs. The training program in Laboratory Medicine is administered in the 2nd year as an elective program.     

Medical education in Japan.

There are 79 medical schools in Japan--42 national, 8 prefectural (i.e., founded by a local government), and 29 private--representing approximately one school for every 1.6 million people. Undergraduate medical education is six years long, typically consisting of four years of preclinical education and then two years of clinical education. High school graduates are eligible to enter medical school. In 36 schools, college graduates are offered admission, but they account for fewer than 10% of the available positions. There were 46,800 medical students in 2006; 32.8% were women.Since 1990, Japanese medical education has undergone significant changes, with some medical schools implementing integrated curricula, problem-based learning tutorials, and clinical clerkships. A model core curriculum was proposed by the government in 2001 that outlined a core structure for undergraduate medical education, with 1,218 specific behavioral objectives. A nationwide common achievement test was instituted in 2005; students must pass this test to qualify for preclinical medical education. It is similar to the United States Medical Licensing Examination step 1, although the Japanese test is not a licensing examination.The National Examination for Physicians is a 500-item examination that is administered once a year. In 2006, 8,602 applicants took the examination, and 7,742 of them (90.0%) passed. A new law requires postgraduate training for two years after graduation. Residents are paid reasonably, and the work hours are limited to 40 hours a week. In 2004, a matching system was started; the match rate was 95.6% (46.2% for the university hospitals and 49.4% for other teaching hospitals).Sustained and meaningful change in Japanese medical education is continuing.     

Mandatory postgraduate medical training in Japan--present state of Nihon University as a private medical school

This is a time of considerable uncertainty about the future of the postgraduate medical education policy of the Japanese government. Strong and visionary academic leadership of laboratory physicians in private medical schools is needed. The medical schools must not only adapt to a changing health care system, but also maintain excellence in education, patient care, and clinical research. In Japan, tradition has it that the comparatively few faculty members at national medical schools are mostly promoted only on the basis of research in experimental medicine, therefore, young medical graduates are increasingly drawn to bench work or molecular medicine, not to clinical practice. Single-minded specialization tends to produce single track minds, which may lack balanced judgment in approaching the appropriateness of both investigation and management. For continuity of care and containment of costs, a year or two of general professional training after graduation preceded by a broad medical education is an invaluable investment. All medical graduates, whatever their intended or unintended final destination (even if not clinical), should spend more than six months in medicine and four or five months in surgery, at least half of each to be spent in the general disciplines, including responsibilities for acute emergency admissions. As certified laboratory physicians we must attempt to attract graduates into laboratory medicine by developing imaginative training programs including common laboratory procedures such as Gram's stain, Wright-Giemsa stain and point of care testing at the patient's bedside or in ambulatory clinics, not only in central clinical laboratories.     

Widening the lens on standardized patient assessment: what the encounter can reveal about the development of clinical competence.

The standardized patient (SP) examination is used in a majority of medical schools to test clinical skills. This examination usually yields both numerical ratings of clinical skill and narrative comments by patients or observers, yet most empirical studies of SP assessment focus on the numerical ratings only. This quantitative focus can lead to a narrow conceptualization of the nature and development of clinical competence. The authors suggest that in addition to utilizing SP numerical ratings, medical educators also use the rich qualitative material produced in the SP examination (e.g., patient comments, videotapes of the examination) to explore students' development of clinical competence, which involves the purposive integration of basic science, technical skill, empathy, communication, professional role, and personal history.     

The measurement of outcomes in the assessment of educational program effectiveness

Postsecondary accrediting agencies recognized by the U.S. Secretary of Education and the Council on Postsecondary Accreditation, including the Liaison Committee on Medical Education (LCME), are required to evaluate educational program effectiveness by determining that institutions and programs document the achievement of their students and graduates in verifiable and consistent ways, indicating that institutional and program purposes are met. For the assessment of medical education programs this represents a departure from the traditional method of inferring quality from institutional compliance with standards for program organization and function. In the new assessment calculus, success is measured as the integrated product of the outcomes, the indicators of achievement that medical schools already are collecting from many sources, for instance, data on premedical achievement and attributes, medical school performance, graduate education ratings and test results, specialty certification, licensure, and practice. Although a recent LCME enquiry showed that 80% of U.S. medical schools were collecting outcome data on students and graduates, there was a lack of coherence and system, little integrated analysis, rare longitudinal study, and limited use of the information to evaluate and revise the curriculum or to validate admissions, promotion, and graduation criteria. The longitudinal study of the quantified results of educational programs need not resurrect old controversies about the linkage between learning in medical school and the quality of doctors' later practice. The purpose of examining outcomes is to gain sharper focus on the achievement of distinctive institutional goals, to facilitate program improvement and renewal, and to better assure the competence of graduates within the boundaries of achievement that schools have drawn as their educational objectives.     

International medical graduates: the Australian perspective.

Australia, like many other developed countries, has faced medical workforce shortages. This situation has been attributed to the increasing demands from an aging population and a decline in the hours worked by medical practitioners. These shortages, which are usually in the areas of greatest medical need in Australia, have led to an increasing dependence on international medical graduates (IMGs). The Australian government is slowly moving towards self sufficiency by expanding education and training opportunities for Australian doctors. In the interim, Australia relies heavily on IMGs to supplement the medical workforce. Australia's population is concentrated in the coastal regions, and IMGs are often required to service the more sparsely populated rural and remote areas, which find it difficult to attract and retain local medical graduates.Health funding in Australia is provided jointly by the federal (central) government and six state and two territory governments. Funding from the federal government provides for university based medical education and general practice postgraduate training. State and territory governments fund postgraduate specialist training and provide funding for a public hospital system.Although a national accreditation process for IMGs exists, many IMGs are recruited directly to Australian hospitals and community practices without adequate assessment of their qualifications or language and clinical skills. The current two-tiered system, in which service demands can override quality and standards, can no longer be tolerated. There is an urgent need for a uniformly applied national standard for all IMGs entering Australia and for a strategy to implement it.     

The predictive validity of selection for entry into postgraduate training in general practice: evidence from three longitudinal studies

Background

The selection methodology for UK general practice is designed to accommodate several thousand applicants per year and targets six core attributes identified in a multi-method job-analysis study

Aim

To evaluate the predictive validity of selection methods for entry into postgraduate training, comprising a clinical problem-solving test, a situational judgement test, and a selection centre.

Design and setting

A three-part longitudinal predictive validity study of selection into training for UK general practice.

Method

In sample 1, participants were junior doctors applying for training in general practice (n = 6824). In sample 2, participants were GP registrars 1 year into training (n = 196). In sample 3, participants were GP registrars sitting the licensing examination after 3 years, at the end of training (n = 2292). The outcome measures include: assessor ratings of performance in a selection centre comprising job simulation exercises (sample 1); supervisor ratings of trainee job performance 1 year into training (sample 2); and licensing examination results, including an applied knowledge examination and a 12-station clinical skills objective structured clinical examination (OSCE; sample 3).

Results

Performance ratings at selection predicted subsequent supervisor ratings of job performance 1 year later. Selection results also significantly predicted performance on both the clinical skills OSCE and applied knowledge examination for licensing at the end of training.

Conclusion

In combination, these longitudinal findings provide good evidence of the predictive validity of the selection methods, and are the first reported for entry into postgraduate training. Results show that the best predictor of work performance and training outcomes is a combination of a clinical problem-solving test, a situational judgement test, and a selection centre. Implications for selection methods for all postgraduate specialties are considered.     

Undergraduate educational environment, perceived preparedness for postgraduate clinical training, and pass rate on the National Medical Licensure Examination in Japan

Background

We investigated the views of newly graduating physicians on their preparedness for postgraduate clinical training, and evaluated the relationship of preparedness with the educational environment and the pass rate on the National Medical Licensure Examination (NMLE).

Methods

Data were obtained from 2429 PGY-1 physicians-in-training (response rate, 36%) using a mailed cross-sectional survey. The Dundee Ready Education Environment Measure (DREEM) inventory was used to assess the learning environment at 80 Japanese medical schools. Preparedness was assessed based on 6 clinical areas related to the Association of American Medical Colleges Graduation Questionnaire.

Results

Only 17% of the physicians-in-training felt prepared in the area of general clinical skills, 29% in basic knowledge of diagnosis and management of common conditions, 48% in communication skills, 19% in skills associated with evidence-based medicine, 54% in professionalism, and 37% in basic skills required for a physical examination. There were substantial differences among the medical schools in the perceived preparedness of their graduates. Significant positive correlations were found between preparedness for all clinical areas and a better educational environment (all p < 0.01), but there were no significant associations between the pass rate on the NMLE and perceived preparedness for any clinical area, as well as pass rate and educational environment (all p > 0.05).

Conclusion

Different educational environments among universities may be partly responsible for the differences in perceived preparedness of medical students for postgraduate clinical training. This study also highlights the poor correlation between self-assessed preparedness for practice and the NMLE.     

Predicting patient change from therapist competence and patient-therapist complementarity in short-term anxiety-provoking psychotherapy: a pilot study.

In this pilot study, therapist competence and patient-therapist complementarity measured by the Structural Analysis of Social Behavior system (SASB; Benjamin, 1974) were examined as to their interrelation and their unique, collective, and interactive contributions to patient change in 20 sessions of short-term anxiety-provoking psychotherapy (STAPP; Sifneos, 1979). Patients were 15 highly educated outpatients, mean age 30 years, with mainly anxiety diagnoses. Therapists were in postgraduate manual-guided STAPP training. Results show that competence in an early session did not relate to patient change. In contrast, patient-therapist complementarity ratings predicted patient change both alone and over and above competence. Predictions were strongest for changes in general distress and dysfunctional attitudes and for shorter term change rather than for longer term change.     

The relationship between the National Board of Medical Examiners' prototype of the Step 2 clinical skills exam and interns' performance.

PURPOSE: To examine the relationship between graduates' performances on a prototype of the National Board of Medical Examiners' Step 2 CS and other undergraduate measures with their residency directors' ratings of their performances as interns. METHOD: Data were collected for the 2001 and 2002 graduates from the study institution. Checklist and interpersonal scores from the prototype Step 2 CS, along with United States Medical Licensing Examination (USMLE) Step 1 and 2 scores and undergraduate grade-point average (GPA), were correlated with residency directors' ratings (average score for six competencies, quartile ranking, and isolated interpersonal communication competency score). Stepwise linear regression was used to identify the best outcome predictors. RESULTS: Quartile ranking was more highly correlated with GPA than Step 2 CS prototype interpersonal score, USMLE Step 2 score, USMLE Step 1 score, and Step 2 CS prototype checklist score. The average score on the residency director's survey was more highly correlated with GPA than USMLE Step 2 score, USMLE Step 1 score, Step 2 CS prototype interpersonal score, and Step 2 CS prototype checklist score. The best predictors for both quartile ranking and average competency score were GPA and Step 2 CS prototype interpersonal score (R(2) = 0.26 and 0.28). CONCLUSION: Both scores from the Step 2 CS prototype significantly correlated with the interns' quartile ranking and average competency score. Only GPA and Step 2 CS prototype interpersonal score accounted for most of the variance of performance in the regression model.