Assessing validity of standardized patient ratings of medical students' communication behavior using the Roter interaction analysis system

OBJECTIVE: The primary objective of this study is to examine concurrent validity of standardized patient (SP) ratings of second year medical students' communication skills with the Roter interaction analysis system (RIAS). METHODS: We designed An Integrated Medical Encounter (AIME), to teach second year medical students the link between communication and clinical reasoning with emphasis placed on understanding the connection between biomedical and psychosocial aspects of patient care. We randomized 120 students to intervention (AIME) and control groups (non-AIME). Students completed two post-intervention SP encounters which were videotaped and coded using RIAS. SPs used a 30-item checklist to rate students' communication behaviors. RESULTS: There were no differences between AIME and non-AIME students in age, ethnicity, gender, or college major; however, more AIME students reported prior health professional work. SPs rated AIME students' rapport-building skills higher (mean [S.E.]: 4.1 [0.15] vs. 3.9 [0.15], p=0.05); however, there were no differences in data gathering, patient education/counseling. RIAS demonstrated that AIME students more frequently used rapport-building statements (60.4 [2.7] vs. 52.1 [2.8], p=0.03). CONCLUSION: The RIAS confirmed SP ratings of differences in AIME and non-AIME students' rapport-building skills. PRACTICE IMPLICATIONS: Future studies in medical education should further examine the minimum number of SP ratings needed to effectively evaluate communication skills curricula when resources are limited.     

Can Verona Coding Definitions of Emotional Sequences (VR-CoDES) be applied to standardized Chinese medical consultations? – A reliability and validity investigation

ObjectiveTo investigate the reliability and validity of the Chinese version of VR-CoDES.MethodsThe VR-CoDES was translated into Chinese, and a focus group was held to discuss its cultural adaptation. Video consultations between 75 fourth-year medical students and 2 standardized patients (SPs) were coded by two raters with the Chinese VR-CoDES. Inter-rater reliability was tested by using ICC. To obtain validity, the SPs reviewed the video consultations to confirm the cues and concerns.ResultsICC was 0.79. Specificity and sensitivity were 0.99 and 0.96 respectively. The SPs expressed considerably more cues (mean = 7.00) than concerns (mean = 0.32). Half of the responses of medical students were explicit reducing space. Focus group participants raised some cultural considerations, and some interactions were difficult to code due to cultural differences.ConclusionThe Chinese VR-CoDES obtained good reliability and validity. Due to differences in the expression of emotions and other differences such as different medical systems between China and Western countries, the Chinese VR-CoDES needs further cultural adaptation.Practice implicationMore consultations in real clinical settings need to be gathered to further support the Chinese VR-CoDES both on validation and cultural adaptation.     

From enrichment to equity: comments on diversifying the K-12 medical school pipeline

Educating a physician workforce that reflects the increasing racial and ethnic diversity of our nation is an ongoing challenge of urgent concern. Many medical school kindergarten through 1 2th grade (K-12) pipeline programs focus on "enriching" underrepresented minority (URM) students using strategies to change or "improve" individual students. This discussion raises concerns over longstanding racial and ethnic inequities in America's public schools that, in part, result in the predictable and systematic underachievement of URM students. These insidious processes can disqualify URM students from successful participation in the medical school pipeline at its earliest stages. The paper also discusses the cultural challenges URM students often face in aspiring to exceptional academic achievement within America's schools. Finally, this paper highlights the need for illustrative examples of medical school-public school partnerships that pursue an agenda of equity to balance the current downstream focus on the enrichment of individual students.     

Effects of using two or more standardized patients to simulate the same case on case means and case failure rates

The effects of using two or more standardized patients (multiple SPs) to simulate the same case in a performance-based examination were studied at the case level by comparing case means and case failure rates for multiple SPs simulating the same case, using data from the classes of 1988, 1989, and 1990 at the Southern Illinois University School of Medicine. For total scores and scores on the students' written answers, the effects on means and failure rates were negligible and could be explained as due to sampling error. For scores on the checklists completed by the SPs, there were more significant differences than would be expected by chance alone, even though the number of significant differences was relatively small. The results demonstrate a need for caution in the interpretation of scores obtained from a case checklist completed by multiple SPs, particularly in regard to making pass-fail decisions.     

The effect of candidates' perceptions of the evaluation method on reliability of checklist and global rating scores in an objective structured clinical examination.

PURPOSE: Process-oriented global ratings, which assess "overall performance" on one or a number of domains, have been purported to capture nuances of expert performance better than checklists. Pilot data indicate that students change behaviors depending on their perceptions of how they are being scored, while experts do not. This study examines the impact of the students' orientation to the rating system on OSCE scores and the interstation reliability of the checklist and global scores. METHOD: A total of 57 third- and fourth-year medical students at one school were randomly assigned to two groups and performed a ten-station OSCE. Group 1 was told that scores were based on checklists. Group 2 was informed that performance would be rated using global ratings geared toward assessing overall competence. All candidates were scored by physician-examiners who were unaware of the students' orientations to the rating system and who used both checklists and global rating forms. RESULTS: A mixed two-factor ANOVA identified a significant interaction of rating form by group (F(1,55) = 5.5, p <.05), with Group 1 (checklist-oriented) having higher checklist scores but lower global scores than did Group 2 (oriented to global ratings). In addition, Group 1 had higher interstation alpha coefficients than did Group 2 for both global scores (0.74 versus 0.63) and checklist scores (0.63 versus 0.40). CONCLUSIONS: The interaction effect on total exam scores suggests that students adapt their behaviors to the system of evaluation. However, the lower reliability coefficients for both forms found in the process-oriented global-rating group suggest that an individual's capacity to adapt to the system of global rating forms is relatively station-specific, possibly depending on his or her expertise in the domain represented in each station.     

The influence of testing context and clinical rotation order on students' OSCE performance.

PURPOSE: To investigate the influence of testing context and rotation order on third-year medical student performance on a common objective structured clinical examination (OSCE) station in both obstetrics-gynecology (ob-gyn) and psychiatry rotations. METHOD: Archival OSCE performance data (in the form of a 25-item binary content checklist) from one class of third-year medical students (n = 141) at Saint Louis University (2002-03) were aggregated and analyzed. RESULTS: Despite the fact that the station was identical in both OSCEs, students were, in general, less likely to inquire about ob-gyn issues on the psychiatry OSCE and less likely to inquire about psychiatric issues on the ob-gyn OSCE, regardless of order of rotation. Order did have a positive effect on some results, such that students were more likely to mention menopause and vaginal dryness on the psychiatry OSCE if they had already had the ob-gyn rotation. CONCLUSION: The testing context may influence student approaches to patients in ways that bias their collection and interpretation of information. OSCE evaluations may better approximate true clinical context and complexity by presenting case scenarios that reflect a broader range of diagnostic possibilities than those limited to the recently completed rotation.     

Completing the picture on student performances in OSCEs: A mixed-methods study on integration of a standardized patient rating

ObjectiveGood physician communication skills increase patient satisfaction and improve healing processes. Although physicians and patients appear to value communicative competencies differently, students are often evaluated solely by physicians. This study examines whether additional assessment of students by ‘standardized patients’ (SPs) is useful.MethodsDuring their Objective Structured Clinical Examination (OSCE) 238 medical students were additionally rated by SPs at 9 stations according to two items that defined the ‘physician–patient relationship’ and ‘communication’. SPs were informed that their assessment was for research purposes only, with no impact on the assessment of the students. SPs also had the opportunity to comment on their rating of the students.ResultsThe SPs rated the communicative competencies of students differently than physicians. The two parts of the SP rating are closely related. Inclusion of SP rating in the OSCE would provide higher measurement precision, with more students failing. SPs considered five factors relevant in their rating: ‘human connection’, ‘information flow’, ‘professionalism’, ‘competence’, and ‘exam situation’.ConclusionOur study suggests inclusion of SP rating as additional assessment of student communication skills.Practice ImplicationsAddition of SP rating in assessments is worthwhile, as it appears to complete the picture of the student performance in their OSCEs.     

Can patient-centered attitudes reduce racial and ethnic disparities in care?

PURPOSE: Patient-centeredness has been advocated to reduce racial/ethnic disparities in health care quality, but no empirical data support such a connection. The authors' purpose was to determine whether students with patient-centered attitudes have better performance and are less likely to demonstrate disparities with African American compared with white standardized patients (SPs). METHOD: Third-year medical students were assessed by SPs at the Clinical Educational Center of the Johns Hopkins University School of Medicine in 2002. One African American and one white actor were trained as SPs for each of four case scenarios; students were randomly assigned to interact with either SP for each case. Before the exam, students were surveyed about their attitudes towards patient-centered medicine. Students with and without patient-centered attitudes were compared with regard to their performance with African American and white SPs. Outcome measures were student exam scores in interpersonal skill, history taking, physical exam, and counseling. RESULTS: All 177 of eligible students participated in all four case scenarios. With white SPs, students with patient-centered attitudes performed similarly to students without patient-centered attitudes in all four areas. However, with African American SPs, students with patient-centered attitudes performed significantly better than students without patient-centered attitudes in interpersonal skills (71.4 versus 69.4, P = .010), history taking (63.8 versus 61.1, P = .003), and counseling (92.1 versus 88.7, P = .002) and not significantly different in physical exam performance (73.6 versus 68.6, P = .311). CONCLUSIONS: Patient-centered attitudes may be more important in improving physician behaviors with African American patients than with white patients and may, therefore, play a role in reducing disparities.     

Student perspectives on diversity and the cultural climate at a U.S. medical school.

PURPOSE: To obtain the perspectives of medical students at one school on racial/ethnic campus diversity and cultural competence and to gain their perceptions of the institutional climate around diversity at their university and of reasons for minority underrepresentation at their medical school. METHOD: A student-driven survey of all medical students (N = 398) at a single medical school in the spring of 2003, supplemented by four focus groups from all racial and ethnic groups on the campus. RESULTS: A large majority of the responding students (n = 216; 54%) endorsed the value of campus diversity and the importance of cultural competence to the process of becoming a clinician. Most students felt their university had achieved a positive cultural climate, characterized by openness to diverse perspectives and attention to equity. Most students also felt that the university's programs and policies reflected a commitment to diversity, but fewer students--those from underrepresented minorities (URMs) in particular--felt that the university truly valued having a diverse student body and faculty. Most students felt that the lack of diversity on campus was a barrier to recruiting and retaining minority candidates. Some minority students also blamed the medical school's limited social, academic, and financial support, as well as inadequate efforts to recruit minority students. CONCLUSIONS: Medical students generally place a high value on campus diversity and cultural competence. URM students in particular felt that their university could do more to implement its commitment to diversity, including making greater efforts to recruit and retain URM students. These views constitute a barometer for medical schools to gauge and track their efforts to enhance campus diversity, incorporate cultural competence education, and create an inclusive and welcoming climate for students of all backgrounds.     

Educational benefits of diversity in medical school: a survey of students.

PURPOSE: Many U.S. medical schools have abandoned affirmative action, limiting the recruitment and reducing the admission of underrepresented minority (URM) students even though research supports the premise that the public benefits from an increase in URM physicians and that URM physicians are likely to serve minority, poor, and Medicaid populations. Faculty and students commonly assume they benefit from peer cultural exchange, and the published evidence for the past two decades supports this notion. This research examined the students' perceptions of the educational merits of a diverse student body by surveying medical students at two schools. METHOD: In 2000, medical students from all four years at Harvard Medical School and the University of California, San Francisco, School of Medicine were enrolled in a telephone survey about the relevance of racial diversity (among students) in their medical education. Students responded to the interviewer's questions on a five-point Likert-type scale. RESULTS: Of the 55% of students who could be located, 97% responded to the survey. Students reported having little intercultural contact during their formative years but significantly more interactions during higher education years, especially in medical school. Students reported contacts with diverse peers greatly enhanced their educational experience. They strongly supported strengthening or maintaining current affirmative action policies in admissions. The responses and demography of the Harvard and UCSF students did not differ significantly, nor did they differ for majority students and URM students-all groups overwhelmingly thought that racial and ethnic diversity among their peers enhanced their education. CONCLUSIONS: Diversity in the student body enhanced the educational experiences of students in two U.S. medical schools.     

A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator.

PURPOSE: To determine the correlation between global ratings and criterion-based checklist scores, and inter-rater reliability of global ratings and criterion-based checklist scores, in a performance assessment using an anesthesia simulator. METHOD: All final-year medical students at the University of Toronto were invited to work through a 15-minute faculty-facilitated scenario using an anesthesia simulator. Students' performances were videotaped and analyzed by two faculty using a 25-point criterion-based checklist and a five-point global rating of competency (1 = clear failure, 5 = superior performance). Correlations between global ratings and checklist scores, as well as specific performance competencies (knowledge, technical skills, and judgment), were determined. Checklist and global scores were converted to percentages; means of the two marks were compared. Mean reliability of a single rater for both checklist and global ratings was determined. RESULTS: The correlation between checklist and global ratings was.74. Mean ratings of both checklist and global scores were low (58.67, SD = 14.96, and 57.08, SD = 24.27, respectively); these differences were not statistically significant. For a single rater, the mean reliability score across rater pairs for checklist scores was.77 (range.58-.93). Mean reliability score across rater pairs for global ratings was.62 (.40-.77). Global ratings correlated more highly with technical skills and judgment (r =.51 and r =.53, respectively) than with knowledge. (r =.24) CONCLUSION: Inter-rater reliability was higher for checklist scores than for global ratings; however, global ratings demonstrated acceptable inter-rater reliability and may be useful for competency assessment in performance assessments using simulators.     

Ability of primary care physician's to break bad news: a performance based assessment of an educational intervention

OBJECTIVE: We have previously described a breaking bad news (BBN) training program for primary care physicians [Ungar L, Alperin M, Amiel GE, Beharier Z, Reis S. Breaking bad news: structured training for family medicine residents. Patient Educ Couns 2002;48:63-68]. In this paper, we present the assessment of an educational intervention aimed at improving this important skill. METHODS: The assessment tool was an eight station objective structured clinical examination (OSCE) utilizing standardized patients (SPs). Intervention and control groups of 17 general practitioners (GP) each were evaluated before and after an educational intervention, or a Balint group (control). RESULTS: Intervention group GPs significantly increased their average grade on the post-test as compared to the pre-test (58.5, S.D. 12.7 versus 68.4, S.D. 9.2), effect size 0.94. Improvement in the control group was minimal (pre-test 57, S.D. 10.4 versus 58.1, S.D. 9.5 for the post-test), effect size 0.23. Reliability of the OSCE was alpha = 0.81. CONCLUSION: The performance assessment used in this study proved to be a reliable and valid tool to assess the ability of physicians to break bad news. It provided evidence of the effectiveness of the intervention. PRACTICE IMPLICATIONS: BBN training can and should be evaluated by valid and reliable measures. SPs can serve as reliable evaluators of BBN training.     

Comparing students' feedback about clinical instruction with their performances

Validation of students' feedback as a measure of teaching effectiveness has been problematic for courses teaching clinical skills. This is true in part because establishing a valid and reliable method of assessing students' mastery of clinical skills has been a stumbling block. Reported here is the correlation of students' performances on an objective structured clinical examination (OSCE) with previously and independently collected feedback from students. In 1987-88, 190 second-year medical students at the University of Minnesota Medical School--Minneapolis spent one fourth of a second-year clinical skills course on neurology randomly assigned to one of four teaching sites--hospitals A, B, C, and D. Following their rotations, 180 of the students completed usable feedback forms. The students were consistently and significantly more positive about the teaching at hospital A. At the end of the year, all 190 students were tested using an OSCE having 20 stations, four of which presented neurologic problems. The students who had the neurology course at hospital A performed better on all four neurology problems, and differences were statistically significant for two of the problems. Feedback in this case accurately reflected a more effective teaching program.     

A mentoring program for underrepresented-minority students at the University of Rochester School of Medicine.

Mentoring underrepresented-minority (URM) students poses a special challenge because most medical schools have few URM faculty and many non-URM faculty hesitate to be mentors for URM students. Some medical students perform less well in the clinical years than would be expected from their pre-clinical performances. One factor is some students' difficulty in adapting to the culture of medicine, which mentors can help students overcome. The University of Rochester School of Medicine created the Medical Student Mentoring Program to address the needs of URM students and non-URM faculty who could be mentors. The program, offered in 1995-96 and 1996-97, trained mentors, created a bicultural support group for URM students, and provided structured mentoring. Interviews were conducted with faculty and students to identify critical areas that influence the success of URM students in their clinical years; URM faculty, residents, and advanced students shared their experiences with the program students at reflection group meetings. Mentors participated in an initial orientation. Of the 42 students eligible during 1995-1997, 30 participated and were assigned to 15 mentors. At the end of the program's first year, the students and mentors gave their reactions, and although there were some differences in their viewpoints, overall they considered the program useful. Non-URM faculty appreciated the support and guidance that allowed them to mentor URM students more effectively. The program ran formally for two years, and some of the mentoring relationships continued into the third year. Loss of funding and change in administrative leadership contributed to the ending of this program. Mentoring continues to be a priority at the medical center, and a new mentoring program has been developed for URM and non-URM medical students.     

Effectiveness of a formal post-baccalaureate pre-medicine program for underrepresented minority students.

PURPOSE: To address the effectiveness of a formal postbaccalaureate (PB) experience for underrepresented minority (URM) students before medical school. The program provided an intense year-long experience of course work, research, and personal development. METHOD: There were 516 participants from one medical school: 15 URM medical students had completed the formal PB program, 58 students had done independent PB work before matriculation, and 443 students were traditional matriculants. Cognitive and academic indicators [college science and non-science grade-point averages (GPAs); biology, physics, and verbal MCAT scores; and percentage scores from first-year medical school courses] were compared for the three groups. RESULTS: Both groups of students with PB experience demonstrated competency in the first year of medical school consistent with traditional students even though the students who had completed the formal PB program had lower MCAT scores and lower college GPAs than did the traditional students. Traditional predictors of academic performance during the first year of medical school did not significantly contribute to actual academic performances of students from the formal PB program. CONCLUSION: The results support the use of a formal PB program to provide academic readiness and support for URM students prior to medical school. Such a program may also improve retention. Noncognitive variables, however, may be important to understanding the success of such students in medical school.     

The Influence of Gender and Underrepresented Minority Status on Medical Student Ranking of Residency Programs

BackgroundPhysician diversity is linked to improved quality of care of diverse patient populations. The transition from medical school to residency is an opportunity to improve and increase workforce diversity in all specialties. However, there is limited published literature on the factors contributing to the ranking of residency programs on women and underrepresented minorities (URMs).ObjectiveTo characterize factors medical students used to rank residency programs and describe any differences based on race/ethnicity or gender.MethodsA mixed-methods study consisting of a web-based survey and semi-structured interviews with National Resident Matching Program (NRMP) participating graduates over a two-year period. The survey assessed demographics and a 6-point Likert scale rating of various factors used to rank residency programs. Unpaired student t-tests were used to compare means. A subset of students was interviewed and a modified grounded theory approach identified decision-making themes as well as the role of gender and URM status.ResultsOut of a total of 316 invitations sent, 148 completed the survey (46.8% response rate), of which 21% of respondents self-identified as URMs. The majority of respondents graduated in 2014 (53%), and were male (51%). Participants ranked program atmosphere, reputation, location, and proximity to family the highest. URM students ranked patient population (p < 0.01), revisit opportunities (p = 0.04), gender diversity (p < 0.01), and ethnic diversity (p < 0.01) significantly higher than non-URM students. Female students ranked patient population (p < 0.01) and gender diversity (p < 0.01) significantly higher than males. Qualitative findings revealed differences in perceptions by URMs and non-URMs of patient population, revisit opportunities, gender diversity, and ethnic diversity.ConclusionsWhile all students prioritized pragmatic factors, women and URM students assess and weigh additional factors related to culture, inclusion, and diversity more than others. By tailoring recruitment strategies to meet the expectations of women and URMs, residency programs can better meet goals in becoming more diverse and inclusive.     

Can specialists and generalists teach clinical skills to second-year medical students with equal effectiveness?

PURPOSE: To compare the effectiveness of specialists and generalists as small-group leaders teaching basic physical examination skills to preclinical medical students. METHOD: Specialists and generalists were randomly assigned to teach physical examination skills to 69 groups of second-year students (n = 288). At the conclusion of the course, the specialist- and generalist-led groups were compared using three measures: students' scores on an objective structured clinical exam (OSCE), students' evaluations of their small-group leaders, and leaders' self-evaluations of confidence in teaching. RESULTS: OSCE scores did not differ between students taught by specialists and generalists (93.5% and 93.8% respectively, p = NS). Students' evaluations of their leaders were similar for nine characteristics rated on a seven-point scale (7 = strongly agree/outstanding; range of results for specialists: 6.20-6.62, for generalists 6.34-6.75, p = NS). Leaders expressed similar levels of confidence (on a seven-point scale; 7 = very confident) in their abilities to teach the neurologic exam (specialists 5.52, generalists 6.19, mean effect size difference 0.44, p = NS) and complete history and physical exam (6.03 and 6.53, mean effect size difference 0.43, p = NS). Specialists were significantly less confident in teaching the cardiovascular exam (5.80 and 6.50, mean effect size difference 0.51, p <.05) and pulmonary exam (5.56 and 6.60, mean effect size difference, 0.80, p <.01). CONCLUSIONS: Specialists and generalists can teach preclinical medical students with equal effectiveness as rated by the students and the students' scores on the OSCE examination, but specialists do not rate themselves as confident as do generalists to teach some skills.     

OSCE在北京市放射科住院医师规范化培训Ⅰ阶段技能考试中应用的评价

本研究对客观结构化临床考试(OSCE)在北京市放射科住院医师规范化培训Ⅰ阶段技能考试中的应用效果进行了评估。研究对象为2014年至2016年参加北京市Ⅰ阶段考试的放射科住院医师,研究考生的学历组成、不同学历考生的平均分及通过率、考生对考试的调查问卷反馈进行了评价。考生组成中本科学历比例逐渐增高,博士生及硕士生考试通过率高于本科生,考试平均分整体趋势为博士生考试平均分高于硕士生和本科生;OSCE这一概念在考生中普及率逐年升高,考生赞同普及这种考试形式,对于胜任力的概念在培训过程中有待推广,尤其要加强沟通能力培训,对于考试的流程普遍反映良好。放射科住院医师未来培训应该将培训内容与学历相结合;考生对OSCE这一形式接受度很高;在日常培训内容应围绕临床胜任力进行,并注重沟通能力的培训,要继续普及应用OSCE。     

Medical student gender and issues of confidence

OBJECTIVE: To review the literature on gender differences and issues of self-confidence in medical students and to present original research on observers' perceptions of medical student confidence. METHODS: One hundred forty-one 3rd year medical students at Indiana University School of Medicine were videotaped during their objective structured clinical examination (OSCE). Trained coders rated how confident the student appeared and coded a variety of nonverbal behaviors at the beginning, middle, and end of the interaction. Analysis focused on gender differences in coders' ratings of perceived confidence. RESULTS: Female medical students were viewed as significantly less confident than male medical students (F(1,133)=4.45, p<0.05), especially at the beginning of the interaction. CONCLUSION: Past research indicates that despite performing equally to their male peers, female medical students consistently report decreased self-confidence and increased anxiety, particularly over issues related to their competence. In a standardized patient interaction examination situation, female medical students also appeared significantly less confident than male medical students to independent observers. PRACTICE IMPLICATIONS: Medical educators should focus on issues of female students' confidence, increasing faculty sensitivity, and publicly recognizing and discussing perceptions of confidence.