Assessing validity of standardized patient ratings of medical students' communication behavior using the Roter interaction analysis system

OBJECTIVE: The primary objective of this study is to examine concurrent validity of standardized patient (SP) ratings of second year medical students' communication skills with the Roter interaction analysis system (RIAS). METHODS: We designed An Integrated Medical Encounter (AIME), to teach second year medical students the link between communication and clinical reasoning with emphasis placed on understanding the connection between biomedical and psychosocial aspects of patient care. We randomized 120 students to intervention (AIME) and control groups (non-AIME). Students completed two post-intervention SP encounters which were videotaped and coded using RIAS. SPs used a 30-item checklist to rate students' communication behaviors. RESULTS: There were no differences between AIME and non-AIME students in age, ethnicity, gender, or college major; however, more AIME students reported prior health professional work. SPs rated AIME students' rapport-building skills higher (mean [S.E.]: 4.1 [0.15] vs. 3.9 [0.15], p=0.05); however, there were no differences in data gathering, patient education/counseling. RIAS demonstrated that AIME students more frequently used rapport-building statements (60.4 [2.7] vs. 52.1 [2.8], p=0.03). CONCLUSION: The RIAS confirmed SP ratings of differences in AIME and non-AIME students' rapport-building skills. PRACTICE IMPLICATIONS: Future studies in medical education should further examine the minimum number of SP ratings needed to effectively evaluate communication skills curricula when resources are limited.     

Widening the lens on standardized patient assessment: what the encounter can reveal about the development of clinical competence.

The standardized patient (SP) examination is used in a majority of medical schools to test clinical skills. This examination usually yields both numerical ratings of clinical skill and narrative comments by patients or observers, yet most empirical studies of SP assessment focus on the numerical ratings only. This quantitative focus can lead to a narrow conceptualization of the nature and development of clinical competence. The authors suggest that in addition to utilizing SP numerical ratings, medical educators also use the rich qualitative material produced in the SP examination (e.g., patient comments, videotapes of the examination) to explore students' development of clinical competence, which involves the purposive integration of basic science, technical skill, empathy, communication, professional role, and personal history.     

"Teach 1, do 1 ... better": superior communication skills in senior medical students serving as standardized patient--examiners for their junior peers.

PURPOSE: To see if senior medical students who had served as standardized patients (SP) demonstrated improvement in their own interpersonal communication skills. METHOD: From George Washington School of Medicine's class of 1996, 154 fourth-year students took a clinical practice examination that used professional SPs. Within the preceding six months, 28 of these students had been SP-examiners in similar examinations for first- and second-year medical students. The professional SPs rated the fourth-year examinees using checklists that measured five dimensions of interpersonal communication skills. Four of these five dimensions were identical to those measured on the examinations for which the fourth-year students had served as SPs. Hypothesizing that the SP-experienced seniors would outscore their inexperienced classmates on those four dimensions, but not on the fifth, the authors analyzed the fourth-year students' scores. P values were computed by the F test from a two-way analysis of variance. RESULTS: As predicted, the group with prior SP experience significantly outscored their inexperienced colleagues in each of the four expected dimensions of interpersonal communication skills, with p values ranging from .000 to .023. The score differential in these dimensions ranged from 3.8 to 11.8 percentage points. As further predicted, there was no significant difference between the scores of the two groups on the fifth dimension. CONCLUSION: Compared with their inexperienced peers, senior medical students with prior SP experience consistently demonstrated superior scores when their own communication skills were tested in a similar manner. The U.S. Medical Licensing Examinations propose to incorporate SP clinical examinations; in response, medical schools will use more SP examinations in their own curricula. Such examinations are expensive when using professional standardized patients; the authors believe that an SP program using senior medical students will prove an attractive alternative. Such programs may have the added advantage of making better communicators of senior medical student teachers as well as the students they teach.     

Relationship between clinical competence and interpersonal and communication skills in standardized-patient assessment.

PURPOSE: To evaluate the relationship between clinical competence and interpersonal and communication skills, in an attempt to clarify current thinking about these two dimensions as measured with standardized-patient (SP) examinations. METHOD: Simple Pearson correlations were computed between total examination scores for clinical competence and interpersonal and communication skills. Three sets of different types of data involving 15 separate examinations were used to explore the generality of the findings. To control for a within-case halo effect and measurement error, corrected cross-half correlations and corrected cross-case correlations were also computed. RESULTS: The simple correlations and the corrected cross-half and cross-case correlations showed moderate and above relationships between these two dimensions in the clinical context. The simple correlations centered around .50, and the corrected cross-half and cross-case correlations were slightly higher, centering around .65 and .70, respectively. CONCLUSION: The authors' thinking is that the moderate relationship between clinical competence and interpersonal and communication skills is not due to a flaw in the measurement of clinical competence, as has been suggested, but rather is a natural consequence of the clinical encounter, which exacts an interdependence of these two dimensions. At least, this possibility must be seriously considered so medical educators can think and act appropriately in the assessment of clinical performance.     

Relationship of tutors' group-dynamics skills to their performance ratings in problem-based learning.

PURPOSE: A tutor's performance is often investigated as a stable teacher characteristic, isolated from the context in which the tutor functions. This study investigated the influence of a tutor's group-dynamics skills in differently functioning tutorial groups on performance ratings. METHOD: Students' ratings of tutors' performances collected from 75 tutorial groups at one school over four different academic years were compared with the independent variables, groups' cohesion scores and sponging scores (the degree to which some students let others do the work), and the intervening variable, tutors' group-dynamics skills. RESULTS: Tutors with strong group-dynamics skills were assigned mean tutor's-performance scores of 7.4 (SD = 0.8) and 8.2 (SD = 0.5), respectively, by groups scoring low and high on cohesion. Tutors with low group-dynamics skills were rated 7.0 (SD = 1.1) and 7.4 (SD = 1.1) by the same groups. The same pattern held for sponging scores. Tutors who had strong group-dynamics skills received mean tutor's-performance scores of 8.1 (SD = 0.6) and 7.4 (SD = 0.8), respectively, from groups with low and high sponging scores. Tutors with weak group-dynamics skills were scored 7.6 (SD = 1.0) and 6.9 (SD = 1.1) by the same groups. CONCLUSIONS: Tutors possessing group-dynamics skills were rated higher by students than were tutors who lacked these skills, irrespective of the quality of a tutorial group's performance. A tutor who evaluates tutorial-group function on a regular basis and makes appointments with students based on these evaluations is seen as performing better than a tutor who does not.     

Assessing competence in communication and interpersonal skills: the Kalamazoo II report.

Accreditation of residency programs and certification of physicians requires assessment of competence in communication and interpersonal skills. Residency and continuing medical education program directors seek ways to teach and evaluate these competencies. This report summarizes the methods and tools used by educators, evaluators, and researchers in the field of physician-patient communication as determined by the participants in the "Kalamazoo II" conference held in April 2002. Communication and interpersonal skills form an integrated competence with two distinct parts. Communication skills are the performance of specific tasks and behaviors such as obtaining a medical history, explaining a diagnosis and prognosis, giving therapeutic instructions, and counseling. Interpersonal skills are inherently relational and process oriented; they are the effect communication has on another person such as relieving anxiety or establishing a trusting relationship. This report reviews three methods for assessment of communication and interpersonal skills: (1) checklists of observed behaviors during interactions with real or simulated patients; (2) surveys of patients' experience in clinical interactions; and (3) examinations using oral, essay, or multiple-choice response questions. These methods are incorporated into educational programs to assess learning needs, create learning opportunities, or guide feedback for learning. The same assessment tools, when administered in a standardized way, rated by an evaluator other than the teacher, and using a predetermined passing score, become a summative evaluation. The report summarizes the experience of using these methods in a variety of educational and evaluation programs and presents an extensive bibliography of literature on the topic. Professional conversation between patients and doctors shapes diagnosis, initiates therapy, and establishes a caring relationship. The degree to which these activities are successful depends, in large part, on the communication and interpersonal skills of the physician. This report focuses on how the physician's competence in professional conversation with patients might be measured. Valid, reliable, and practical measures can guide professional formation, determine readiness for independent practice, and deepen understanding of the communication itself.     

Validity and generalizability of global ratings in an objective structured clinical examination

The performance of foreign medical graduates on multistation standardized patient-based tests and used to determine the validity and generalizability of global ratings of their clinical competence made by expert examiners. Data were derived from the entrance examinations of the 1989 and 1990 applicants to the Ontario Pre-Internship Program and the exit examination of 24 participants from the 1989 cohort. For each candidate, the examiners completed a detailed checklist and two five-point global ratings dealing with the candidate's approach to the patients' problem and attitude toward the patient. Generalizability coefficients for both ratings were satisfactory and stable across cohorts. Construct validity of the global ratings was demonstrated by comparing entry and exit ratings and by evidence of significant and positive correlations between the global ratings and total test scores. Tentative evidence of criterion validity of the global ratings was demonstrated. These findings suggest that global ratings by expert examiners can be used as an effective form of assessment in multistation standardized patient examinations.     

Measuring outcomes of undergraduate medical education: residency directors' ratings of first-year residents.

PURPOSE: There is a growing recognition of the need to show the relationship between undergraduate medical education (UME) and achievements during residency. This study provides reliability and validity evidence for a residency rating scale as well as a method for gathering comparison information about first-year residents. METHOD: A 25-item rating scale measuring important areas of physician functioning was mailed to residency directors of 485 graduates of the 1998-2000 classes of the University of Kansas School of Medicine. The same rating scale was sent to residency directors for a comparison sample of 251 graduates of other U.S. medical schools who were residents at the University of Kansas Medical Center. Each item on the rating scale was rated on a five-point Likert scale. Principal-components analysis, correlational analyses, internal consistency reliability analysis, and mean comparisons were used to provide evidence of reliability and validity. RESULTS: A total of 382 (82%) usable rating scales were returned. A principal-components analysis extracted five factors that accounted for 86% of the variance. The final factors were (1) interpersonal communication, (2) clinical skills, (3) population-based health care, (4) record-keeping skills, and (5) critical appraisal skills. The internal consistency of the entire scale was.98, with coefficients for the five factors ranging from.92 to.97. The correlations between the five factors and measures of undergraduate performance ranged from.21 to.49. Group analyses revealed that residents with high GPAs and USMLE Step 1 and Step 2 scores tended to be rated higher than those with lower scores. CONCLUSIONS: The rating scale demonstrated adequate reliability and validity and showed that residency directors' ratings are a useful outcome measure for UME performance.     

Validation of the ComCare index for rater-based assessment of medical communication and interpersonal skills

ObjectivesTo develop and validate a short instrument to assess undergraduate medical students’ communication and interpersonal skills in videographed history taking situations with simulated patients.MethodsSixty-seven undergraduate medical students participating in an assessment including videographed physician-patient encounters for history taking with five simulated patients were included in this study. The last video of each participant’s consultation hour was rated by two independent assessors with the eight-item ComCare index for assessment of communication and interpersonal skills newly designed for the external rater perspective (ComCareR). We compared the sum scores of the ComCareR with ratings of the same videos with the Kalamazoo Communication Skills Assessment Form from an observational perspective (KCSAFd-video) and the Global Rating scale (GR), which also measure communication and interpersonal skills.ResultsThe ComCareR showed an excellent interrater reliability (ICC = .85). We found a small but significant correlation with the KCSAFd-video Interpersonal Competence (ρ = .34, 95% CI [.10,.54]) and a high positive correlation with the GR (ρ = .59, 95% CI [.40,.73]).ConclusionsThe ComCareR is a valid and brief index for holistic assessment of communication and interpersonal skills in physician-patient encounters.Practice implicationsThe ComCareR can be used for quick rater-based assessment of physicians’ communication and interpersonal skills.     

Completing the picture on student performances in OSCEs: A mixed-methods study on integration of a standardized patient rating

ObjectiveGood physician communication skills increase patient satisfaction and improve healing processes. Although physicians and patients appear to value communicative competencies differently, students are often evaluated solely by physicians. This study examines whether additional assessment of students by ‘standardized patients’ (SPs) is useful.MethodsDuring their Objective Structured Clinical Examination (OSCE) 238 medical students were additionally rated by SPs at 9 stations according to two items that defined the ‘physician–patient relationship’ and ‘communication’. SPs were informed that their assessment was for research purposes only, with no impact on the assessment of the students. SPs also had the opportunity to comment on their rating of the students.ResultsThe SPs rated the communicative competencies of students differently than physicians. The two parts of the SP rating are closely related. Inclusion of SP rating in the OSCE would provide higher measurement precision, with more students failing. SPs considered five factors relevant in their rating: ‘human connection’, ‘information flow’, ‘professionalism’, ‘competence’, and ‘exam situation’.ConclusionOur study suggests inclusion of SP rating as additional assessment of student communication skills.Practice ImplicationsAddition of SP rating in assessments is worthwhile, as it appears to complete the picture of the student performance in their OSCEs.     

A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator.

PURPOSE: To determine the correlation between global ratings and criterion-based checklist scores, and inter-rater reliability of global ratings and criterion-based checklist scores, in a performance assessment using an anesthesia simulator. METHOD: All final-year medical students at the University of Toronto were invited to work through a 15-minute faculty-facilitated scenario using an anesthesia simulator. Students' performances were videotaped and analyzed by two faculty using a 25-point criterion-based checklist and a five-point global rating of competency (1 = clear failure, 5 = superior performance). Correlations between global ratings and checklist scores, as well as specific performance competencies (knowledge, technical skills, and judgment), were determined. Checklist and global scores were converted to percentages; means of the two marks were compared. Mean reliability of a single rater for both checklist and global ratings was determined. RESULTS: The correlation between checklist and global ratings was.74. Mean ratings of both checklist and global scores were low (58.67, SD = 14.96, and 57.08, SD = 24.27, respectively); these differences were not statistically significant. For a single rater, the mean reliability score across rater pairs for checklist scores was.77 (range.58-.93). Mean reliability score across rater pairs for global ratings was.62 (.40-.77). Global ratings correlated more highly with technical skills and judgment (r =.51 and r =.53, respectively) than with knowledge. (r =.24) CONCLUSION: Inter-rater reliability was higher for checklist scores than for global ratings; however, global ratings demonstrated acceptable inter-rater reliability and may be useful for competency assessment in performance assessments using simulators.     

The effect of candidates' perceptions of the evaluation method on reliability of checklist and global rating scores in an objective structured clinical examination.

PURPOSE: Process-oriented global ratings, which assess "overall performance" on one or a number of domains, have been purported to capture nuances of expert performance better than checklists. Pilot data indicate that students change behaviors depending on their perceptions of how they are being scored, while experts do not. This study examines the impact of the students' orientation to the rating system on OSCE scores and the interstation reliability of the checklist and global scores. METHOD: A total of 57 third- and fourth-year medical students at one school were randomly assigned to two groups and performed a ten-station OSCE. Group 1 was told that scores were based on checklists. Group 2 was informed that performance would be rated using global ratings geared toward assessing overall competence. All candidates were scored by physician-examiners who were unaware of the students' orientations to the rating system and who used both checklists and global rating forms. RESULTS: A mixed two-factor ANOVA identified a significant interaction of rating form by group (F(1,55) = 5.5, p <.05), with Group 1 (checklist-oriented) having higher checklist scores but lower global scores than did Group 2 (oriented to global ratings). In addition, Group 1 had higher interstation alpha coefficients than did Group 2 for both global scores (0.74 versus 0.63) and checklist scores (0.63 versus 0.40). CONCLUSIONS: The interaction effect on total exam scores suggests that students adapt their behaviors to the system of evaluation. However, the lower reliability coefficients for both forms found in the process-oriented global-rating group suggest that an individual's capacity to adapt to the system of global rating forms is relatively station-specific, possibly depending on his or her expertise in the domain represented in each station.     

A comparison of physician examiners', standardized patients', and communication experts' ratings of international medical graduates' English proficiency.

PURPOSE: To assess the quality of ratings of interviewing skills and oral English proficiency provided on a clinical skills OSCE by physician examiners, standardized patients (SPs), and communication skills experts. METHOD: In 1998, 73 candidates to the Ontario International Medical Graduate (OIMG) Program completed a 29-station OSCE-type clinical skills selection examination. Physician examiners, SPs, and communication skills experts assessed components of oral English proficiency and interview performance. Based on these results, the frequency and generalizability of English-language flags, physician examiners' indications that spoken English skills were bad enough to significantly impede communication with patients; the reliability of the OIMG's Interview and Oral Performance Scales and generalizability of overall interview and oral performance ratings; and comparisons of repeated assessments by experts were calculated. Principal-components analysis was applied to the panels' ratings to determine a more economical expression of the language proficiency and interview communication skills results. RESULTS: The mean number of English-language flags per candidate was 2.1, the median was 1.0, and Cronbach's alpha of the ratings was 0.63. Means, SDs, and alphas of the physician examiners' and SPs' ratings of the interview performance scale were 9.15/10, 0.43, 0.36, and 9.30/10, 0. 56, 0.50, respectively. Corresponding values for overall interview performance ratings were 3.08/4, 0.30, 0.33, and 3.34/4, 0.32, 0.47. Means, SDs, and alphas of the physician examiners' and SPs' ratings of the oral performance scale were 8.54/10, 0.74, 0.78, and 8.74/10, 1.00, 0.76. Corresponding values for overall ratings of oral performance were 3.85/5, 0.51, 0.68, and 4.08/5, 0.60, 0.68. For the two experts' ratings of two contiguous five-minute interview stations, internal consistencies were 0.88 and 0.78. For the two experts' ratings of standardized ten-minute interviews, internal consistencies were 0.81 and 0.92. Correlations between the mean values of the experts' ratings of the ten- and five-minute stations were 0.45 and 0.51. Three factors emerged from the PCA, language proficiency, physician examiners' ratings of interview proficiency, and SPs' ratings of interview proficiency. CONCLUSIONS: Consistency between the physician examiners' and SPs' ratings of English proficiency was observed; less agreement was observed in their ratings of interviewing skills, and little agreement was observed between the experts' ratings. Communication skills results may be validly expressed by three measures: one overall global rating of language proficiency provided by physician examiners or SPs, and overall global ratings of interview proficiency provided separately by physician examiners and SPs.     

Can we rely on simulated patients’ satisfaction with their consultation for assessing medical students’ communication skills? A cross-sectional study

Background

In medical education, teaching methods offering intensive practice without high utilization of faculty resources are needed. We investigated whether simulated patients’ (SPs’) satisfaction with a consultation could predict professional observers’ assessment of young doctors’ communication skills.

Methods

This was a comparative cross-sectional study of 62 videotaped consultations in a general practice setting with young doctors who were finishing their internship. The SPs played a female patient who had observed blood when using the toilet, which had prompted a fear of cancer. Immediately afterwards, the SP rated her level of satisfaction with the consultation, and the scores were dichotomized into satisfaction or dissatisfaction. Professional observers viewed the videotapes and assessed the doctors’ communication skills using the Arizona Communication Interview Rating Scale (ACIR). Their ratings of communication skills were dichotomized into acceptable versus unacceptable levels of competence.

Results

The SPs’ satisfaction showed a predictive power of 0.74 for the observers’ assessment of the young doctors and whether they reached an acceptable level of communication skills. The SPs’ dissatisfaction had a predictive power of 0.71 for the observers’ assessment of an unacceptable communication level. The two assessment methods differed in 26 % of the consultations. When SPs felt relief about their cancer concern after the consultation, they assessed the doctors’ skills as satisfactory independent of the observers’ assessment.

Conclusions

Accordance between the dichotomized SPs’ satisfaction score and communication skills assessed by observers (using the ACIR) was in the acceptable range.These findings suggest that SPs’ satisfaction scores may provide a reliable source for assessing communication skills in educational programs for medical trainees (students and young doctors).Awareness of the patient’s concerns seems to be of vital importance to patient satisfaction.

     

Peer-assessment of medical communication skills: The impact of students’ personality,academic and social reputation on behavioural assessment

Objective

Peer-assessment of communication skills may contribute to mastery of assessment criteria. When students develop the capacity to judge their peers’ performance, they might improve their capacity to examine their own clinical performance. In this study peer-assessment ratings are compared to teacher-assessment ratings. The aim of this paper is to explore the impact of personality and social reputation as source of bias in assessment of communication skills.

Methods

Second year students were trained and assessed history taking communication skills. Peers rated the students’ personality and academic and social reputation.

Results

Peer-assessment ratings were significantly correlated with teacher-ratings in a summative assessment of medical communication. Peers did not provide negative ratings on final scales but did provide negative ratings on subcategories. Peer- and teacher-assessments were both related to the students’ personality and academic reputation.

Conclusion

Peer-assessment cannot replace teacher-assessment if the assessment should result in high-stake decisions about students. Our data do not confirm the hypothesis that peers are overly biased by personality and reputation characteristics in peer-assessment of performance.

Practice implications

Early introduction of peer-assessment in medical education would facilitate early acceptance of this mode of evaluation and would promote early on the habit of critical evaluation of professional clinical performance and acceptance of being evaluated critically by peers.     

OSCE checklists do not capture increasing levels of expertise.

PURPOSE: To evaluate the effectiveness of binary content checklists in measuring increasing levels of clinical competence. METHOD: Fourteen clinical clerks, 14 family practice residents, and 14 family physicians participated in two 15-minute standardized patient interviews. An examiner rated each participant's performance using a binary content checklist and a global process rating. The participants provided a diagnosis two minutes into and at the end of the interview. RESULTS: On global scales, the experienced clinicians scored significantly better than did the residents and clerks, but on checklists, the experienced clinicians scored significantly worse than did the residents and clerks. Diagnostic accuracy increased for all groups between the two-minute and 15-minute marks without significant differences between the groups. CONCLUSION: These findings are consistent with the hypothesis that binary checklists may not be valid measures of increasing clinical competence.     

The impact of expert- and peer feedback on communication skills of undergraduate dental students – a single-blinded,randomized, controlled clinical trial

ObjectiveTo evaluate the effect of peer- and expert feedback on communication skills of undergraduate dental students.MethodsAll students of the first clinical treatment course (n = 46) were randomly assigned into two groups. For three times a medical-dental interview/consultation of each student with a real patient was videotaped. After every consultation the videos were assessed either by a person experienced in communication (expert group) or by a fellow student (peer group), giving the students feedback regarding their chairside performed communication skills. Before and after the feedback-interventions all students conducted an interview with simulated patients, which was rated using a validated global rating and analyzed statistically.ResultsGlobal ratings mean scores after feedback-intervention were significantly improved (p < 0.05). Thereby, no significant differences in the overall assessment could be observed between expert and peer feedback (p > 0.05).ConclusionDuring this study students improved their communication skills in dentist-patient interactions. The communication experience of the feedback provider seems not to have any impact on the communication skills in undergraduate dental students.Practice implicationsThe clinical courses in dentistry offer the opportunity to implement peer-feedback interventions in real treatment situation as part of communication training to longitudinally improve communication skills.     

Social skills deficits and self-evaluation among depressed and nondepressed psychiatric inpatients

Depressed psychiatric inpatients, nondepressed psychiatric inpatients, and nonpsychiatric controls role-played responses to 28 standardized interpersonal situations. Judges blindly rated these responses on overall social skill and component measures, and subjects rated their own social skills. Judges rated depressed and nondepressed psychiatric patients as having significant problems in social skills compared to normals. No differences were found between the two patient groups in judges' ratings of social skill. Depressed patients rated their own recent interpersonal behavior and optimal social skills significantly lower than did subjects in the other groups. Results suggest that social skills deficits are not specific to depression and that depressives and other psychiatric groups may differ primarily in their self-appraisals of social competence.     

The validity of lecturer ratings by students and trained observers.

This study sought to determine to what degree student ratings of specific lecturer characteristics relate to trained observer ratings of such characteristics and to identify the distinguishing delivery characteristics of highly rated lecturers. The 15 lowest-rated lecturers and the 15 highest-rated lecturers, based on the mean ratings of students from two consecutive years (1982 and 1983) in a large multi-instructor course, served as the target group. Blinded non-student raters observed the lectures in 1984 (two per lecture) and completed quantitative and qualitative forms. For all six subscores from the quantitative form, statistically significant differences between the lecturers given the highest and lowest ratings were obtained. It is concluded that the students' ratings were stable across the three years; the independent observers discriminated between the lecturers the students rated the highest and the lowest; voice presentation characteristics discriminated the most effectively; and nonmedical non-student observers are potentially a useful source of information regarding faculty teaching skills.