Using Standardized Patients to Assess the Interpersonal Skills of Physicians: Six Years' Experience With a High-Stakes Certification Examination

Communication and interpersonal skills are essential elements of a physician's clinical expertise. Since 1998, the interpersonal competencies of over 37,000 internationally-trained physicians have been assessed as part of the Educational Commission for Foreign Medical Graduates (ECFMG®) Clinical Skills Assessment (CSA®). Standardized patients (SPs) provided ratings of interpersonal skills along 4 dimensions: skills in interviewing and collecting information; skills in counseling and delivering information; rapport; and personal manner. The content of the rating scale, the development and implementation of training materials and procedures, and the psychometric characteristics of the measures are described. Data from over 400,000 simulated patient encounters were analyzed. Correlations with other measures supported the construct validity of the assessment. A generalizability study showed that the ratings were reproducible over encounters. Analysis of individual SP ratings indicated that they were consistent in their application of the scoring rubric. Overall, the findings indicate that SPs, with proper training and a benchmarked scoring rubric, can provide accurate and defensible ratings of physicians' interpersonal skills. These results may also generalize to other clinical skills assessments, or other evaluations that employ raters to judge communication abilities.     

Evaluating the spoken English proficiency of graduates of foreign medical schools

PURPOSE: The purpose of this study was to gather additional evidence for the validity and reliability of spoken English proficiency ratings provided by trained standardized patients (SPs) in high-stakes clinical skills examination. METHOD: Over 2500 candidates who took the Educational Commission for Foreign Medical Graduates' (ECFMG) Clinical Skills Assessment (CSA) were studied. The CSA consists of 10 or 11 timed clinical encounters. Standardized patients evaluate spoken English proficiency and interpersonal skills in every encounter. Generalizability theory was used to estimate the consistency of spoken English ratings. Validity coefficients were calculated by correlating summary English ratings with CSA scores and other external criterion measures. Mean spoken English ratings were also compared by various candidate background variables. RESULTS: The reliability of the spoken English ratings, based on 10 independent evaluations, was high. The magnitudes of the associated variance components indicated that the evaluation of a candidate's spoken English proficiency is unlikely to be affected by the choice of cases or SPs used in a given assessment. Proficiency in spoken English was related to native language (English versus other) and scores from the Test of English as a Foreign Language (TOEFL). DISCUSSION: The pattern of the relationships, both within assessment components and with external criterion measures, suggests that valid measures of spoken English proficiency are obtained. This result, combined with the high reproducibility of the ratings over encounters and SPs, supports the use of trained SPs to measure spoken English skills in a simulated medical environment.     

Using a standardised patient assessment to measure professional attributes

INTRODUCTION: Professionalism is an important topic in medical education today. While much work has focused on defining professionalism and teaching medical students the appropriate interpersonal behaviours, relatively little research has looked at meaningful ways of assessing the relevant attributes. METHOD: The Educational Commission for Foreign Medical Graduates (ECFMG) clinical skills assessment (CSA) uses standardised patients (SPs) to evaluate the readiness of graduates of international medical schools to enter accredited graduate training programmes in the USA. Doctor interpersonal skills, including professional qualities such as rapport, are evaluated as part of the CSA. Attentiveness, attitude and empathy, all facets of professional behaviour, are specifically targeted as part of the assessment. RESULTS: To date, over 35 000 candidates have been assessed, encompassing more than 370 000 individual SP encounters. Based on a 1-year cohort of examinees, the reliability of the individual professionalism-related component scores ranged from 0.61 to 0.70. Doctors who had graduated from medical school more recently, or were younger, generally obtained higher ratings. Professional qualities were only marginally related to measures of basic science and clinical science proficiency. Female candidates were rated significantly higher than male candidates on all dimensions. CONCLUSIONS: While some professional behaviours are probably best measured using formats such as surveys, self-assessment and critical incident techniques, certain aspects of the domain can be reliably and validly measured in SP examinations.     

Reliability of consultation skills assessments using standardised versus real patients

Medical Education 2011; 45 : 578–584 Objectives Training in and assessment of consultation skills are high on the agenda of vocational training institutes for postgraduate training. There is a need to establish valid and reliable instruments to assess consultation skills in authentic settings. We investigated the number of assessors and observations needed to achieve reliable assessments of the consultation skills of general practice trainees (GPTs) using a communication instrument (MAAS‐Global) and either standardised patient (SP) encounters or videotaped real patient (RP) encounters. Methods Eight teachers at the Vrije Universiteit (VU) University Medical Centre in Amsterdam attended a training course on the use of the MAAS‐Global instrument, which they subsequently used to assess the consultation skills of 53 GPTs in 176 videotaped consultations (102 with SPs, 74 with RPs). All consultations were randomly allocated and assessed by two teachers independently. The reliability of the ratings was estimated using generalisability theory. Results It was easier to obtain acceptable reliability using RP consultations than SP consultations. Two assessors and five consultations were required to achieve minimal reliability (generalisability coefficient 0.7) with RPs, whereas three assessors and 30 consultations were needed to achieve minimal reliability with SPs. Conclusions Inter‐observer and context variability in the assessment of the consultation skills of GPTs remains high. To achieve acceptable levels of reliability, large samples of observations are required in both formats, but, interestingly, RP encounters require a smaller sample than SP encounters.     

Evaluating the spoken English proficiency of international medical graduates: detecting threats to the validity of standardised patient ratings

PURPOSE: To investigate potential threats to the validity of the spoken English proficiency ratings provided by standardised patients (SPs) in high-stakes clinical skills examinations. METHOD: Spoken English ratings from 43 327 patient encounters were studied. These involved over 5000 candidates, 40% of whom were female and 33% of whom self-reported English to be their native language. Over 100 SPs were involved in the study, 51% of whom were female and 90% of whom were native English speakers. Possible performance differences in English ratings were studied as a function of candidate and SP gender, and as a function of candidate and SP native language (English versus all other languages). RESULTS: No significant candidate by SP gender effect was detected. There were no meaningful differences in mean English ratings as a function of SP or candidate gender. Likewise, English ratings did not vary as a function of either candidate or SP native language. While candidate mean English ratings were not associated with the native language of the SP, native English-speaking candidates did achieve significantly higher ratings. DISCUSSION: The lack of significant interaction between candidate and SP gender, and candidate and SP native language, suggests that the SPs provided unbiased English ratings. These results, combined with the expected higher English ratings given to candidates with English-speaking backgrounds, provides additional evidence to support the validity and fairness of spoken English proficiency ratings provided by standardised patients.     

Rater errors in a clinical skills assessment of medical students

The authors used a many-faceted Rasch measurement model to analyze rating data from a clinical skills assessment of 173 fourth-year medical students to investigate four types of rater errors: leniency, inconsistency, the halo effect, and restriction of range. Students performed six clinical tasks with 6 standardized patients (SPs) selected from a pool of 17 SPs. SPs rated the performance of each student in six skills: history taking, physical examination, interpersonal skills, communication technique, counseling skills, and physical examination etiquette. SPs showed statistically significant differences in their rating severity, indicating rater leniency error. Four SPs exhibited rating inconsistency. Four SPs restricted their ratings in high categories. Only 1 SP exhibited a halo effect. Administrators of objective structured clinical examinations should be vigilant for various types of rater errors and attempt to reduce or eliminate those errors to improve the validity of inferences based on objective structured clinical examination scores.     

Impact of a new curriculum on medical students' interpersonal and interviewing skills

OBJECTIVES: In 1998 we reported on the rise and fall of medical student communication skills during the 4 years of medical school. Since then, the University of Connecticut School of Medicine has completed a major curriculum renewal project with an emphasis on early clinical work, lifelong learning and more ambulatory training. The goals of this study were to compare students' interviewing and interpersonal skills in standardised patient (SP) assessments in the old and new curricula and to assess the success of the new curriculum in preventing a decline in student skills in this domain. METHODS: The clinical skills of 202 students were measured longitudinally during encounters with SPs in each of their 4 years of medical school. Students in this study and the earlier study were evaluated using the Arizona Clinical Interviewing Rating (ACIR) Scale. RESULTS: Compared with students from the previous curriculum, students on the new curriculum in this study showed an improvement in ACIR scores. Year 1 mean ACIR scores (1 = poor to 5 = excellent) were, respectively, 3.6 for the old curriculum cohort and 4.0 for the new curriculum group. In Year 4 the mean score for the old curriculum cohort was 3.7 and that for the new curriculum group was 3.8. Students on the new curriculum still showed a decline in ACIR scores from Years 1 to 4, but it was not as severe a decline as it had been previously. CONCLUSIONS: Pre-clinical medical students perform better on measures of interpersonal communication than their clinical counterparts. The students who participated in the new curriculum demonstrated an earlier acquisition of and a less steep decline in interviewing and interpersonal skills during the course of medical school.     

Global Ratings of Student Performance in a Standardized Patient Examination: Is the Whole More than the Sum of the Parts?

Standardized patient examinations (SPE)are widely used in medical education to assess skillsthat cannot be measured with written examinations.Trained actors termed standardized patients (SPs) areused to simulate patients with specific medicalproblems. SPs typically use behaviorally specificchecklists and rating scales to evaluate examinees. This study explored the use of faculty and SP globalratings of students' clinical and interpersonal skillsin an SPE. The reliability of global ratings wasfound to on par with more specific behaviorallyanchored ratings. Global ratings were also found tobe predictive of written tests of clinical knowledgeand ratings of actual clinical performance aftercontrolling for behaviorally anchored ratings. Facultyglobal ratings were more reliable and more predictiveof other performance than SP global ratings. Theseresults suggest global ratings by faculty observersand possibly SPs can provide unique and usefulinformation in these performance-based examinations. This revised version was published online in June 2006 with corrections to the Cover Date.     

Standardized patients: a promising tool for health education and health promotion

Standardized patients (SPs) are trained actors who are used to engage health care providers in various types of clinical encounters for the purposes of training and evaluation. In the past, SPs have most often been a tool for training clinicians in "traditional" medical skills such as the taking of medical histories or the conduct of physical exams. More recently, however, SPs have been increasingly used to assist in the development and assessment of psychosocial skills, including those related to patient-centered communication, relationship building, and motivational interviewing. Given this shift, it is argued that the time is ripe for exploring ways that health educators can adopt SP methodologies, whether for providing continuing education for individual providers or evaluating organizations or programs. This article introduces ways of using SPs and discusses strengths and challenges related to various approaches.     

Investigation of standardized patient ratings of humanistic competence on a medical licensure examination using Many-Facet Rasch Measurement and generalizability theory

Humanistic doctor–patient interaction has been measured for eight years using the Global Patient Assessment (GPA) tool in the national osteopathic clinical skills medical licensure examination. Standardized patients (SPs) apply the GPA tool to rate examinees’ competence on doctor–patient communication, interpersonal skills, and professionalism. Many-Facet Rasch Measurement was employed to evaluate the overall functioning of the GPA rating scale and to estimate measurement errors associated with characteristics of SP raters and medical case presentations. Generalizability theory was applied to investigate variance components corresponding to each facet of interest. For the 2010–2011 testing cycle, 50,090 SP ratings were analyzed. Although SP raters varied in leniency/stringency of rating, SPs differentiated the six GPA aspects in difficulty, and utilized a reasonable range of the 9-point scale. Reliability indices resulted in sufficient examinee separation, 0.94, from the Rasch model and sufficient dependability from the generalizability analysis for raw scores, 0.83, and transformed Rasch scores, 0.97. Results indicate that medical students’ humanistic competence can be reliably measured through the GPA tool in the simulated environment. These measurement models supplement other means of observation and quality control with valuable information about the psychometric quality of SP ratings of humanistic competence.     

Communication Skills Training and Patients' Satisfaction

Effective physician-patient encounters require the doctor to have consulting skills that facilitate communication flow. When adequate communication does not occur, patients express dissatisfaction with their medical interactions. Many medical students show interview behaviors that may not contribute to patients' satisfaction. Poor communication skills reduce the reliability of elicited medical information and lead to reduced satisfaction for both patients and students. In the present study, a communication skills training course was evaluated using ratings of students' videotaped history-taking interviews with patients and patients' satisfaction ratings. Trained students showed significantly improved consultation skills and techniques compared with a group of control students who displayed few changes in behavior over the course of the study. Satisfaction ratings given by patients of students improved significantly after training, whereas ratings given by patients of control group students decreased over the same period.     

Are Interpersonal Skills Ratings Influenced by Gender in A Clinical Skills Assessment Using Standardized Patients?

Purpose: The purpose of this study was to explore possible performance differences in interpersonal skills (IPS) ratings as a function of candidate and standardized patient (SP) gender. Methodology: The IPS scores and SP characteristics for 79,999 patient encounters were studied. This included 18,325 (20.36%) female candidate to female SP, 26,872 (29.86%) male candidate to female SP, 18,281 (20.31%) female candidate to male SP, and 16,521 (29.47%) male candidate to male SP interactions. Results: The analysis did not reveal a significant candidate gender by SP gender effect. There were no meaningful differences in IPS scores as a function of SP or candidate gender. Conclusions: The non-significant interaction between SP gender and candidate gender provides some evidence that male and female candidates are being assessed equivalently by male and female SPs. This result, combined with the extremely weak relationship between gender (candidate or SP) and IPS ratings, provides additional support for the fairness and defensibility of the IPS measures. This revised version was published online in June 2006 with corrections to the Cover Date.     

Adjusting medical school admission: assessing interpersonal skills using situational judgement tests

Context Today’s formal medical school admission systems often include only cognitively oriented tests, although most medical school curricula emphasise both cognitive and non‐cognitive factors. Situational judgement tests (SJTs) may represent an innovative approach to the formal measurement of interpersonal skills in large groups of candidates in medical school admission processes. This study examined the validity of interpersonal video‐based SJTs in relation to a variety of outcome measures. Methods This study used a longitudinal and multiple‐cohort design to examine anonymised medical school admissions and medical education data. It focused on data for the Flemish medical school admission examination between 1999 and 2002. Participants were 5444 candidates taking the medical school admission examination. Outcome measures were first‐year grade point average (GPA), GPA in interpersonal communication courses, GPA in non‐interpersonal courses, Bachelor’s degree GPA, Master’s degree GPA and final‐year GPA (after 7 years). For students pursuing careers in general practice, additional outcome measures (9 years after sitting examinations) included supervisor ratings and the results of an interpersonal objective structured clinical examination (OSCE), a general practice knowledge test and a case‐based interview. Results Interpersonal skills assessment carried out using SJTs had significant added value over cognitive tests for predicting interpersonal GPA throughout the curriculum, doctor performance, and performance on an OSCE and in a case‐based interview. For the other outcomes, cognitive tests emerged as the better predictors. Females significantly outperformed males on the SJT (d = ? 0.26). The interpersonal SJT was perceived as significantly more job‐related than the cognitive tests (d = 0.55). Conclusions Video‐based SJTs as measures of procedural knowledge about interpersonal behaviour show promise as complements to cognitive examination components. The interpersonal skills training received during medical education does not negate the selection of students on the basis of interpersonal skills. Future research is needed to examine the use of SJTs in other cultures and student populations.     

Using simulated patients to develop doctors' skills in facilitating behaviour change: addressing childhood obesity

Medical Education 2010: 44 : 706–715 Objectives This study aimed to describe the application, feasibility and outcomes of using simulated patients (SPs) to increase the skills of general practitioners (GPs) delivering a behavioural intervention to reduce childhood overweight and mild obesity. Methods Five female actors were trained as SPs. A total of 67 GPs from 46 general practices in Melbourne, Victoria, Australia, conducted two simulated consultation visits regarding healthy lifestyle family behaviour change, during which they practised their skills and received formative feedback. The GPs and SPs rated GP performance immediately after each consultation. Subsequently, 139 parents of overweight or obese 5–9‐year‐old children rated GP performance during real‐life consultations. Other measures included child body mass index (BMI) Z‐scores (at baseline and at a 9‐month follow‐up) and GP‐reported levels of comfort and competence and the perceived value of SP visits. Results Simulated patient ratings, but not GP self‐ratings, of GP performance predicted both parent ratings of real‐life consultations (Spearman’s rho 0.39 for correlation with SP rating at Visit 1) and subsequent reductions in BMI Z‐scores between baseline and follow‐up (Visit 1, rho ? 0.45; Visit 2, rho ? 0.46). GP levels of comfort and competence were maintained during and after the SP visits. A total of 95% of GPs rated simulated consultations as useful, although only 18% said they would pay for them. Conclusions Simulated patient assessment may predict real patient feedback and clinical outcomes, helping to identify doctors who require further training in behaviour change techniques. Randomised controlled trials may establish whether SPs actually raise skills or improve outcomes.     

The effect of examinee and patient ethnicity in clinical-skills assessment with standardized patients

Context: Ethnicity has been a continuing concern for the valid assessment of clinical performance with standardized patients (SPs). The concern is that examinee ethnicity and SP ethnicity might interact, such that examinees might score higher in encounters with SPs of the same ethnicity. Objective: To test for an interaction of examinee ethnicity and SP ethnicity on clinical performance in an SP examination. Main Outcome Measures: History-taking andphysical-examination scores and interpersonal-and communication-skills scores, both based on checklists completed by SPs. Poststation scores for answers to case-related questions concerning pathophysiology, diagnosis, test selection, and test interpretation. Setting And Participants: Two graduating classes of over 1,000 fourth-year medical students each in the New York City Consortium were tested on the SP assessment administered at The Morchand Centerof Mount Sinai School of Medicine. Design And Analysis: The primary analyses were two-way (2 × 2) analyses, to test the main and interaction effects of examinee ethnicity and SP ethnicity. Effect-size measures (standardized mean differences, d) were computed to provide a sharper picture of the effects. Results: Of the 24 interaction analyses, only three were statistically significant (notsignificantly more than expected by chance) and the results were mixed: one analysis showed better examinee performance in encounters with SPs of thesame ethnic background and the other two showed the opposite. For all 24 interactions, significant or not, the results showed weak effects and no consistent pattern. White examinees scored on average 0.12 standard deviations above black examinees in encounters with white SPs, and 0.11 standard deviations higher in encounters with black SPs. Conclusions: These initial results are encouraging and should dispel some of the concern about ethnicity in SP assessment, at least about the operation of an examinee-by-SP-ethnicity interaction that would pose a serious threat to the validity of the examination scores.This revised version was published online in October 2005 with corrections to the Cover Date.     

An evaluation of the impact of training Honduran health care providers in interpersonal communication.

OBJECTIVE: To evaluate the impact of interpersonal communication (IPC) training on practice and patient satisfaction and to determine the acceptability of this training to providers in a developing country. DESIGN: The study used a pre-post design with treatment and control groups. Data collection methods included interaction analysis of audio-taped clinical encounters, patient exit interviews, and a self-administered questionnaire for health providers. STUDY PARTICIPANTS: Interaction analysis was based on an experimental group of 24 doctors and a control group of eight with multiple observations for each provider). Exit interviews were carried out with 220 pre-test patients and 218 post-test patients. All 87 health providers who received training responded to the self-administered questionnaire. INTERVENTION: A brief in-service training programme on interpersonal communications was presented in three half-day sessions; these focused on overall socio-emotional communication, problem solving skills and counselling. MAIN OUTCOME MEASURES AND RESULTS: The IPC intervention was associated with more communication by trained providers (mean scores of 136.6 versus 94.4; P = 0.001), more positive talk (15.93 versus 7.99; P = 0.001), less negative talk (0.11 versus 0.59; P = 0.018), more emotional talk (15.7 versus 5.5; P = 0.021), and more medical counselling (17.3 versus 11.3; P = 0.026). Patients responded by communicating more (mean scores of 113.8 versus 79.6; P = 0.011) and disclosing more medical information (54.7 versus 41.7; P = 0.002). Patient satisfaction ratings were higher for providers who had received the training and providers reported training to be relevant and useful. CONCLUSIONS: Further validation of IPC skills and simplification of assessment methods are needed if IPC is to be an area for routine monitoring and quality improvement.     

Patient involvement in teaching and assessing intimate examination skills: a systematic review

Medical Education 2010: 44 : 347–357 Objectives There are recognised difficulties in teaching and assessing intimate examination skills that relate to the sensitive nature of the various examinations and the anxiety faced by novice learners. This systematic review provides a summary of the evidence for the involvement of real patients (RPs) and simulated patients (SPs) in the training of health care professionals in intimate examination skills. Methods For the review, ‘intimate examinations’ included pelvic, breast, testicular and rectal examinations. Major databases were searched from the start of the database to December 2008. The synthesis of findings is integrated by narrative structured to address the main research questions, which sought to establish: the objectives of programmes involving RPs and SPs as teachers of intimate examination skills; reasons why SPs have been involved in this training; the evidence for the effectiveness of such training programmes; the evidence for measures of anxiety in students learning how to perform intimate examinations; how well issues of sexuality are addressed in the literature; any reported negative effects of involvement in teaching on the patients, and suggestions for practical strategies for involving patients in the teaching of intimate examination skills. Results A total of 65 articles were included in the review. Involving patients in teaching intimate examination skills offers advantages over traditional methods of teaching. Objective evidence for the effectiveness of this method is demonstrated through improved clinical performance, reduced anxiety and positive evaluation of programmes. Practical strategies for implementing such programmes are also reported. Conclusions There is evidence of a short‐term positive impact of patient involvement in the teaching and assessment of intimate examination skills; however, evidence of longer‐term impact is still limited. The influences of sexuality and anxiety related to such examinations are explored to some extent, but the psychological impact on learners and patients is not well addressed.     

Consulting skills training and medical students' interviewing efficiency

In more traditional medical education, medical students took a patient's medical history by asking a series of sequenced, routine questions, covering presenting medical problem(s); medical history; social and personal history; systems review; and physical examination. Following this process, the student then attempted to derive the patient's medical problems. This inductive problem-solving paradigm may not assist students to prepare for their future interviewing needs, given doctors use a hypothetico-deductive, problem-solving approach when interviewing patients and numerous researchers have developed specialized communication skills training programmes designed to enhance students' interviewing skills. Students given specific consulting skills training have tended to show significantly greater interpersonal effectiveness and improved interview behaviours compared with students who experience traditional patient clerking training. These improvements in interviewing tend to persist over the period of students' medical training. The aim of the present study was to determine whether specialized communication skills training helped students elicit greater quantity and quality of information from patients and if so, whether such information assisted students in improving their diagnostic skills. Videotaped history-taking interviews conducted by students trained in communication skills and untrained (control) students were rated for their interview efficiency. A comparison of ratings given by experimentally naive, independent observers revealed that trained students were more efficient, but took no longer than their control group counterparts to elicit fuller, more relevant information. However, the student groups did not differ in the accuracy or scope of their medical diagnoses. It is argued that students' lack of medical knowledge in this early phase of their clinical training militated against their being able to use their interviewing competence to derive more potentially accurate medical diagnoses.     

Real-patient evaluation of communication skills teaching for GP registrars

BACKGROUND: Five thousand eight hundred and eighty-five patient- completed questionnaires were used to evaluate the effectiveness of an interpersonal skills module designed for a vocational training programme for GPs. OBJECTIVES: It was anticipated that patient-based assessments would detect a significant improvement in the interpersonal skills of GP Registrars who undertook the module. METHOD: A quasi- experimental design using an intervention and control group (comprising 68 GP Registrars) was used to monitor the outcomes of the interpersonal skills module. RESULTS: Patient ratings of interpersonal skills were significantly higher for those GP Registrars who participated in the interpersonal skills module. CONCLUSIONS: Patient-based assessments are a useful evaluation method for assessing the quality of the doctor- patient relationship.      

The Use of Standardized Patients To Teach and Test Interpersonal and Communication Skills with Students in Speech-Language Pathology

The purpose of this study was to investigate the use of standardized patients (SPs) with aphasia to teach interpersonal and communication skills to new graduate student-clinicians in Speech-Language Pathology, and to test those skills via serial Objective Structured Clinical Examinations (OSCEs). This study had three phases: (1) clinical teaching using SPs, (2) mid-term evaluation via a single case OSCE, and (3) end-of-term evaluation via a multi-case OSCE. These phases were integrated with classroom teaching and testing components over a 16-week academic semester. Eighteen students participated while concurrently enrolled in a course on diagnosis and management of aphasia taught by the first author. One half of the class received initial instruction via a combination of didactic lecture and standardized patient interaction, while the other half of the class was taught initially via didactic lecture only. On OSCEI, there was no statistically significant difference between the two groups in their interpersonal and communication skills, with the class as a whole demonstrating less than acceptable competency. After targeting these skills in all students via additional class lectures, there was a statistically and clinically significant improvement in their competency in this area on OSCE II. Student feedback was positive. This revised version was published online in September 2006 with corrections to the Cover Date.