Reliability of narrative assessment data on communication skills in a summative OSCE

ObjectiveTo quantitatively estimate the reliability of narrative assessment data regarding student communication skills obtained from a summative OSCE and to compare reliability to that of communication scores obtained from direct observation.MethodsNarrative comments and communication scores (scale 1–5) were obtained for 14 graduating pharmacy students across 6 summative OSCE stations with 2 assessors per station who directly observed student performance. Two assessors who had not observed the OSCE reviewed narratives and independently scored communication skills according to the same 5-point scale. Generalizability theory was used to estimate reliability. Correlation was used to evaluate the relationship between scores from each assessment method.ResultsA total of 168 narratives and communication scores were obtained. The G-coefficients were 0.571 for scores provided by assessors present during the OSCE and 0.612 for scores from assessors who provided scores based on narratives only. Correlation between the two sets of scores was 0.5.ConclusionReliability of communication scores is not dependent on whether assessors directly observe student performance or assess written narratives, yet both conditions appear to measure communication skills somewhat differently.Practice implicationsNarratives may be useful for summative decision-making and help overcome the current limitations of using solely quantitative scores.     

Assessment of patient communication skills during OSCE: examining effectiveness of a training program in minimizing inter-grader variability

Objective

To assess effectiveness of a training program in reducing inter-grader variability in grading communication skills during an objective structured clinical exam (OSCE).

Methods

Global communication (GC) skills are assessed by standardized participants (SP) and faculty during each OSCE using a 6 item rubric. Despite criteria delineated in the GC rubric, inter-grader variability was observed. During 2008-2009 academic year, a training program was implemented before each OSCE to achieve more consistent interpretation and grading in GC skills. GC grades between SP and faculty for 2nd and 3rd level student OSCEs during 2008-2009 (post-training) were compared to 2007-2008 (pre-training). Data was analyzed using repeated measures ANOVA.

Results

274 and 281 students participated in OSCEs during 2007-2008 and 2008-2009 academic years, respectively. Training significantly (P < .001) decreased grader variability between SPs and faculty. There was a greater mean difference between faculty vs. SP before training (faculty 14.68, SP 15.87) compared to after training (faculty 13.51, SP 13.78). Mean GC scores for both faculty and SPs also decreased significantly after training.

Conclusion and practice implications

A training program may be necessary to reduce inter-rater variability in assessment of OSCE communication skills if it is to be truly helpful to student pharmacists learning to counsel patients.     

The socioformative rubrics in the OSCE to assess the level of achievement of the competencies comprising the profile of the physician graduate

IntroductionOne of the challenges for teachers in the health area is competency assessment. The OSCE is designed to assess the performance of students in specific clinical situations. The objective of this study was to assess the level of achievement of the competencies proposed in the graduates' profile by applying the OSCE with socioformative rubrics and assessing their relationship with the written final exam and the general average of the degree.MethodA cross-sectional study was carried out with graduates of the career of medicine; in which an OSCE was applied as part of its degree exam. The rubrics were the instrument that was applied to assess competencies.ResultsThe significant differences between the students who applied the OSCE in the month of January compared to May, were also observed in the results of the written exam, which indicates that the students of the 4th. OSCE developed in better way their skills during their training process. The correlational analysis highlights a significant positive correlation between the average of the OSCE and the average of the written exam of the students, which makes it clear that an adequate performance in the written exam that assesses the theoretical knowledge is related to an adequate performance in the clinical skills demonstrated in the OSCE.ConclusionThe OSCE, is an objective and reliable test, suitable as part of the professional exam of the career of medicine; with the use of the rubric, it is possible to determine the level of achievement with which the student graduates.     

Can we rely on simulated patients’ satisfaction with their consultation for assessing medical students’ communication skills? A cross-sectional study

Background

In medical education, teaching methods offering intensive practice without high utilization of faculty resources are needed. We investigated whether simulated patients’ (SPs’) satisfaction with a consultation could predict professional observers’ assessment of young doctors’ communication skills.

Methods

This was a comparative cross-sectional study of 62 videotaped consultations in a general practice setting with young doctors who were finishing their internship. The SPs played a female patient who had observed blood when using the toilet, which had prompted a fear of cancer. Immediately afterwards, the SP rated her level of satisfaction with the consultation, and the scores were dichotomized into satisfaction or dissatisfaction. Professional observers viewed the videotapes and assessed the doctors’ communication skills using the Arizona Communication Interview Rating Scale (ACIR). Their ratings of communication skills were dichotomized into acceptable versus unacceptable levels of competence.

Results

The SPs’ satisfaction showed a predictive power of 0.74 for the observers’ assessment of the young doctors and whether they reached an acceptable level of communication skills. The SPs’ dissatisfaction had a predictive power of 0.71 for the observers’ assessment of an unacceptable communication level. The two assessment methods differed in 26 % of the consultations. When SPs felt relief about their cancer concern after the consultation, they assessed the doctors’ skills as satisfactory independent of the observers’ assessment.

Conclusions

Accordance between the dichotomized SPs’ satisfaction score and communication skills assessed by observers (using the ACIR) was in the acceptable range.These findings suggest that SPs’ satisfaction scores may provide a reliable source for assessing communication skills in educational programs for medical trainees (students and young doctors).Awareness of the patient’s concerns seems to be of vital importance to patient satisfaction.

     

Assessing medical students' awareness of and sensitivity to diverse health beliefs using a standardized patient station.

PURPOSE: To assess students' performances on a health-beliefs communication OSCE station to determine whether there were differences in cultural competence based on the students' ethnic backgrounds. METHOD: A total of 71 students completed a health-beliefs communication OSCE station in which they were required to address the health beliefs and cultural concerns of a standardized patient (SP) portraying an African American woman with diabetes. The SPs rated students' performances on a ten-item interview assessment checklist. Scores on the station were standardized within SPs to adjust for differences in their use of the rating scale. A factor analysis was performed to determine conceptual constructs on the interview assessment checklist. Subscale means were computed for each student. T-tests of these subscale scores were conducted to investigate gender and ethnic differences between subgroups of students. The underrepresented minority (URM) students (five African Americans and three Mexican Americans) were compared with all other students, and the white students were compared with all others. To assess the magnitudes of the differences between subgroups, effect sizes (ES(m)) were computed for means comparisons. RESULTS: Factor analysis formed two factors: Disease Beliefs and Management, and Cultural Concerns. Two remaining items loaded on a third factor that had reliability too low to support further analysis. Meaningful differences were found in cultural sensitivity based on students' ethnic backgrounds. The URM students performed better than did all other students in addressing the patient's concerns about altering culturally-based dietary behaviors for diabetes self-care [URM students' mean standardized score (SD) = 0.42 (0.15); all others = -0.01 (0.67); ES(m) = 1.05]. White students performed better than did all other students in assessing the patient's concerns about using insulin to control her blood sugar levels [white students' mean standardized score (SD) = 0.13 (0.40); all others = -0.10 (0.64); ES(m) = 0.4]. CONCLUSION: Cultural competency deficits and differences were measurable using a health-beliefs communications station, and these differences were meaningful enough to warrant faculty discussion and research about how to ensure that students master this competency.     

The use of simulation in medical student education on the topic of breaking bad news: A systematic review

BackgroundSimulated patients (SPs) are widely used, but the most effective way of utilising them in undergraduate breaking bad news (BBN) medical education is unknown.ObjectivesTo conduct a systematic review into SP’s use in developing BBN skills in medical students.Methods14 databases searched with the terms “Medical education”, “Patient simulation”, “Bad news”. Data was systematically extracted, and thematic analysis undertaken.ResultsOf 2117 articles screened, 29 publications met the inclusion criteria. These demonstrated a variety of SP models, including actors as patients (65.5%), peers (7.0%), and cancer survivors (3.5%). with delivery at varying times in the curricula. SPs are uniformly reported as having positive impact, but there is a lack of high-quality evidence comparing the use of differing forms of training. There was some evidence that virtual SPs were as useful as in-person SPs.ConclusionsSPs allow students to practise vital BBN communication skills without risking detriment to patient care. Despite the heterogeneity of ways in which SPs have been used, the benefits of different approaches and when and how these should be delivered remains unclear.Practice implicationsFurther educational development and research is needed about the use of SPs to support undergraduate BBN communication skills development.     

The effect of physician self-disclosure on patient self-disclosure and patient perceptions of the physician

ObjectivesPhysician self-disclosure is typically seen as patient-centered communication because it creates rapport and is seen as an expression of empathy. Given that many physician behaviors affect patients differently depending on whether they are shown by a female or male physician, we set out to test whether physician self-disclosure affects patients’ intentions to self-disclose and patients’ perceptions of their physicians depending on physicians’ gender.MethodTwo hundred and forty-four participants were recruited and randomly assigned to read one of 4 vignettes as if they were the patient in the dialogue (analogue patient design). They were then asked to report how they would react to the physician and how they perceived the him or her.ResultsPhysicians who self-disclosed were perceived as more empathic than physicians who did not, regardless of physician and patient gender. Physician self-disclosure had an effect on the behavioral intentions of the analogue patients, and this was moderated by physician gender. Analogue patients indicated to be more willing to self-disclose to female than to male physicians who self-disclosed.ConclusionIt is important to consider physician gender when training physicians in patient-centered communication because the same behavior can have different effects on patients depending on whether it originates from a female or a male physician.Practical implicationsPhysicians can use self-disclosure to express empathy. When female physicians do so, they might obtain more personal information from patients, which can positively affect diagnosis and treatment.     

Using a computer simulation for teaching communication skills: A blinded multisite mixed methods randomized controlled trial

ObjectivesTo assess advanced communication skills among second-year medical students exposed either to a computer simulation (MPathic-VR) featuring virtual humans, or to a multimedia computer-based learning module, and to understand each group’s experiences and learning preferences.MethodsA single-blinded, mixed methods, randomized, multisite trial compared MPathic-VR (N = 210) to computer-based learning (N = 211). Primary outcomes: communication scores during repeat interactions with MPathic-VR’s intercultural and interprofessional communication scenarios and scores on a subsequent advanced communication skills objective structured clinical examination (OSCE). Multivariate analysis of variance was used to compare outcomes. Secondary outcomes: student attitude surveys and qualitative assessments of their experiences with MPathic-VR or computer-based learning.ResultsMPathic-VR-trained students improved their intercultural and interprofessional communication performance between their first and second interactions with each scenario. They also achieved significantly higher composite scores on the OSCE than computer-based learning-trained students. Attitudes and experiences were more positive among students trained with MPathic-VR, who valued its providing immediate feedback, teaching nonverbal communication skills, and preparing them for emotion-charged patient encounters.ConclusionsMPathic-VR was effective in training advanced communication skills and in enabling knowledge transfer into a more realistic clinical situation.Practice implicationsMPathic-VR’s virtual human simulation offers an effective and engaging means of advanced communication training.     

Exploring patient-centeredness: The relationship between self-reported empathy and patient-centered communication in medical trainees

Objective

To explore the relationships between self-reported Empathy and the patient-centered communication patterns of physician trainees.

Detecting the effects of physician training in self-care interviewing skills: Coding of standardized patient (SP) visit recordings versus SP post-visit ratings

ObjectiveTo compare how coder ratings of standardized patient (SP) visit recordings and SP ratings of the visits detect primary care physician (PCP) training in self-efficacy enhancing interviewing techniques (SEE IT).MethodsAnalyses of data from 50 PCPs who participated in a randomized controlled trial of SEE IT training, which led to increased SEE IT use during three SP visits 1–3 months post-intervention. Untrained SPs rated SEE IT use post-visit. Subsequently, three trained coders generated a consensus SEE IT rating from visit audio recordings. SPs and coders were blinded to provider study arm, and coders to SP ratings.ResultsSP and coder ratings were correlated (r = 0.62). In detecting the intervention effect, the areas under the receiver operating characteristic curve were 0.80 (95% CI 0.74–0.87) and 0.76 (95% CI 0.69–0.84) for consensus coder and SP ratings, respectively (difference 0.04, 95% CI −0.04–0.11; z = 1.04, p = 0.30).ConclusionSP ratings were not significantly different from coder ratings of SP visit recordings in detecting PCP SEE IT training.Practice implicationsIf similar findings are observed in larger studies, it would suggest a greater role for SP ratings in detecting provider interviewing skills training, given the relative simplicity, low cost, and non-intrusiveness of the approach.     

The impact of expert- and peer feedback on communication skills of undergraduate dental students – a single-blinded,randomized, controlled clinical trial

ObjectiveTo evaluate the effect of peer- and expert feedback on communication skills of undergraduate dental students.MethodsAll students of the first clinical treatment course (n = 46) were randomly assigned into two groups. For three times a medical-dental interview/consultation of each student with a real patient was videotaped. After every consultation the videos were assessed either by a person experienced in communication (expert group) or by a fellow student (peer group), giving the students feedback regarding their chairside performed communication skills. Before and after the feedback-interventions all students conducted an interview with simulated patients, which was rated using a validated global rating and analyzed statistically.ResultsGlobal ratings mean scores after feedback-intervention were significantly improved (p < 0.05). Thereby, no significant differences in the overall assessment could be observed between expert and peer feedback (p > 0.05).ConclusionDuring this study students improved their communication skills in dentist-patient interactions. The communication experience of the feedback provider seems not to have any impact on the communication skills in undergraduate dental students.Practice implicationsThe clinical courses in dentistry offer the opportunity to implement peer-feedback interventions in real treatment situation as part of communication training to longitudinally improve communication skills.     

Predictors of medical students’ ethical decision-making: A pilot study using the Theory of Interpersonal Behavior

ObjectiveTo understand medical students’ (MS) ethical decision-making using the Theory of Interpersonal Behavior (TIB).MethodsWe conducted two rounds of focus groups to develop a TIB-based questionnaire by eliciting students’ perspectives on an ethical dilemma they will encounter in a standardized patient (SP) station, in which an SP “surgeon” asked them to intubate a sedated patient whom the student knew had requested no student involvement. We administrated questionnaires to 241 third-year MS following this SP station, asking for their decisions in the SP station and if a surgeon made the same request in their clerkship. Confirmatory factor analysis (CFA) was used to test whether observed data fit the proposed TIB-based model.ResultsThe CFA provided an acceptable fit to the a priori proposed model. Fifty-five percent of students indicated they would intubate in an actual situation versus 18% in the SP station (p < 0.05). Using logistic regression, TIB domains affect and facilitating factors reported significant association with students’ decisions in both the SP and hypothesized actual situations.ConclusionsThe TIB appears to be an effective theoretical framework for explaining students’ ethical decision-making.Practice implicationsThe TIB may guide design and assessment of educational programs for professional formation.     

An innovative OSCE clinical log station: a quantitative study of its influence on Log use by medical students

Background

A Clinical Log was introduced as part of a medical student learning portfolio, aiming to develop a habit of critical reflection while learning was taking place, and provide feedback to students and the institution on learning progress. It was designed as a longitudinal self-directed structured record of student learning events, with reflection on these for personal and professional development, and actions planned or taken for learning.As incentive was needed to encourage student engagement, an innovative Clinical Log station was introduced in the OSCE, an assessment format with established acceptance at the School. This study questions: How does an OSCE Clinical Log station influence Log use by students?

Methods

The Log station was introduced into the formative, and subsequent summative, OSCEs with careful attention to student and assessor training, marking rubrics and the standard setting procedure. The scoring process sought evidence of educational use of the log, and an ability to present and reflect on key learning issues in a concise and coherent manner.

Results

Analysis of the first cohort’s Log use over the four-year course (quantified as number of patient visits entered by all students) revealed limited initial use. Usage was stimulated after introduction of the Log station early in third year, with some improvement during the subsequent year-long integrated community-based clerkship. Student reflection, quantified by the mean number of characters in the ‘reflection’ fields per entry, peaked just prior to the final OSCE (mid-Year 4). Following this, very few students continued to enter and reflect on clinical experience using the Log.

Conclusion

While the current study suggested that we can’t assume students will self-reflect unless such an activity is included in an assessment, ongoing work has focused on building learner and faculty confidence in the value of self-reflection as part of being a competent physician.

     

Eliciting patient preferences in shared decision-making (SDM): Comparing conversation analysis and SDM measurements

ObjectiveTo explore how physicians bring up patient preferences, and how it aligns with assessments of shared decision-making.MethodsQualitative conversation analysis of physicians formulating hypotheses about the patient’s treatment preference was compared with quantitative scores on SDM and ‘patient preferences’ using OPTION(5) and MAPPIN’SDM.ResultsPhysicians occasionally formulate hypotheses about patients’ preferences and then present a treatment option on the basis of that (“if you think X + we can do Y”). This practice may promote SDM in that the decisions are treated as contingent on patient preferences. However, the way these hypotheses are formulated, simultaneously constrains the patient’s freedom of choice and exerts a pressure to accept the physician’s recommendation. These opposing effects may in part explain cases where different assessment instruments yield large variations in SDM measures.ConclusionEliciting patient preferences is a complex phenomenon that can be difficult to reduce into an accurate number. Detailed analysis can shed light on how patient preferences are elicited, and its consequences for patient involvement. Comparing CA and SDM measurements can contribute to specifying communicative actions that SDM scores are based on.Practice implicationsOur findings have implications for SDM communication skills training and further development of SDM measurements.     

Learning how to break bad news from worked examples: Does the presentation format matter when hints are embedded? Results from randomised and blinded field trials

ObjectivesVideo-based worked examples enable medical students to successfully prepare for breaking-bad-news (BBN) encounters with simulated patients (SPs). This is especially true when examples include hints that signal important content. This paper investigates whether the beneficial effect of hints only applies to video-based worked examples or also text-based examples.MethodsOne-hundred-and-forty-seven fourth-year medical students attending a BBN training participated in either of two equally scaffolded, randomised field trials. Prior to encountering SPs, the students worked through an e-learning module introducing the SPIKES protocol for delivering bad news; it contained the same worked example presented to either of four groups as text or video, with or without additional hints denoting the SPIKES steps being implemented.ResultsOnly a main effect of ‘hints’ was revealed, implying that students in the hints groups delivered the news to an SP significantly more appropriately than those in the without-hints groups.ConclusionsIndependent of their presentation format, worked examples with hints best foster students’ BBN skills learning.Practice implicationsIn addition to video, text-based worked examples can effectively prepare students for BBN simulations if hints are included. This offers an affordable alternative to video examples, as text examples can be generated with less effort.     

"Teach 1, do 1 ... better": superior communication skills in senior medical students serving as standardized patient--examiners for their junior peers.

PURPOSE: To see if senior medical students who had served as standardized patients (SP) demonstrated improvement in their own interpersonal communication skills. METHOD: From George Washington School of Medicine's class of 1996, 154 fourth-year students took a clinical practice examination that used professional SPs. Within the preceding six months, 28 of these students had been SP-examiners in similar examinations for first- and second-year medical students. The professional SPs rated the fourth-year examinees using checklists that measured five dimensions of interpersonal communication skills. Four of these five dimensions were identical to those measured on the examinations for which the fourth-year students had served as SPs. Hypothesizing that the SP-experienced seniors would outscore their inexperienced classmates on those four dimensions, but not on the fifth, the authors analyzed the fourth-year students' scores. P values were computed by the F test from a two-way analysis of variance. RESULTS: As predicted, the group with prior SP experience significantly outscored their inexperienced colleagues in each of the four expected dimensions of interpersonal communication skills, with p values ranging from .000 to .023. The score differential in these dimensions ranged from 3.8 to 11.8 percentage points. As further predicted, there was no significant difference between the scores of the two groups on the fifth dimension. CONCLUSION: Compared with their inexperienced peers, senior medical students with prior SP experience consistently demonstrated superior scores when their own communication skills were tested in a similar manner. The U.S. Medical Licensing Examinations propose to incorporate SP clinical examinations; in response, medical schools will use more SP examinations in their own curricula. Such examinations are expensive when using professional standardized patients; the authors believe that an SP program using senior medical students will prove an attractive alternative. Such programs may have the added advantage of making better communicators of senior medical student teachers as well as the students they teach.     

Use of an in-house-developed, 3D-printed mannequin for emergency medicine training among medical students

IntroductionTo describe, validate, and evaluate the effectiveness of an in-house-developed mannequin versus standard commercial mannequins for emergency medicine training among undergraduate medical students.Materials and methodsFirstly, we developed a 3-in-1 model consisting of 3 modules (sampling techniques, fluid resuscitation and IV drug administration, and minor surgery). Forty participants of 2nd-year medical students were enrolled. Each student completed three scenarios through the objective structured clinical examination (OSCE) simulation assessment. The questionnaire was used to rate the mannequins' overall quality, usability, comparability, cost efficiency, and realism.ResultsAlthough students using an in-house-developed mannequin had similar OSCE scores to a standard mannequin, students rated an in-house-developed mannequin easier to use. On the other hand, most students agreed that the standard commercial mannequin was more realistic than a substitute one. The costs of the materials needed for mannequin fabrication were less than 100 USD, and it was functionally comparable.ConclusionAn in-house-developed mannequin was well accepted by students and teachers and could be used to deliver and assess clinical skills for medical students effectively at low-cost.