Slicing it thin: new methods for brief sampling analysis using RIAS-coded medical dialogue

Objective

To explore the relationship between one-minute slices and full-session interaction and the predictive validity of the slices to ratings of affect and rapport.

Methods

Third-year medical students (n = 253) were videotaped during an OSCE. All interaction was coded using the Roter Interaction Analysis System (RIAS) and samples were drawn at minutes 1, 5, and 9 and extracted from the coded database. The slices were related in multivariate analysis to full-session interaction, corrected for slice content, and correlated with affect ratings of participants and independently rated judgments of rapport.

Results

One-minute slices explained 33% of full-session variance in student interaction and 30% of variance in standardized patient interaction. Slices were significantly correlated with affective ratings of participants and independent judgments of rapport in a similar pattern as full-session interaction analysis.

Conclusions

One-minute slices of interaction can provide a meaningful degree of insight into OSCE session communication with both concurrent and predictive validity to ratings of session affect and rapport.

Practice implications

Evidence of concurrent and predictive validity further supports use of this approach as a research tool that provides an efficient means of analyzing processes of care, examining variation in communication throughout a visit and predicting visit outcomes.     

An examination of the validity of EPSCALE using factor analysis

Objective

To examine the validity and utility of the Explanation and Planning Scale (EPSCALE) instrument, a widely used scale for teaching and assessment of explanation and planning skills used by clinicians during the medical interview.

Methods

Data obtained across 4 OSCE stations during medical student final MB examinations. Exploratory factor analysis, using a single factor and two factor models (based on prior theory) and a six factor empirical model, suggested by parallel analysis.

Participants

124 medical students sitting final MB examinations at the University of Cambridge.

Results

A single factor model represented a very poor fit. A two factor model with factors labelled ‘Explanation’ and ‘Planning’ produced an improved fit, but the best was seen with a six factor model, with factors which broadly corresponded to the domains of the Calgary–Cambridge guide.

Conclusions

These factor models provide supportive evidence for the construct validity of EPSCALE.

Practice implications

EPSCALE can justifiably be used in the assessment of shared-decision making skills.     

Outcome of parent-physician communication skills training for pediatric residents

Objective

Communication skills represent an essential component of clinical competence. In the field of pediatrics, communication between physicians and patients’ parents is characterized by particular difficulties. To investigate the effects of a parent-physician communication skills training program on OSCE performance and self-efficacy in a group control design.

Methods

Parallel to their daily work in the outpatient department, intervention-group experienced clinicians in practice (n = 14) participated in a communication training with standardized parents. Control-group physicians (n = 14) did not receive any training beyond their daily work. Performance was assessed by independent video ratings of an OSCE. Both groups rated their self-efficacy prior to and following training.

Results

Regarding OSCE performance, the intervention group demonstrated superior skills in building relationships with parents (p < .024) and tended to perform better in exploring parents’ problems (p < .081). The communication training program led to significant improvement in self-efficacy with respect to the specific training objectives in the intervention group (p < .046).

Conclusion

Even in physicians with considerable experience, structured communication training with standardized parents leads to significant improvement in OSCE performance and self-efficacy.

Practise implications

Briefness and tight structure make the presented communication training program applicable even for experienced physicians in daily clinical practice.     

Representing randomness in the communication of individualized cancer risk estimates: effects on cancer risk perceptions, worry, and subjective uncertainty about risk

Objective

To test the effect of novel representations of randomness on risk perceptions, worry, and subjective uncertainty about individualized colorectal cancer risk estimates.

Methods

A web-based factorial experiment was conducted, in which 225 adults aged 40 years and older were provided with hypothetical individualized colorectal cancer risk estimates, using 5 different textual and visual representations varying in expressed randomness. Outcome measures were perceived cancer risk, cancer worry, and subjective uncertainty about cancer risk; the moderating effect of dispositional optimism was also examined.

Results

Representational format was significantly associated with subjective uncertainty about cancer risk, but not with perceived cancer risk or worry. A format using software-based animation to express randomness dynamically led to the highest subjective uncertainty, although a static visual non-random format also increased uncertainty. Dispositional optimism moderated this effect; between-format differences in uncertainty were significant only for participants with low optimism.

Conclusion

Representing randomness in individualized estimates of cancer risk increases subjective uncertainty about risk. A novel dynamic visual format produces the greatest effect, which is moderated by individual differences in optimism.

Practice implications

Novel representations of randomness may be effective in improving people's understanding of the essential uncertainty pertaining to individualized cancer risk estimates.     

Evaluating verbal and non-verbal communication skills, in an ethnogeriatric OSCE

Objective

Communication during medical interviews plays a large role in patient adherence, satisfaction with care, and health outcomes. Both verbal and non-verbal communication (NVC) skills are central to the development of rapport between patients and healthcare professionals. The purpose of this study was to assess the role of non-verbal and verbal communication skills on evaluations by standardized patients during an ethnogeriatric Objective Structured Clinical Examination (OSCE).

Methods

Interviews from 19 medical students, residents, and fellows in an ethnogeriatric OSCE were analyzed. Each interview was videotaped and evaluated on a 14 item verbal and an 8 item non-verbal communication checklist. The relationship between verbal and non-verbal communication skills on interview evaluations by standardized patients were examined using correlational analyses.

Results

Maintaining adequate facial expression (FE), using affirmative gestures (AG), and limiting both unpurposive movements (UM) and hand gestures (HG) had a significant positive effect on perception of interview quality during this OSCE. Non-verbal communication skills played a role in perception of overall interview quality as well as perception of culturally competent communication.

Conclusion

Incorporating formative and summative evaluation of both verbal and non-verbal communication skills may be a critical component of curricular innovations in ethnogeriatrics, such as the OSCE.     

Assessment of patient communication skills during OSCE: examining effectiveness of a training program in minimizing inter-grader variability

Objective

To assess effectiveness of a training program in reducing inter-grader variability in grading communication skills during an objective structured clinical exam (OSCE).

Methods

Global communication (GC) skills are assessed by standardized participants (SP) and faculty during each OSCE using a 6 item rubric. Despite criteria delineated in the GC rubric, inter-grader variability was observed. During 2008-2009 academic year, a training program was implemented before each OSCE to achieve more consistent interpretation and grading in GC skills. GC grades between SP and faculty for 2nd and 3rd level student OSCEs during 2008-2009 (post-training) were compared to 2007-2008 (pre-training). Data was analyzed using repeated measures ANOVA.

Results

274 and 281 students participated in OSCEs during 2007-2008 and 2008-2009 academic years, respectively. Training significantly (P < .001) decreased grader variability between SPs and faculty. There was a greater mean difference between faculty vs. SP before training (faculty 14.68, SP 15.87) compared to after training (faculty 13.51, SP 13.78). Mean GC scores for both faculty and SPs also decreased significantly after training.

Conclusion and practice implications

A training program may be necessary to reduce inter-rater variability in assessment of OSCE communication skills if it is to be truly helpful to student pharmacists learning to counsel patients.     

The feasibility of a multi-format Web-based assessment of physicians' communication skills

Objective

Little is known about the best approaches and format for measuring physicians’ communication skills in an online environment. This study examines the reliability and validity of scores from two Web-based communication skill assessment formats.

Methods

We created two online communication skill assessment formats: (a) MCQ (multiple-choice questions) consisting of video-based multiple-choice questions; (b) multi-format including video-based multiple-choice questions with rationales, Likert-type scales, and free text responses of what physicians would say to a patient. We randomized 100 general internists to each test format. Peer and patient ratings collected via the American Board of Internal Medicine (ABIM) served as validity sources.

Results

Seventy-seven internists completed the tests (MCQ: 38; multi-format: 39). The adjusted reliability was 0.74 for both formats. Excellent communicators, as based on their peer and patient ratings, performed slightly better on both tests than adequate communicators, though this difference was not statistically significant. Physicians in both groups rated test format innovative (4.2 out of 5.0).

Conclusion

The acceptable reliability and participants’ overall positive experiences point to the value of ongoing research into rigorous Web-based communication skills assessment.

Practice implications

With efficient and reliable scoring, the Web offers an important way to measure and potentially enhance physicians’ communication skills.     

Supporting parents of very young children with type 1 diabetes: Results from a pilot study

Objective

To assess initial efficacy and feasibility of a telephone-based supportive intervention for parents of young children with type 1 diabetes (T1D) designed to improve parental quality of life (QOL) through decreased parental stress, increased social support, and improved daily management of their child's diabetes.

Methods

The research team developed a brief program based on social cognitive theory for parents of young children with T1D. Twenty-four parents (88% mothers) of young children with T1D (ages 2-5 years) participated in a pilot study of the program and completed psychosocial questionnaires and a program satisfaction survey.

Results

Paired t-tests of pre- and post-intervention scores suggested a favorable within-group impact for the intervention group, as evidenced by decreased pediatric parenting stress and a trend for increased perceived social support. The program was well-received, with the majority of participants rating it as helpful and interesting.

Conclusion

Assisting parents with the unique challenges of diabetes management in young children through implementation of a structured intervention is promising.

Practice implications

A telephone-based intervention focused on child development, coping, and problem-solving skills has the potential to positively impact parents’ QOL and may have implications for children's health.     

Evaluating the validity evidence of an OSCE: results from a new medical school

Background

To prevent the problems of traditional clinical evaluation, the “Objective Structured Clinical Examination (OSCE)” was presented by Harden as a more valid and reliable assessment instrument. However, an essential condition to guarantee a high-quality and effective OSCE is the assurance of evidence to support the validity of its scores. This study examines the psychometric properties of OSCE scores, with an emphasis on consequential and internal structure validity evidence.

Methods

Fifty-three first year medical students took part in a summative OSCE at the Lebanese American University-School of Medicine. Evidence to support consequential validity was gathered by using criterion-based standard setting methods. Internal structure validity evidence was gathered by examining various psychometric measures both at the station level and across the complete OSCE.

Results

Compared to our actual method of computing results, the introduction of standard setting resulted in lower students’ average grades and a higher cut score. Across stations, Cronbach’s alpha was moderately low.

Conclusion

Gathering consequential and internal structure validity evidence by multiple metrics provides support for or against the quality of an OSCE. It is critical that this analysis be performed routinely on local iterations of given tests, and the results used to enhance the quality of assessment.

     

Psychometric properties of the Dickman Impulsivity Instrument in suicide victims and living controls of rural China

Background

We are not aware of any of impulsivity trait for Chinese rural youths. This study evaluated the psychometric properties of Dickman Impulsivity Instrument in suicide cases and living controls in rural China.

Methods

The participants, 392 suicide victims and 416 controls were respectively selected, and the psychological autopsy method was used to collect information. The Exploratory Factor Analysis was processed to evaluate the construct validity. The Cronbach's alpha was computed to evaluate the internal consistency, and the Spearman rank correlation coefficients between STAI Trait Anxiety Inventory, Coping Responses Inventory and Dickman Impulsivity Instrument were calculated to evaluate the convergent validity.

Results

Dysfunctional and Functional impulsivity were extracted for both samples. The mean of DI scores in suicide cases was larger than that in controls, but it was reversed for FI. There were no significant differences between DI and FI in suicide cases, but in controls the mean of DI scores was significantly smaller than that of FI. The Cronbach's alpha was around .863 and .779 respectively for DI and FI scales in suicides, and it was around .746 and .680 in controls. The DI and FI were significantly correlated with each other. Both the DI and FI were correlated with Approach and Avoidance Coping Response Inventory, and both of the scales were grossly independent with Trait Anxiety Inventory in two samples.

Conclusion

The results indicated that the Dickman Impulsivity Instrument was adequate to measure impulsivity trait for suicide victims and living controls through proxy data in rural China.     

Validity of a low literacy version of the Decisional Conflict Scale

Objective

To evaluate the psychometric properties of the 4-factor low literacy Decisional Conflict Scale (DCS-LL) with men eligible for prostate cancer screening (PCS).

Methods

We used baseline (T0; n = 149) and post-intervention (T2; n = 89) data from a randomized, controlled trial of a PCS decision aid to assess internal consistency reliability and construct, discriminant, and factor validity.

Results

There was evidence of excellent internal consistency reliability (α’s ≥ .80) and fair construct validity (most r's ≥ .40) for the DCS-LL except for the Supported subscale. The DCS-LL was able to discriminate between men who had decided and those who had not. There was evidence for the original 4-factor model at T0 but exploratory analysis suggested a 3-factor solution at T0 and T2 with Informed and Value Clarity as one factor.

Conclusion

For men eligible for PCS, feeling informed and feeling clear about values may not reflect distinct cognitive processes. Feeling supported may not be a factor contributing to uncertainty.

Practice Implications

Research should address whether current DCS subscales best represent the factors that contribute to uncertainty for PCS and for other screening decisions. Research should also explore the influence of health literacy on the factor structure of the DCS-LL.     

Development of a tool to assess the impact of a brief counseling curriculum: Validation of the Attitudes to Psychological Interventions and Counseling in Primary Care (APIC-PC) survey

Objective

To develop and validate an instrument to assess knowledge and attitudes regarding the use of psychological interventions in primary care.

Methods

A 13-item questionnaire was developed based on literature review and expert panel discussion. Exploratory and confirmatory factor analyses were carried out to test the construct validity of the subscales. Item-scale correlations were used to test the construct validity of the items. Internal reliability was tested by Cronbach's alpha. Responsiveness of the instrument was evaluated by using independent t-test of pre and post program scores.

Results

Exploratory factor analysis extracted four factors: skills and knowledge, confidence to provide counseling, willingness to provide counseling, and curriculum needs. Confirmatory factor analysis demonstrated excellent goodness-of-fit. Item scale correlations confirmed convergent and discriminant validity. Good responsiveness was shown on independent t-test of the pre and post responses.

Conclusions

The Attitudes to Psychological Interventions and Counseling in Primary Care (APIC-PC) survey is a valid and responsive instrument for assessing knowledge and attitudes regarding psychological approaches in patient management (mental health care).

Practice implications

Psycho-social counseling skills training is increasingly being recognised as a valuable component of undergraduate primary care medical education. The APIC-PC is a useful tool for evaluating the impact of such programs.     

Development and validation of a low-literacy Chronic Obstructive Pulmonary Disease knowledge Questionnaire (COPD-Q)

Objective

The objective of this study was to develop a content valid, understandable, readable, and reliable Chronic Obstructive Pulmonary Disease knowledge Questionnaire (COPD-Q).

Methods

Twenty-one questions were generated as potential items to include in the COPD-Q. Twenty-two content experts provided both qualitative and quantitative assessments of two COPD-Q drafts. Ten patient volunteers completed a field test to assess clarity of individual COPD-Q items. An additional 24 patient volunteers completed a pilot test to determine internal consistency and test-retest reliability of the COPD-Q. The Flesch Reading Ease (FRE) was used to assess reading grade level of the COPD-Q.

Results

Thirteen items were rated as “essential” (content validity ratio = p < 0.05) by content experts and composed the final COPD-Q. Internal consistency, using Cronbach's alpha, was 0.72. Test-retest reliability, using intraclass correlation coefficient, was 0.90. The FRE score of the COPD-Q was 74.7 (equivalent to a 5th grade reading level).

Conclusion

The COPD-Q is a valid, readable and reliable knowledge assessment instrument for assessing COPD knowledge in patients who may have low health literacy skills.

Practice implications

Health care providers can use the COPD-Q to tailor counseling efforts according to individual patient needs, and to assess the effectiveness of their educational interventions.     

Development and validation of a patient measure of doctor-patient caring

Objective

Conceptualising the doctor-patient relationship as a ‘window mirror’ exposes care delivery from doctor to self, doctor to patient, patient to self, and patient to doctor. These directions have not been measured concurrently. We aimed to develop and validate a patient questionnaire informed by this model.

Methods

A modified-Delphi exercise was conducted to develop, and face and content validate, the questionnaire. Stage 2 surveyed 495 patients in general practice to assess the internal consistency and construct validity of the questionnaire.

Results

The questionnaire is face and content valid. Its internal reliability and construct validity appear good. Patients who care more about their doctor also care more about themselves. A patient or doctor who cares about the other person is associated with increased self-care by that person.

Conclusions

Further development and testing of the patient questionnaire is warranted to validate measurement of how patients perceive the caring they and their doctor give, and receive from, each other.

Practice implications

From the patient perspective the questionnaire may increase awareness of the importance of family doctors and patients caring about each other and themselves. It may inform and evaluate medical students, educational programmes and caring in doctor-patient relationships.     

Medical nutrition therapy for overweight youth in their medical home: The KIDPOWER experience

Objective

To determine the feasibility and impact of implementing a standardized medical nutrition therapy (MNT) protocol to treat overweight children in their medical home. Primary outcomes evaluated were changes in eating behaviors, television habits, and weight status.

Methods

A MNT protocol with handouts (known as KIDPOWER) was delivered by a registered dietitian (RD) to overweight children (2-20 years) in 9 primary care practices in a rural community. A medical record review yielded height, weight and selected patient/parent reported eating and TV behaviors from each MNT visit. Mean values were compared by paired samples t-tests. Changes in behaviors and BMI z score were compared using repeated measures ANOVA.

Results

For patients with at least three MNT visits (n = 109) significant improvements in eating and TV viewing habits were seen by the third visit. Weight status, as determined by a significant decline in mean BMI z score, was improved.

Conclusion

The KIDPOWER protocol delivered by a RD in the pediatric medical home aided overweight youth in making behavior changes that positively impacted their weight status.

Practice implications

Reimbursement to RDs for treating childhood obesity is improving and KIDPOWER provides a model and tools for community practice.     

Measuring oral contraceptive knowledge: a review of research findings and limitations

Objective

Poor oral contraceptive (OC) knowledge may contribute to premature OC discontinuation and unintended pregnancy. Yet, to understand relationships between OC knowledge and contraceptive behavior, knowledge must be adequately measured. This review evaluates the findings and methodological limitations of research in which OC knowledge has been measured.

Methods

We performed a systematic review of primary research from January 1965 to January 2009. Studies were audited for study characteristics, purpose for measuring OC knowledge, key findings and measurement properties including administration method, knowledge domains, reliability, validity, health literacy and cultural sensitivity.

Results

We reviewed 21 studies: 18 cohort studies, including one psychometric evaluation, and three randomized trials. Results on OC knowledge outcomes were variable. Measures were largely self-administered survey (n = 15) and lacked assessment of all OC knowledge domains. Information on measures’ characteristics, reliability, validity, health literacy and cultural sensitivity was limited.

Conclusion

Existing OC knowledge measures lack critical psychometric elements, leading to inconsistent and unreliable findings.

Practice implications

Poor OC knowledge measurement precludes identifying counseling needs and developing interventions for contraceptive behavior change. Future research considerations include: measurement information in publications, psychometric evaluations, formal reliability/validity techniques, and attention to all OC knowledge domains, health literacy and cultural sensitivity.     

A computerized education module improves patient knowledge and attitudes about appropriate antibiotic use for acute respiratory tract infections

Background

Over-use of antibiotics for acute respiratory infections (ARIs) increases antimicrobial resistance, treatment costs, and side effects. Patient desire for antibiotics contributes to over-use.

Objective

To explore whether a point-of-care interactive computerized education module increases patient knowledge and decreases desire for antibiotics.

Methods

Bilingual (English/Spanish) interactive kiosks were available in 8 emergency departments as part of a multidimensional intervention to reduce antibiotic prescribing for ARIs. The symptom-tailored module included assessment of symptoms, knowledge about ARIs (3 items), and desire for antibiotics on a 10-point visual analog scale. Multivariable analysis assessed predictors of change in desire for antibiotics.

Results

Of 686 adults with ARI symptoms, 63% initially thought antibiotics might help. The proportion of patients with low (1-3 on the scale) desire for antibiotics increased from 22% pre-module to 49% post-module (p < .001). Self-report of “learning something new” was associated with decreased desire for antibiotics, after adjusting for baseline characteristics (p = .001).

Conclusion

An interactive educational kiosk improved knowledge about antibiotics and ARIs. Learning correlated with changes in personal desire for antibiotics.

Practice implications

By reducing desire for antibiotics, point-of-care interactive educational computer technology may help decrease inappropriate use for antibiotics for ARIs.     

Measuring providers' verbal and nonverbal emotion recognition ability: reliability and validity of the Patient Emotion Cue Test (PECT)

Objective

To describe the development and validation of the Patient Emotion Cue Test (PECT) as a tool to measure providers’ emotion cue recognition ability.

Methods

The PECT consists of 47 video clips depicting emotion cues that systematically vary in intensity of both verbal and nonverbal contents. The PECT assesses the provider's ability to detect and identify patients’ emotion cues accurately. A multi-stage development process produced the PECT. Reliability and validity were assessed in three studies.

Results

Scores on the PECT are normally distributed with significantly above chance responding. Across three studies, the PECT demonstrates convergent validity through significant correlations with standardized tests representing multiple channels of emotion recognition, including the face, body, and voice. The PECT shows adequate inter-item and split-half reliability.

Conclusion

The PECT is an easily administered, reliable, and valid test of emotion cue recognition.

Practice implications

The PECT can be used in future research on providers’ emotion recognition ability, for evaluating self-assessment of ability, and as a teaching tool in medical schools.     

Promoting advance care planning as health behavior change: development of scales to assess Decisional Balance, Medical and Religious Beliefs, and Processes of Change

Objective

To develop measures representing key constructs of the Transtheoretical Model (TTM) of behavior change as applied to advance care planning (ACP) and to examine whether associations between these measures replicate the relationships posited by the TTM.

Methods

Sequential scale development techniques were used to develop measures for Decisional Balance (Pros and Cons of behavior change), ACP Values/Beliefs (religious beliefs and medical misconceptions serving as barriers to participation), Processes of Change (behavioral and cognitive processes used to foster participation) based on responses of 304 persons age ≥ 65 years.

Results

Items for each scale/subscale demonstrated high factor loading (>.5) and good to excellent internal consistency (Cronbach α .76-.93). Results of MANOVA examining scores on the Pros, Cons, ACP Values/Beliefs, and POC subscales by stage of change for each of the six behaviors were significant, Wilks’ λ = .555-.809, η² = .068-.178, p ≤ .001 for all models.

Conclusion

Core constructs of the TTM as applied to ACP can be measured with high reliability and validity.

Practice implications

Cross-sectional relationships between these constructs and stage of behavior change support the use of TTM-tailored interventions to change perceptions of the Pros and Cons of participation in ACP and promote the use of certain Processes of Change in order to promote older persons’ engagement in ACP.