How reliable are assessments of clinical teaching?

BACKGROUND:  Learner feedback is the primary method for evaluating clinical faculty, despite few existing standards for measuring learner assessments.
OBJECTIVE:  To review the published literature on instruments for evaluating clinical teachers and to summarize themes that will aid in developing universally appealing tools.
DESIGN:  Searching 5 electronic databases revealed over 330 articles. Excluded were reviews, editorials, and qualitative studies. Twenty-one articles describing instruments designed for evaluating clinical faculty by learners were found. Three investigators studied these papers and tabulated characteristics of the learning environments and validation methods. Salient themes among the evaluation studies were determined.
MAIN RESULTS:  Many studies combined evaluations from both outpatient and inpatient settings and some authors combined evaluations from different learner levels. Wide ranges in numbers of teachers, evaluators, evaluations, and scale items were observed. The most frequently encountered statistical methods were factor analysis and determining internal consistency reliability with Cronbach's α. Less common methods were the use of test-retest reliability, interrater reliability, and convergent validity between validated instruments. Fourteen domains of teaching were identified and the most frequently studied domains were interpersonal and clinical-teaching skills.
CONCLUSIONS:  Characteristics of teacher evaluations vary between educational settings and between different learner levels, indicating that future studies should utilize more narrowly defined study populations. A variety of validation methods including temporal stability, interrater reliability, and convergent validity should be considered. Finally, existing data support the validation of instruments comprised solely of interpersonal and clinical-teaching domains.     

A literature review of “resident-as-teacher” curricula

OBJECTIVES: To examine the evaluation methods of resident teaching courses and to estimate the effectiveness of these teaching courses. DESIGN: We searched the literature from 1975 to May 2003 using the PubMed MESH terms internship and residency and teaching; 1,436 articles were identified and 77 contained information regarding teaching courses. Fourteen articles contained information regarding outcomes of resident teaching courses and were selected for intensive review.     

A systematic review of work-place interventions for alcohol-related problems

Aims   The aims of this study were to (1) gauge any improvement in methodological quality of work-place interventions addressing alcohol problems; and (2) to determine which interventions most effectively reduce work-place-related alcohol problems.
Methods   A literature search was undertaken of the data bases, Ovid Medline, PsychINFO, Web of Science, Scopus, HSELINE, OSHLINE and NIOSHTIC-2 for papers published between January 1995 and September 2007 (inclusive). Search terms varied, depending on the database. Papers were included for analysis if they reported on interventions conducted at work-places with the aim of reducing alcohol problems. Methodological adequacy of the studies was assessed using a method derived from the Cochrane Collaboration guidelines.
Results   Ten papers reporting on work-place alcohol interventions were located. Only four studies employed randomized controlled trials (RCT), but all these had methodological problems. Weaknesses in all studies related to representativeness of samples, consent and participation rates, blinding, post-test time-frames, contamination and reliability, and validity of measures used. All except one study reported statistically significant differences in measures such as reduced alcohol consumption, binge drinking and alcohol problems.
Conclusions   The literature review revealed few methodologically adequate studies of work-place alcohol interventions. Study designs, types of interventions, measures employed and types of work-places varied considerably, making comparison of results difficult. However, it appears from the evidence that brief interventions, interventions contained within health and life-style checks, psychosocial skills training and peer referral have potential to produce beneficial results.     

Life Support Courses: Are They Effective?

Study objective: To determine the effectiveness of life support courses for health care providers on the basis of one of three outcomes: (1) patient mortality and morbidity, (2) retention of knowledge or skills, and (3) change in practice behavior. Methods: English-language articles from 1975 to 1992 were identified through MEDLINE and ERIC searches, bibliographies of articles, and current abstracts. Studies were considered relevant if they included a study population of life support providers, an intervention of any of the identified life support courses, and assessment of at least one of the three listed outcomes. Relevant studies were selected and validity scores were assigned to them by agreement of two independent reviewers, using a structured form to assess validity. Data on setting, methods, participants, intervention, and outcomes were then abstracted and verified. Results: Seventeen of 67 identified studies pertaining to life support courses met the inclusion criteria. (1) All three mortality and morbidity studies indicated a positive impact, with an overall odds ratio of .28 (95% confidence interval [CI], .22 to .37). (2) No net increase in scores was found in 5 of 8 studies of retention of knowledge and in 8 of 9 studies of skills retention. Two of three studies reporting refresher activities yielded positive effects on knowledge retention. Outcomes were not significantly different between groups taught with modular or didactic techniques. (3) Studies assessing behavioral outcome were methodologically weak. Conclusion: Among providers, retention of knowledge and skills acquired by participation in support courses is poor. However, refresher activities increase knowledge retention. Modular courses are as good as lectures for learning course material. There is evidence that use of the Advanced Trauma Life Support course has decreased mortality and morbidity. Further studies of patient outcome and provider behaviors are warranted. [Jabbour M, Osmond MH, Klassen TP: Life support courses: Are they effective? Ann Emerg Med December 1996;28:690-698.]     

Clinician-teachers' Self-assessments Versus Learners' Perceptions

Understanding how clinician-teachers' self-assessments compare to learners' impressions can serve to help educators place each of these evaluations in the appropriate context. Past participants of the Johns Hopkins Faculty Development Program and other physician-teachers were surveyed in 2002 regarding their teaching skills and behaviors. We surveyed their learners to compare teacher and learner assessments of teaching proficiency, behaviors, enjoyment, and career satisfaction. In each area, learners' ratings were statistically significantly higher than their teachers' self-ratings. Though it is unclear whether teachers' or learners' assessments are a more accurate reflection of the truth, the more positive learner ratings should promote self-confidence in clinician-educators regarding their teaching abilities.     

Integrating teaching skills and clinical content in a faculty development workshop

Incorporating clinical content into medical education faculty development programs has been proposed as a strategy to consolidate faculty continuing medical education time and enhance learning. We developed a faculty development program for ambulatory internal medicine preceptors that integrated primary care genetics with ambulatory precepting. The instructional strategies addressed both areas simultaneously and included facilitated discussions, mini-lectures, trigger tapes, and role plays. To evaluate the program, we conducted a pre-post trial. Skills were measured by retrospective pre-post self-reported ratings and behaviors by self-reported implementation of commitment to change (CTC) statements. Participants' (N = 26) ambulatory precepting and primary care genetics skill ratings improved after the intervention. They listed an average of 2.4 clinical teaching CTC statements and 2.0 clinical practice CTC statements. By 3 months after the workshop, preceptors, as a group, fully implemented 32 (38%), partially implemented 35 (41%), and failed to implement 18 (21%) CTC statements. The most common barrier to clinical teaching change was insufficient skills (8 of 25; 32%) and to clinical practice change was lack of a suitable patient (15 of 25; 60%). Integrating clinical content with clinical teaching in a faculty development workshop is feasible, can improve clinical and teaching skills, and can facilitate behavior change.     

Teaching the one-minute preceptor. A randomized controlled trial

OBJECTIVE: The One-Minute Preceptor (OMP) model of faculty development is used widely to improve teaching, but its effect on teaching behavior has not been assessed. We aim to evaluate the effect of this intervention on residents' teaching skills. DESIGN: Randomized controlled trial. SETTING: Inpatient teaching services at both a tertiary care hospital and a Veterans Administration Medical Center affiliated with a University Medical Center. PARTICIPANTS: Participants included 57 second- and third-year internal medicine residents that were randomized to the intervention group (n = 28) or to the control group (n = 29). INTERVENTION: The intervention was a 1-hour session incorporating lecture, group discussion, and role-play. MEASUREMENTS AND MAIN RESULTS: Primary outcome measures were resident self-report and learner ratings of resident performance of the OMP teaching behaviors. Residents assigned to the intervention group reported statistically significant changes in all behaviors (P <.05). Eighty-seven percent of residents rated the intervention as "useful or very useful" on a 1-5 point scale with a mean of 4.28. Student ratings of teacher performance showed improvements in all skills except "Teaching General Rules." Learners of the residents in the intervention group reported increased motivation to do outside reading when compared to learners of the control residents. Ratings of overall teaching effectiveness were not significantly different between the 2 groups. CONCLUSIONS: The OMP model is a brief and easy-to-administer intervention that provides modest improvements in residents' teaching skills.     

Effect of the Japanese preventive-care version of the Minimum Data Set – Home Care on the health-related behaviors of community-dwelling, frail older adults and skills of preventive-care managers: A quasi-experimental study conducted in Japan

Aim:   To determine whether the Japanese preventive-care version of the Minimum Data Set-Home Care improves the health-related behaviors of older adults and the skills of preventive-care managers.
Methods:   Municipal preventive-care managers were instructed on the use of the Japanese preventive-care version of the Minimum Data Set – Home Care and asked to employ it in their interactions with clients during the intervention period (intervention group). The health-related behaviors of older adults (maintenance of self-care and consumption of a balanced diet) were assessed by self-rating methods. The skills of the preventive-care managers were assessed by considering the number of and variations in the needs of the clients, as reflected in the care plans formulated by the managers.
Results:   The clients' self-care levels were higher in the intervention group than in the control group ( P  < 0.05). A greater number of needs, as reflected in the care plans, were noted in the intervention group than in the control group ( P  < 0.05), and the variation in the assessed needs was greater in the former than in the latter.
Conclusion:   This study suggests that the Japanese preventive-care version of the Minimum Data Set – Home Care may improve the skills of preventive-care managers, and consequently, the health-related behaviors of frail older clients.     

Clinician-teachers’ self-assessments versus learners’ perceptions

Understanding how clinician-teachers’ self-assessments compare to learners’ impressions can serve to help educators place each of these evaluations in the appropriate context. Past participants of the Johns Hopkins Faculty Development Program and other physician-teachers were surveyed in 2002 regarding their teaching skills and behaviors. We surveyed their learners to compare teacher and learner assessments of teaching proficiency, behaviors, enjoyment, and career satisfaction. In each area, learners’ ratings were statistically significantly higher than their teachers’ self-ratings. Though it is unclear whether teachers’ or learners’ assessments are a more accurate reflection of the truth, the more positive learner ratings should promote self-confidence in clinician-educators regarding their teaching abilities. This paper was presented at the Society of General Internal Medicine’s National Meeting in Vancouver, Canada, April 29 to May 5, 2003, as “A Comparison of Teachers’ Self-assessments with Learners’ Perceptions.”     

Can wire‐guided cannulation reduce the risk of post‐endoscopic retrograde cholangiopancreatography pancreatitis? A meta‐analysis of randomized controlled trials

Background and Aim:  The use of wire-guided cannulation (WGC) for prevention of post-endoscopic retrograde cholangiopancreatography (ERCP) pancreatitis (PEP) is controversial. The aim of the present study was to assess the overall effect of WGC for PEP compared with conventional contrast-assisted cannulation by carrying out a meta-analysis of all available randomized controlled trials (RCT).
Methods:  Electronic databases, including PubMed, EMBASE, the Cochrane library and the Science Citation Index, were searched to retrieve relevant trials. In addition, meeting abstracts and the reference lists of retrieved articles were reviewed for further relevant studies. Outcome measures were the incidence of PEP.
Results:  Four RCT, enrolling a total of 1413 patients, were included. The meta-analysis failed to indicate a significant association between the use of WGC and the reduction of PEP (RR 0.34; 95% CI: 0.10–1.17; P  = 0.09). Subgroup analysis including trials without cross-over design showed a significant benefit with the use of WGC in reducing PEP (RR 0.20; 95% CI: 0.09–0.40; P  < 0.00001) and trials without precut used failed to indicate a significant differences between the two group (RR 0.38; 95% CI: 0.01–11.73; P  = 0.58).
Conclusions:  This meta-analysis showed only a non-significant reduction in the rate of PEP with the use of WGC. Further well-designed RCT are required to confirm the effect of WGC, especially in patients who were easier to cannulate.     

Cost analysis of nursing home registered nurse staffing times

Objectives: To examine potential cost savings from decreased adverse resident outcomes versus additional wages of nurses when nursing homes have adequate staffing.
Design: A retrospective cost study using differences in adverse outcome rates of pressure ulcers (PUs), urinary tract infections (UTIs), and hospitalizations per resident per day from low staffing and adequate staffing nursing homes. Cost savings from reductions in these events are calculated in dollars and compared with costs of increasing nurse staffing.
Setting: Eighty-two nursing homes throughout the United States.
Participants: One thousand three hundred seventy-six frail elderly long-term care residents at risk of PU development.
Measurements: Event rates are from the National Pressure Ulcer Long-Term Care Study. Hospital costs are estimated from Medicare statistics and from charges in the Healthcare Cost and Utilization Project. UTI costs and PU costs are from cost-identification studies. Time horizon is 1 year; perspectives are societal and institutional.
Results: Analyses showed an annual net societal benefit of $3,191 per resident per year in a high-risk, long-stay nursing home unit that employs sufficient nurses to achieve 30 to 40 minutes of registered nurse direct care time per resident per day versus nursing homes that have nursing time of less than 10 minutes. Sensitivity analyses revealed a robust set of estimates, with no single or paired elements reaching the cost/benefit equality threshold.
Conclusion: Increasing nurse staffing in nursing homes may create significant societal cost savings from reduction in adverse outcomes. Challenges in increasing nurse staffing are discussed.     

Students' Drinker Prototypes and Alcohol Use in a Naturalistic Setting

Background:  Perceptions about the type of people who drink, also referred to as drinker prototypes, may strengthen young people's motivation to engage in alcohol use. Previous research has shown that drinker prototypes are related to alcohol consumption in both adolescents and young adults. However, the evidence for the strength of these relationships remains inconclusive. One of the caveats in former studies is that all insights about prototype relations are based on self-reported data from youngsters themselves, mostly gathered in a class situation, which may contain bias due to memory distortions and self-presentation concerns.
Methods:  The present study examined the impact of drinker prototypes on young adults' drinking patterns by using a less obtrusive measure to assess alcohol consumption, i.e. ad lib drinking among friend groups in the naturalistic setting of a bar lab. Drinker prototypes, self-reported alcohol use in the past, and observed alcohol intake in the bar lab were assessed among 200 college students. Relations between participants' drinker prototypes and their self-reported and observed drinking behavior were examined by computing correlations and conducting multilevel analyses.
Results:  Drinker prototypes were related to both self-reported and observed alcohol use. However, the drinking patterns of friend group members had a strong impact on participants' individual drinking rates in the bar lab. After these group effects had been controlled for, only heavy drinker prototypes showed relations with observed alcohol intake in the bar lab.
Conclusions:  These findings further establish the value of drinker prototypes in predicting young adults' drinking behavior and suggest that people's motivation to drink alcohol in real-life drinking situations is related to their perceptions about heavy drinkers.     

Alcohol in a social context: findings from event-contingent recording studies of everyday social interactions

Background:  Data concerning the effects of alcohol on social interaction in everyday life are limited.
Methods:  Healthy volunteers in 4 studies of social behaviors and mood were instructed to complete record forms immediately after a social interaction had occurred, a method known as event-contingent recording. Record forms asked questions about quarrelsome, agreeable, dominant, and submissive behaviors; about aspects of mood; and, in 3 studies, about perceptions of others. Each form also contained a question about alcohol consumption prior to a social interaction. For the present report, only social interactions taking place in the evening and outside the work setting were included. Only individuals who consumed alcohol at least once in these circumstances were included ( n  = 171).
Results:  Social interactions involving alcohol were primarily characterized by higher levels of agreeable behaviors, by perceptions of greater agreeableness in others, and by more positive mood. Alcohol consumption was not associated with higher levels of quarrelsomeness.
Conclusions:  Alcohol consumption in a social context may have predominantly positive effects, an observation which is at odds with most alcohol-induced aggression experiments performed in laboratory settings. Drinking in everyday life may be less likely to result in aggression because, unlike in most laboratory experiments, individuals can choose among a variety of behaviors in response to social cues and the alcohol dose consumed is usually lower. Event-contingent recording provides a new approach for the study of alcohol's effects in everyday life and the conditions in which alcohol might result in interpersonal aggression.     

Overweight and perception of overweight as predictors of smokeless tobacco use and of cigarette smoking in a cohort of Swedish adolescents

Aims   To study the association between measured or perceived overweight in adolescence and subsequent uptake of cigarette smoking and of the Swedish smokeless tobacco 'snus' (oral moist snuff).
Design   Population-based prospective cohort study with 7 years' follow-up.
Setting   Self-administered questionnaires and school nurses' visits.
Participants   A total of 2922 children of both sexes and mean age 11.6 years at recruitment, resident in the Stockholm region, Sweden.
Measurements   Tobacco use was self-reported at baseline and on six subsequent surveys. Subjects' height and weight were measured by school nurses during the first 4 years, self-reported thereafter. Overweight perception was self-reported at the age of 15 years.
Findings   Overweight and perception of overweight were not associated with subsequent uptake of either smoking or snus among males. Among females, overweight at baseline was associated with uptake of smoking [adjusted hazard ratio (HR) 1.34, 95% confidence interval (CI) 1.09–1.63], but not of snus. A similar pattern was found with overweight status during follow-up. Among girls with low-educated parents, overweight at baseline predicted the uptake of both snus and smoking. Among 15-year-old females who never used tobacco perceived overweight was associated with subsequent uptake of smoking (adjusted HR 1.71, 95% CI 1.20–2.46), but not of snus.
Conclusions   In Sweden, adolescent girls with actual or perceived overweight are at increased risk to start smoking, while indications that this increased risk applies to smokeless tobacco (snus) are limited to girls of low socio-economic status.     

Measuring disability in older adults: The International Classification System of Functioning, Disability and Health (ICF) framework

Background:   Despite the importance of disability to geriatric medicine, no large scale study has validated the activity and participation domains of the International Classification System of Functioning, Disability, and Health (ICF) in older adults. The current project was designed to conduct such as analysis, and then to examine the psychometric properties of a measure that is based on this conceptual structure.
Methods:   This was an archival analysis of older adults ( n  = 1388) who had participated in studies within our Claude D Pepper Older Americans Independence Center. Assessments included demographics and chronic disease status, a 23-item Pepper Assessment Tool for Disability (PAT-D) and 6-min walk performance.
Results:   Analysis of the PAT-D produced a three-factor structure that was consistent across several datasets: activities of daily living disability, mobility disability and instrumental activities of daily living disability. The first two factors are activities in the ICF framework, whereas the final factor falls into the participation domain. All factors had acceptable internal consistency reliability (>0.70) and test–retest (>0.70) reliability coefficients. Fast walkers self-reported better function on the PAT-D scales than slow walkers: effect sizes ranged from moderate to large (0.41–0.95); individuals with cardiovascular disease had poorer scores on all scales than those free of cardiovascular disease. In an 18-month randomized clinical trial, individuals who received a lifestyle intervention for weight loss had greater improvements in their mobility disability scores than those in a control condition.
Conclusion:   The ICF is a useful model for conceptualizing disability in aging research, and the PAT-D has acceptable psychometric properties as a measure for use in clinical research.     

They just don't get enough! Variable intern experience in bedside procedural skills

Background: Medical school and resident training programmes offer different learning opportunities and outcomes. The aim of the study was to assess medical student and intern experience in common clinical procedures.
Methods: Interns employed in a metropolitan teaching hospital from 2000 to 2004 completed a survey of experience and confidence in clinical procedures at the beginning and end of their intern year. Attendance at and the contribution to procedural confidence of a voluntary procedural skill-training programme were examined.
Results: For the 314 interns, clinical experience before and during internship varied for each procedure and between year cohorts as did training programme attendance (44–84%). Student procedural confidence was predicted by pre-intern experience either on patients or by simulation ( β  = 0.17, 95% confidence interval (CI) 0.02–0.21, P  = 0.03) and age >30 years on commencing internship ( β  = 8.44, 95%CI 3.03–14.06, P  = 0.003. Adjusted R ² = 0.08, P  = 0.002). Intern procedural confidence by year's end was predicted by attendance at the training programme ( β  = 0.48, 95%CI 0.34–0.62, P  < 0.001), intern experience with patient procedures ( β  = 0.34, 95%CI 0.21–0.47, P  < 0.001) and a clear decision to enter a postgraduate training programme ( β  = 0.13, 95%CI 0.04–0.22, P  = 0.007, Adjusted R ² = 0.50, P  < 0.001).
Conclusion: Interns and students receive variable experience to carry out procedural skills on patients. This makes designing training programmes difficult as training needs vary each year. Both mandatory supervision of key skills and opportunities to supplement limited experience are needed during the intern year to ensure a uniform experience.     

The role of personality dispositions to risky behavior in predicting first-year college drinking

Aims   US college student drinking is associated with enormous risks to health, safety and productivity. Recent advances in personality research that have delineated multiple, separate dispositions to engage in risky behaviors may help to clarify the personality contribution to risk for this problem.
Design   The authors compared the prospective roles of sensation seeking, lack of planning, lack of perseverance, negative urgency and positive urgency (dispositions to engage in rash action when in an unusually negative or positive mood, respectively) in predicting increases in drinking frequency, drinking quantity and negative outcomes from consumption across the first year of college.
Setting   University of Kentucky campus.
Participants   A total of 418 first-year US college students enrolled in an Introduction to Psychology course during the first assessment; 293 participants completed both phases of the study.
Measurements   Participants completed self-report measures of personality and drinking behavior twice during the first year of college [the UPPS-R Impulsive Behavior Scale, positive urgency measure (PUM) and Drinking Styles Questionnaire (DSQ)].
Findings   Whereas sensation seeking related to increases in the frequency with which college students drank alcohol, positive urgency predicted increases in (i) the quantity of alcohol students consumed at any given drinking episode and (ii) negative outcomes experienced from drinking.
Conclusions   It appears that although sensation seeking is a risk factor for participation in drinking behaviors, risk for increased quantity of consumption and its negative outcomes may be more a function of dyscontrol stemming from high positive mood for college students.     

A Traditionally Administered Short Course Failed to Improve Medical Students' Diagnostic Performance

BACKGROUND:  Quite often medical students or novice residents have difficulty in ruling out diseases even though they are quite unlikely and, due to this difficulty, such students and novice residents unnecessarily repeat laboratory or imaging tests.
OBJECTIVE:  To explore whether or not a carefully designed short training course teaching Bayesian probabilistic thinking improves the diagnostic ability of medical students.
PARTICIPANTS AND METHODS:  Ninety students at 2 medical schools were presented with clinical scenarios of coronary artery disease corresponding to high, low, and intermediate pretest probabilities. The students' estimates of test characteristics of exercise stress test, and pretest and posttest probability for each scenario were evaluated before and after the short course.
RESULTS:  The pretest probability estimates by the students, as well as their proficiency in applying Bayes's theorem, were improved in the high pretest probability scenario after the short course. However, estimates of pretest probability in the low pretest probability scenario, and their proficiency in applying Bayes's theorem in the intermediate and low pretest probability scenarios, showed essentially no improvement.
CONCLUSION:  A carefully designed, but traditionally administered, short course could not improve the students' abilities in estimating pretest probability in a low pretest probability setting, and subsequently students remained incompetent in ruling out disease. We need to develop educational methods that cultivate a well-balanced clinical sense to enable students to choose a suitable diagnostic strategy as needed in a clinical setting without being one-sided to the "rule-in conscious paradigm."     

The association between alcohol consumption and biomarkers of alcohol exposure with total serum immunoglobulin E levels

Background:  A number of studies have shown that self-reported alcohol intake is positively associated with total serum immunoglobin E (IgE) levels. The purpose of the present study was to investigate the association of self-reported alcohol consumption and different biomarkers of alcohol exposure to total serum IgE levels in a general adult population.
Methods:  A total of 3,443 subjects aged 20 to 79 years from the population-based cross-sectional Study of Health in Pomerania (SHIP) were included in the analyses. Information on alcohol consumption and serum carbohydrate-deficient transferrin (CDT), gamma-glutamyl transferase (GGT), aspartate-amino transferase (ASAT), andalanine-amino transferase (ALAT) levels were measured. Multivariable linear regression models were performed separately in atopic and nonatopic subjects.
Results:  In nonatopic subjects positive associations between self-reported alcohol consumption as well as all considered biomarkers of alcohol exposure and total serum IgE levels were found. Further the results also suggested positive associations between alcohol consumption as well as alcohol biomarkers and total serum IgE level in atopic subjects, even though not all tests for linear trend reached statistical significance.
Conclusions:  In conclusion, biomarkers of alcohol exposure were positively associated with total serum IgE levels supporting that the positive association between self-reported alcohol intake and IgE levels observed in previous studies is real and not due to misclassification of alcohol intake or confounding by other factors that may be linked to both alcohol intake and total serum IgE levels.