Reinforcing outpatient medical student learning using brief computer tutorials: the Patient-Teacher-Tutorial sequence

ABSTRACT: BACKGROUND: At present, what students read after an outpatient encounter is largely left up to them. Our objective was to evaluate the education efficacy of a clinical education model in which the student moves through a sequence that includes immediately reinforcing their learning using a specifically designed computer tutorial. METHODS: Prior to a 14-day Pediatric Emergency rotation, medical students completed pre-tests for two common pediatric topics: Oral Rehydration Solutions (ORS) and Fever Without Source (FWS). After encountering a patient with either FWS or a patient needing ORS, the student logged into a computer that randomly assigned them to either a) completing a relevant computer tutorial (e.g. FWS patient: FWS tutorial = "in sequence") or b) completing the non-relevant tutorial (e.g. FWS patient: ORS tutorial = "out of sequence"). At the end of their rotation, they were tested again on both topics. Our main outcome was post-test scores on a given tutorial topic, contrasted by whether done in- or out-of-sequence. RESULTS: Ninety-two students completed the study protocol with 41 in the 'in sequence' group. Pre-test scores did not differ significantly. Overall, doing a computer tutorial in sequence resulted in significantly greater post-test scores (z-score 1.1 (SD 0.70) in sequence vs. 0.52 (1.1) out-of-sequence; 95% CI for difference +0.16, +0.93). Students spent longer on the tutorials when they were done in sequence (12.1 min (SD 7.3) vs. 10.5 (6.5)) though the difference was not statistically significant (95% CI diff: -1.2 min, +4.5). CONCLUSIONS: Outpatient learning frameworks could be structured to take best advantage of the heightened learning potential created by patient encounters. We propose the Patient-Teacher-Tutorial sequence as a framework for organizing learning in outpatient clinical settings.     

A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator.

PURPOSE: To determine the correlation between global ratings and criterion-based checklist scores, and inter-rater reliability of global ratings and criterion-based checklist scores, in a performance assessment using an anesthesia simulator. METHOD: All final-year medical students at the University of Toronto were invited to work through a 15-minute faculty-facilitated scenario using an anesthesia simulator. Students' performances were videotaped and analyzed by two faculty using a 25-point criterion-based checklist and a five-point global rating of competency (1 = clear failure, 5 = superior performance). Correlations between global ratings and checklist scores, as well as specific performance competencies (knowledge, technical skills, and judgment), were determined. Checklist and global scores were converted to percentages; means of the two marks were compared. Mean reliability of a single rater for both checklist and global ratings was determined. RESULTS: The correlation between checklist and global ratings was.74. Mean ratings of both checklist and global scores were low (58.67, SD = 14.96, and 57.08, SD = 24.27, respectively); these differences were not statistically significant. For a single rater, the mean reliability score across rater pairs for checklist scores was.77 (range.58-.93). Mean reliability score across rater pairs for global ratings was.62 (.40-.77). Global ratings correlated more highly with technical skills and judgment (r =.51 and r =.53, respectively) than with knowledge. (r =.24) CONCLUSION: Inter-rater reliability was higher for checklist scores than for global ratings; however, global ratings demonstrated acceptable inter-rater reliability and may be useful for competency assessment in performance assessments using simulators.     

Global ratings of videotaped performance versus global ratings of actions recorded on checklists: a criterion for performance assessment with standardized patients.

PURPOSE: To test whether global ratings of checklists are a viable alternative to global ratings of actual clinical performance for use as a criterion for standardized-patient (SP) assessment. METHOD: Five faculty physicians independently observed and rated videotaped performances of 44 medical students on the seven SP cases that comprise the fourth-year assessment administered at The Morchand Center of Mount Sinai School of Medicine to students in the eight member schools in the New York City Consortium. A year later, the same panel of raters reviewed and rated checklists for the same 44 students on five of the same SP cases. RESULTS: The mean global ratings of clinical competence were higher with videotapes than checklists, whereas the mean global ratings of interpersonal and communication skills were lower with videotapes. The correlations for global ratings of clinical competence showed only moderate agreement between the videotape and checklist ratings; and for interpersonal and communication skills, the correlations were somewhat weaker. CONCLUSION: The results raise serious questions about the viability of global ratings of checklists as an alternative to ratings of observed clinical performance as a criterion for SP assessment.     

医学生心理弹性现状及其影响因素

目的:探究医学生心理弹性现状及其影响因素,为掌握医学生心理健康状况,提高心理弹性水平提供参考依据。方法:应用成人心理弹性量表(Resilience Scale for Adults,RSA)对1173名医学生进行调查,所得数据采用SPSS 17.0软件进行分析。结果:医学生心理弹性总均分为(4.03±0.39)分。心理弹性各维度平均值从高到低依次排序为:家庭凝聚力、自我效能、组织风格、社交能力和社会资源。女生在家庭凝聚力(t=-4.251,P0.001)和自我效能(t=-1.995,P0.05)维度得分均高于男生;男生在社交能力维度得分高于女生(t=2.486,P0.05)。非独生子女在自我效能维度得分高于独生子女(t=-2.065,P0.05);独生子女在社交能力维度得分高于非独生子女(t=2.765,P0.01)。来自城市家庭居住地的医学生在社交能力维度得分高于来自农村家庭居住地的医学生(t=2.785,P0.01)。不同年龄阶段的医学生在心理弹性(F=3.478,P0.05)、家庭凝聚力维度(F=2.659,P0.05)、社交能力维度(F=2.832,P0.05)得分比较,差异均具有统计学意义。不同年级的医学生心理弹性和各维度得分比较,差异均无统计学意义(P0.05)。结论:医学生的心理弹性总体处于中等水平。性别、是否独生子女、家庭居住地和年龄对医学生的心理弹性有一定的影响。应积极开展有针对性的心理健康教育,促进医学生的心理健康成长。     

A faculty development program to train tutors to be discussion leaders rather than facilitators.

PURPOSE: During 2003, 2004, and 2005, the role of 70 tutors was changed from that of facilitator to discussion leader, in a preclinical PBL learning course, Gastrointestinal Pathophysiology, by use of three key business school teaching strategies: questions, summaries, and schematics. The purpose of this study was to learn what difference this new approach made. METHOD: During each of the three study years, 171 (2003), 167 (2004), and 170 (2005) students were given Likert-scale attitudinal questionnaires to rate whether their tutors encouraged student direction of the tutorials and whether the summaries and closure schematics benefited their learning. Students' overall course evaluations and mean USMLE scores were quantitatively analyzed, pre- and postintervention. A variety of statistical tests were used to assess the statistical significance of means at the confidence level of .05. RESULTS: In the third year of the program, student ratings indicated that their tutors were significantly better at encouraging student direction of the tutorials than in the first year (P < .05). The students reported that the tutorial made a more important contribution to their learning (P < .05), and the course objectives were better stated (P = .038) and better met (P = .007). Overall satisfaction with the course also improved significantly (P = .006). Part I gastrointestinal system mean scores of the USMLE showed a statistically significant increase in 2005 compared with 2001 or 2002. CONCLUSIONS: The tutor as a discussion leader who questions, summarizes, and uses schematics to illustrate concepts had a significant and positive impact on learning in tutorials, achieving course objectives, improving overall course satisfaction, and increasing a standardized national exam's mean score.     

Can specialists and generalists teach clinical skills to second-year medical students with equal effectiveness?

PURPOSE: To compare the effectiveness of specialists and generalists as small-group leaders teaching basic physical examination skills to preclinical medical students. METHOD: Specialists and generalists were randomly assigned to teach physical examination skills to 69 groups of second-year students (n = 288). At the conclusion of the course, the specialist- and generalist-led groups were compared using three measures: students' scores on an objective structured clinical exam (OSCE), students' evaluations of their small-group leaders, and leaders' self-evaluations of confidence in teaching. RESULTS: OSCE scores did not differ between students taught by specialists and generalists (93.5% and 93.8% respectively, p = NS). Students' evaluations of their leaders were similar for nine characteristics rated on a seven-point scale (7 = strongly agree/outstanding; range of results for specialists: 6.20-6.62, for generalists 6.34-6.75, p = NS). Leaders expressed similar levels of confidence (on a seven-point scale; 7 = very confident) in their abilities to teach the neurologic exam (specialists 5.52, generalists 6.19, mean effect size difference 0.44, p = NS) and complete history and physical exam (6.03 and 6.53, mean effect size difference 0.43, p = NS). Specialists were significantly less confident in teaching the cardiovascular exam (5.80 and 6.50, mean effect size difference 0.51, p <.05) and pulmonary exam (5.56 and 6.60, mean effect size difference, 0.80, p <.01). CONCLUSIONS: Specialists and generalists can teach preclinical medical students with equal effectiveness as rated by the students and the students' scores on the OSCE examination, but specialists do not rate themselves as confident as do generalists to teach some skills.     

Effects of tutors with subject expertise on the problem-based tutorial process

With the increased interest in problem-based, small-group learning in medical education, a debate has arisen about whether the tutor should be an expert in the subject under discussion. This 1988 study at Harvard Medical School demonstrates that tutors' expertise has important effects on the process of discussion in a problem-based tutorial. In comparing discussions of subjects in which the tutors described themselves as expert with those in which they did not, the authors found that the tutors with expertise tended to take a more directive role in tutorials: they spoke more often and for longer periods, provided more direct answers to the students' questions, and suggested more of the topics for discussion. Tutor-to-student exchanges predominated, with less student-to-student discussion. These effects endanger an important goal of problem-based learning: the development of students' skills in active, self-directed learning.     

Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study

ABSTRACT: BACKGROUND: Over two-thirds of UK medical schools are augmenting their selection procedures for medical students by using the United Kingdom Clinical Aptitude Test (UKCAT), which employs tests of cognitive and non-cognitive personal qualities, but clear evidence of the tests' predictive validity is lacking. This study explores whether academic performance and professional behaviours that are important in a health professional context can be predicted by these measures, when taken before or very early in the medical course. METHODS: This prospective cohort study follows the progress of the entire student cohort who entered Hull York Medical School in September 2007, having taken the UKCAT cognitive tests in 2006 and the non-cognitive tests a year later. This paper reports on the students' first and second academic years of study. The main outcome measures were regular, repeated tutor assessment of individual students' interpersonal skills and professional behaviour, and annual examination performance in the three domains of recall and application of knowledge, evaluation of data, and communication and practical clinical skills. The relationships between non-cognitive test scores, cognitive test scores, tutor assessments and examination results were explored using the Pearson product--moment correlations for each group of data; the data for students obtaining the top and bottom 20% of the summative examination results were compared using Analysis of Variance. RESULTS: Personal qualities measured by non-cognitive tests showed a number of statistically significant relationships with ratings of behaviour made by tutors, with performance in each year's objective structured clinical examinations (OSCEs), and with themed written summative examination marks in each year. Cognitive ability scores were also significantly related to each year's examination results, but seldom to professional behaviours. The top 20% of examination achievers could be differentiated from the bottom 20% on both non-cognitive and cognitive measures. CONCLUSIONS: This study shows numerous significant relationships between both cognitive and non-cognitive test scores, academic examination scores and indicators of professional behaviours in medical students. This suggests that measurement of non-cognitive personal qualities in applicants to medical school could make a useful contribution to selection and admission decisions. Further research is required in larger representative groups, and with more refined predictor measures and behavioural assessment methods, to establish beyond doubt the incremental validity of such measures over conventional cognitive assessments.     

Comparative assessment of multiple choice questions versus short essay questions in pharmacology examinations

This retrospective study compared the performance of medical students in multiple choice questions (MCQs) and short essay questions (SEQs). During the 3 year analysis, 533 students had an average score of 51.34% (SD 9.9) in the SEQ and 64.71%(SD 9.9)in the MCQs. Regression analysis showed a significant correlation(r=0.64, P<0.01) between MCQs and SEQs. When student performance was grouped by final course grade, a statistically significant correlation between MCQs and SEQs scores existed only for the 405 students who received a passing grade (r=0.21, P<0.01). The MCQ and SEQ scores were not correlated for the 128 students who failed (r=0.11, P=0.08) or for 70 students who achieved distinctions (r=-0.27, P=0.13). MCQ scores were significantly higher (P<0.01) than SEQ for each of the groups when analyzed by the two-way ANOVA test. The result of this study suggests that for most students, the strong correlation between MCQ and SEQ indicates that student performance was independent of testing format. For students at either end of the performance spectrum, the lack of correlation suggests that the performance in one of the testing format had a strong influence on the final course grade. In addition, those students who failed the course were likely to be weak in both testing modalities, whereas students in all grade groups were more likely to perform better in the MCQs than SEQs.     

Relationship of medical students' admission interview scores to their dean's letter ratings.

The authors examined the relationship between the admission interview scores for 62 students in the 1986 entering class at Dartmouth Medical School and the students' dean's letter ratings given four years later; they found the relationship to be significant (V = .372, p = .014) and the interview scores to be better independent predictors of the ratings than were total Medical College Admission Test scores or science grade-point averages. Among the 17 students receiving "strong" admission interview scores, 53% received dean's letter ratings in the top one-third and 47% received ratings in the lower two-thirds. Of those 34 who received "medium" interview scores, 68% received ratings in the lower two-thirds; all 11 students who received "weak" interview scores received ratings in the lower two-thirds. The authors suggest (1) that admission interview scores help schools to identify more clearly those applicants most likely to become strong, competitive performers in residency and (2) that the significant relationship between interview scores and dean's letter ratings indicates a need to discover what qualities the interview actually measures and to consider the methods by which interviewers are trained, rather than to forsake the interview.     

An advanced communication skills course for fourth-year, post-clerkship students.

OBJECTIVE: A novel five-module advanced communication skills course entitled "Doctor-Patient Relationships" was planned and implemented in 2000-01 at the University of British Columbia (UBC). The course was part of the final four-month component of the new MD undergraduate program: Effective Skills for Medical Practice. The goals of the communication skills course were to (1) address problems experienced by the students so far; (2) address deficiencies in achieving the UBC exit competencies; (3) help the students pass the Medical Council of Canada examinations, in particular objectives related to the Considerations of the Legal, Ethical, and Organizational aspects of the practice of medicine (CLEO); and (4) help students prepare for their roles beyond undergraduate medicine (residency, independent practice). DESCRIPTION: The course was developed by an interdisciplinary team (family practice, pathology, pediatrics, psychiatry, surgery) with input from students. The broad strengths and weaknesses of their communication skills training were identified by seven third-year medical students who kept logs over the course of their clinical clerkships to document their learning of communication skills. Analysis of these logs plus feedback meetings with the students revealed attitudinal and skills issues that needed to be addressed in the new course. The goals and principles of the course were in part agreed upon by focus groups with students, attended by faculty observers, to ensure their relevance to students. The first module "Beyond the Mask: Surviving and Thriving in Residency Training" is designed to focus students' attention on the personal relevance of developing excellence in communication skills in preparation for residency training. It includes a video of residents talking about their experiences of communication problems to trigger reflection and discussion. In the remaining four modules the students are required to put communication skills together with their medical knowledge. Each module includes pre-readings, video demonstrations (in sessions 4 and 5), practice with standardized patients (total of 14 scenarios) and structured feedback from SPs, students, and tutor. The themes of the sessions are "Dealing with Emotionally Challenging Patient Situations (informing about bad news), "Compliance and Patient Information," "Informed Consent and Shared Decision Making," and "Difficult Physician-Patient Encounters." Each module lasts two hours. The course was implemented for 120 students, facilitated by 14 tutors (seven to eight students per group). DISCUSSION: Student involvement in many different ways provided an important reality check and made us think about how to present the new course so that it was relevant and interesting to students. Attention to student input was a major contributor to the good evaluations given the course. Students rated the course highly: the relevance of the weekly themes was rated 4.21 on a five-point scale; the effectiveness of the SP interviews, 4.10; the effectiveness of the group discussion and feedback, 4.18; and overall course effectiveness in enhancing communication skills, 3.91. The tutors also rated the course highly, and the students rated the tutors highly. Minor changes will be made to the course next year based on the specific suggestions for improvement, which were identified.     

The use of questionnaires to assess achievement of course goals in medical students' longitudinal community-based clinical experiences.

PURPOSE: To examine the usefulness of questionnaires for assessing achievement of course goals in medical students' longitudinal community-based clinical experiences. METHOD: In 1997, the authors surveyed 114 first-year students and their preceptors in a longitudinal community-based program at The Joan and Sanford I. Weill Medical College of Cornell University. The questionnaire used a Likert scale to assess students' and preceptors' pre-course expectations for achieving specific course goals and their post-course perceptions of having met those goals. The students also rated global learning and satisfaction during each office preceptor session, and faculty assessed the students' physical examination skills at the end of the course. RESULTS: For all goals assessed, the preceptors scored their students' achievement of course goals significantly higher than did the students themselves (p < .01). The students invariably scored their post-course perceptions of having achieved the goals lower than they did their pre-course expectations (p < .001). Before the course, the preceptors were confident in their ability to teach the curricular material; this confidence remained after the course. Global learning and satisfaction scores were high and all students performed satisfactorily in the demonstration examination. CONCLUSIONS: Students and preceptors may not agree on students' achievement of course goals. Furthermore, despite students' high ratings of global satisfaction and learning, and despite their satisfactory performance of physical examination skills, their ratings of post-course performance may be affected by pre-course expectations. The authors suggest that questionnaires assessing students' and preceptors' perceptions of students' achievement of specific goals should be independently verified before making decisions to modify objectives and activities in these kinds of courses.     

Validation and feasibility of problem-based learning in rehabilitation of patients with coronary artery disease

A patient education programme applying problem-based learning (PBL) was developed for patients with coronary artery disease (CAD). Groups with 6-8 patients and a tutor from the rehabilitation team met nine times for 1.5h each. The feasibility and validity of the model was evaluated using patient questionnaires, interviews with tutors and video observations of tutorials. The participants were active (69% of all input) and discussions of acquired knowledge and lifestyle changes took place in all groups. A total of 89% of the patients reported implementation of lifestyle changes and over 90% rated their learning and overall experience of the programme as acceptable or high and the demands as acceptable. Shortcomings were the limited use of some of the steps in the problem-solving process and tutors' difficulties in adapting to their new role; their answering of questions was higher than planned (35% of their total input). The programme was feasible in clinical routine.     

What do medicine clerkship preceptors do best?

PURPOSE: Students' ratings of preceptors are widely used in medical education for feedback and evaluation purposes. The present study investigated students' ratings of the clinical teaching skills of inpatient attending physicians, inpatient residents, and outpatient attending physicians to assess differences among types of preceptors and relative strengths and weaknesses. METHOD: A total of 268 students from three academic years (1997-2000) at one medical school rated preceptors on an end-of-clerkship evaluation, for a total of 1,680 ratings. When the ratings were aggregated by preceptors' names and types, there were 691 mean ratings of preceptors. Relative strengths and weaknesses were identified. Differences in mean ratings by preceptor type (inpatient attending physician, inpatient resident, and outpatient attending physician) were evaluated, and strengths and weaknesses were identified by rank ordering the items' means. RESULTS: Students tended to rate outpatient attending physicians higher than inpatient attending physicians or residents. Areas where ratings suggested relative strengths included showing an interest in teaching, respecting students' opinions, and being available to students. Areas of relative weakness included increasing physical examination and interviewing skills. CONCLUSIONS: Students' ratings are useful for identifying strengths and weakness for groups of preceptors and, as such, are important sources of information for setting priorities for faculty development efforts.     

Is there a relationship between attending physicians' and residents' teaching skills and students' examination scores?

PURPOSE: Faculty development programs and faculty incentive systems have heightened the need to validate a connection between the quality of teaching and students' learning. This study was designed to determine the association between attending physicians' and residents' teacher ratings and their students' examination scores. METHOD: From a database of 362 students, 138 faculty, and 107 residents in internal medicine, student-faculty (n = 476) and student-resident (n = 474) pairs were identified. All students were in their third year, rotating on inpatient general medicine and cardiology services, July 1994 through June 1996, at a single institution. The outcome measure for students' knowledge was the NBME Subject Examination in internal medicine. To control for students' baseline knowledge, the predictors were scores on the USMLE Step 1 and a sequential examination (a clinically-based pre- and post-clerkship examination). Teaching abilities of faculty and residents were rated by a global item on the post-clerkship evaluation. Faculty's ratings used only scores from prior to the study period; residents' ratings included those scores students gave during the study period. RESULTS: Multivariate analyses showed faculty's teaching ratings were a small but significant predictor of the increase in students' knowledge. Residents' teaching ratings did not predict an increase in students' knowledge. CONCLUSION: Attending faculty's clinical teaching ability has a positive and significant effect on medical students' learning.     

Specialty choices of students who actually have choices: the influence of excellent clinical teachers.

PURPOSE: To determine the influence of the quality of attending physicians and residents on the specialty choices of excellent medical students, who actually have a broad choice of specialties. METHOD: In 1993-94 and 1994-95, 169 third-year students at the University of Kentucky College of Medicine were randomly assigned to two one-month rotations on general medicine inpatient wards. At the end of each rotation, the students confidentially evaluated the attending physician and the supervising resident (different for each rotation) with whom they had worked. Data were collected for 62 attending physicians and 89 residents. The authors analyzed the influences of the "best" and "worst" clinical instructors (those rated in the top and the bottom 20% by all students with whom they had worked over the two years) on "excellent" medical students (the 52 students whose USMLE I scores were in the top 30% of their class). RESULTS: Using regression approaches from the general linear model, the authors found that independent predictors of internal medicine residency choice for excellent medical students were exposure to highly rated internal medicine attendings (p = .02) and residents (p = .03). Nine of 29 (30%) of the excellent students who worked with a "best" medicine clinical instructor chose an internal medicine residency, while none of the 23 excellent medical students who did not work with a "best" medicine clinical instructor did so. The authors found no correlation in students' ratings of their pairs of attendings and residents, suggesting that rater bias did not explain the results. CONCLUSION: Better medical students who work with the best internal medicine attending physicians and residents in their internal medicine clerkship are more likely to choose an internal medicine residency.     

Pathological diagnosis of columnar cell lesions of the breast: are there issues of reproducibility?

AIMS: To assess inter/intraobserver variability in the interpretation of a series of digitised images of columnar cell lesions (CCLs) of the breast. METHODS: After a tutorial on breast CCL, 39 images were presented to seven staff pathologists, who were instructed to categorize the lesions as follows: 0, no columnar cell change (CCC) or ductal carcinoma in situ (DCIS); 1, CCC; 2, columnar cell hyperplasia; 3, CCC with architectural atypia; 4, CCC with cytological atypia; 5, DCIS. Concordance with the tutor's diagnosis and degree of agreement among pathologists for each image were determined. The same set of images was re-presented to the pathologists one week later, their diagnoses collated, and inter/intraobservor reproducibility and level of agreement for individual images analysed. RESULTS: Diagnostic reproducibility with the tutor ranged from moderate to substantial (kappa values, 0.439-0.697) in the first exercise. At repeat evaluation, intraobserver agreement was fair to perfect (kappa values, 0.271-0.832), whereas concordance with the tutor varied from fair to substantial (kappa values, 0.334-0.669). There was unanimous agreement on more images during the second exercise, mainly because of agreement on the diagnosis of DCIS. The lowest agreement was seen for CCC with cytological atypia. CONCLUSIONS: Interobserver and intraobserver agreement is good for DCIS, but more effort is needed to improve diagnostic consistency in the category of CCC with cytological atypia. Continued awareness and study of these lesions are necessary to enhance recognition and understanding.     

Undergraduate technical skills training guided by student tutors – Analysis of tutors' attitudes,tutees' acceptance and learning progress in an innovative teaching model

Background  

Skills labs provide a sheltered learning environment. As close supervision and individual feedback were proven to be important in ensuring effective skills training, we implemented a cross-year peer tutor system in our skills lab of internal medicine that allowed intense training sessions with small learning groups (3–4 students) taught by one student tutor.