A comparison of clinical teaching evaluations by resident and peer physicians

The purpose of this study was to compare the reliability of inpatient teaching evaluations by resident and peer physicians on Mayo internal medicine hospital services. Three resident and three peer evaluators observed 10 consecutively chosen attending physicians on the Mayo hospital services. Evaluations by resident and peer physicians were compared in terms of mean scores. Kendall's coefficient of concordance (KCC) was used to summarize inter-rater reliabilities and Cronbach's coefficient alpha was used to determine internal consistencies of evaluations by residents and peers. Results of this study revealed that mean scores of the 13 evaluation items were generally higher for resident than peer physicians. None of the items completed by residents had KCC scores >0.5, whereas 10 of the items completed by peers had KCC scores >0.5. Likewise, none of the residents' items had KCC p-values <0.05, whereas nine of the peers' items had KCC p-values <0.05. The overall internal consistency was higher for peers (alpha = 0.76) than for residents (alpha = 0.71). In conclusion, resident physicians uniformly rate faculty highly. Furthermore, peer evaluations yield higher inter-rater and internal reliabilities than resident evaluations, indicating that peer physicians are more reliable than residents for assessing bedside teaching.     

Evaluation in undergraduate medical education: Conceptualizing and validating a novel questionnaire for assessing the quality of bedside teaching

Background: Evaluation is an integral part of curriculum development in medical education. Given the peculiarities of bedside teaching, specific evaluation tools for this instructional format are needed. Development of these tools should be informed by appropriate frameworks. The purpose of this study was to develop a specific evaluation tool for bedside teaching based on the Stanford Faculty Development Program’s clinical teaching framework.

Methods: Based on a literature review yielding 47 evaluation items, an 18-item questionnaire was compiled and subsequently completed by undergraduate medical students at two German universities. Reliability and validity were assessed in an exploratory full information item factor analysis (study one) and a confirmatory factor analysis as well as a measurement invariance analysis (study two).

Results: The exploratory analysis involving 824 students revealed a three-factor structure. Reliability estimates of the subscales were satisfactory (α?=?0.71–0.84). The model yielded satisfactory fit indices in the confirmatory factor analysis involving 1043 students.

Discussion: The new questionnaire is short and yet based on a widely-used framework for clinical teaching. The analyses presented here indicate good reliability and validity of the instrument. Future research needs to investigate whether feedback generated from this tool helps to improve teaching quality and student learning outcome.     

Twelve tips for peer observation of teaching

This paper outlines twelve tips for undertaking peer observation of teaching in medical education, using the peer review model and the experiences of the authors. An accurate understanding of teaching effectiveness is required by individuals, medical schools, and universities to evaluate the learning environment and to substantiate academic and institutional performance. Peer Observation of Teaching is one tool that provides rich, qualitative evidence for teachers, quite different from closed-ended student evaluations. When Peer Observation of Teaching is incorporated into university practice and culture, and is conducted in a mutually respectful and supportive way, it has the potential to facilitate reflective change and growth for teachers.     

An operational measure of physician lifelong learning: its development,components and preliminary psychometric data

Despite the emphasis placed on physicians' lifelong learning, no psychometrically sound instrument has been developed to provide an operational measure of the concept and its components among physicians. The authors designed this study to develop a tool for measuring physician lifelong learning, to identify its underlying components and to assess its psychometric properties. A 37-item questionnaire was developed, based on a review of literature and the results of two pilot studies. Psychometric analyses of the responses of 160 physicians identified 19 items that were included in the Jefferson Scale of Physician Lifelong Learning. Factor analysis of the 19 items showed five meaningful factors that were consistent with the definition and major features of lifelong learning. They were 'need recognition', 'research endeavor', 'self-initiation', 'technical skills' and 'personal motivation'. The method of contrasted groups provided evidence in support of the validity of the five factors. The factors' reliability was assessed by coefficient alpha. It is concluded that lifelong learning is a multifaceted concept, and its operational measure is feasible for evaluating different educational programs and for studying group differences among physicians.     

A general practice workplace-based assessment instrument: Content and construct validity

Abstract

Introduction: Relatively few general practice (GP) workplace-based assessment instruments have been psychometrically evaluated. This study aims to establish the content validity and internal consistency of the General Practice Registrar Competency Assessment Grid (GPR-CAG).

Methods: The GPR-CAG was constructed as a formative assessment instrument for Australian GP registrars (trainees). GPR-CAG items were determined by an iterative literature review, expert opinion and pilot-testing process. Validation data were collected, between 2014 and 2016, during routine clinical teaching visits within registrars’ first two general practice training terms (GPT1 and GPT2) for registrars across New South Wales and the Australian Capital Territory. Factor analysis and expert consensus were used to refine items and establish GPR-CAG’s internal structure. GPT1 and GPT2 competencies were analysed separately.

Results: Data of 555 registrars undertaking GPT1 and 537 registrars undertaking GPT2 were included in analyses. A four-factor, 16-item solution was identified for GPT1 competencies (Cronbach’s alpha range: 0.71–0.83) and a seven-factor 27-item solution for GPT2 competencies (Cronbach’s alpha: 0.63–0.84). The emergent factor structures were clinically characterisable and resonant with existing medical education competency frameworks.

Discussion: This study establishes initial evidence for the content validity and internal consistency of GPR-CAG. GPR-CAG appears to have utility as a formative GP training WBA instrument.     

Twelve tips to teaching (legal and ethical aspects of) research ethics/responsible conduct of research

Teaching research ethics is a requirement within modern health science, nursing and medical curricula. We have drawn on our experience of designing, developing and integrating the teaching of research ethics in a new, fully integrated medical school curriculum, delivered using Problem Based Learning and the recent literature relating to the teaching of research ethics to produce the following 12 Top Tips designed to encourage readers to seek opportunities to embed this teaching within a variety of curricula.     

AMEE Medical Education Guide No. 22: Refreshing lecturing: a guide for lecturers

This guide provides an overview of research on lecturing, a model of the processes of lecturing and suggestions for improving lecturing, learning from lectures and ways of evaluating lectures. Whilst primarily directed at teachers in the healthcare professions, it is equally applicable to all teachers in higher education. Lectures are the most ubiquitous method of teaching so they are an important part of a teacher's repertoire. Lectures are at least as effective as other methods of teaching at imparting information and explaining. Intention, transmission and output are the basis of a model of lecturing. The key skills of preparing lectures, explaining and varying student activities may be derived from the model. Preparation is based on purposes, content, various structures of lectures and the preparation of audiovisual aids. The essential ingredients of explaining are clarity, interest and persuasion. By varying activities, one can renew attention and develop student learning. Learning from lectures can be improved by teaching students the structure of lectures and methods of listening and note-taking. Student ratings of lectures are useful but over-used and limited ways of evaluating lectures. Equally important is peer review and more important than either student ratings or peer feedback is reflection on the practice of lecturing by individuals and course teams.     

An international study on teachers’ conceptions of learning and teaching and corresponding teacher profiles

Abstract

Introduction: Teachers’ conceptions of learning and teaching (COLT) affect their teaching behaviour. The 18 item COLT instrument has been developed in the Netherlands and comprises three scales, ‘teacher centredness’, ‘appreciation of active learning’ and ‘orientation to professional practice’. Previously we found five teacher profiles. The aim of this study was to find out if the COLT instrument can be used in an international setting.

Methods: Data were collected with the web-based COLT. Cronbach’s alphas of the three COLT scales were calculated. Subsequently a cluster analysis was conducted to identify different teacher profiles, followed by a split half validation procedure.

Results: Respondents (n?=?708) worked in 28 countries. Cronbach’s alphas were 0.67, 0.54, and 0.66. A six-cluster solution fitted best, based on meaning and explained variance. The sixth teacher profile scored high on ‘teacher centredness’, average on ‘appreciation of active learning’ and low on ‘orientation to professional practice’. The split half validation resulted in a Cohen’s kappa of 0.744.

Discussion: Cronbach’s alphas indicated acceptable reliablities for all three subscales. The new, sixth profile was labelled ‘neo-transmitter’.

Conclusion: We found evidence supporting the validity of the use of COLT in an international context and identified a new, sixth teacher profile.     

Revisiting the D-RECT tool: Validation of an instrument measuring residents’ learning climate perceptions

Introduction: Credible evaluation of the learning climate requires valid and reliable instruments in order to inform quality improvement activities. Since its initial validation the Dutch Residency Educational Climate Test (D-RECT) has been increasingly used to evaluate the learning climate, yet it has not been tested in its final form and on the actual level of use – the department.

Aim: Our aim was to re-investigate the internal validity and reliability of the D-RECT at the resident and department levels.

Methods: D-RECT evaluations collected during 2012–2013 were included. Internal validity was assessed using exploratory and confirmatory factor analyses. Reliability was assessed using generalizability theory.

Results: In total, 2306 evaluations and 291 departments were included. Exploratory factor analysis showed a 9-factor structure containing 35 items: teamwork, role of specialty tutor, coaching and assessment, formal education, resident peer collaboration, work is adapted to residents’ competence, patient sign-out, educational atmosphere, and accessibility of supervisors. Confirmatory factor analysis indicated acceptable to good fit. Three resident evaluations were needed to assess the overall learning climate reliably and eight residents to assess the subscales.

Conclusion: This study reaffirms the reliability and internal validity of the D-RECT in measuring residency training learning climate. Ongoing evaluation of the instrument remains important.     

Use of interaction cards to evaluate clinical performance

Currently in-training evaluation in Kuwait depends on the use of the global rating scale at the end of clinical rotation clerkships. Such a scale is inconsistent, subjective, and suffers from deficiencies such as positive skewness of the distribution of ratings and poor reliability. The aim of the study was to assess the inter-rater variation and the reliability of the recently introduced Interaction Card (IC) method for evaluating clinical performance and to measure the agreement between trainees' overall performance evaluation by the currently used global rating scale and the IC summative evaluation. In the study, 370 evaluators encountered 50 trainees during their basic clinical training rotations (internal medicine, surgery, obstetrics and gynecology, and pediatrics) at six hospitals. A total of 9146 encounters were conducted focusing on six clinical performance domains: clinical skills (taking history, case sheet, and physical examination), professional behaviour, case presentation, diagnosis, therapy and handling of emergencies. The method demonstrated significant inter-rater variation in the overall IC ratings according to specialty, rank of evaluator and hospital (p < 0.001). The Interaction Card was found to be reliable, as shown by the internal consistency between the six domains (Cronbach's alpha = 0.914). There was low correlation (Spearman rank correlation coefficient, rs = 0.337), and low agreement (Kappa = 0.131) between the global rating scale and Interaction Cards summative evaluations. The IC method provided instantaneous formative feedback and summative evaluation for clinical performance to trainees. The method can be generalized to encompass training and examinations programmes for all categories of trainees in most clinical specialties.     

How to find a web interface for successful education (WISEbyte)

'Give a man a fish and you feed him for a day. Teach him how to fish, and you feed him for a lifetime…'. Although the exact origin of this proverb is unknown, its meaning is clear and wisdom self-evident. In the field of health professions education, there are many websites that can be used as teaching aids, some of which have undergone peer review. Some organizations have created repositories of online teaching materials hosted by various organizations. You can certainly find a lot of 'fish' there to feed your appetite for high-quality teaching materials. For examples of repositories that contain online teaching materials, see Table 1. However, these repositories and other lists of websites cannot be comprehensive, so it is important to know the basic review skills to evaluate websites that may be useful for your teaching that you come across in your journeys around the web. This article intends to teach you 'how to fish' for useful web-based teaching resources to help you succeed as a clinical teacher.     

Validation of the PHEEM instrument in a Danish hospital setting

The Postgraduate Hospital Educational Environment Measure (PHEEM) has been translated into Danish and then validated with good internal consistency by 342 Danish junior and senior hospital doctors. Four of the 40 items are culturally dependent in the Danish hospital setting. Factor analysis demonstrated that seven items are interconnected. This information can be used to shorten the instrument by perhaps another three items.     

Comparison of ambulatory and inpatient experiences in five specialties

This study compared medical student evaluations of ambulatory and inpatient components of third-year clerkships in internal medicine, obstetrics/gynaecology, paediatrics, psychiatry, and surgery. Seventy-two students completed evaluation questionnaires at the conclusion of each of five clerkships over 12 months. Although ambulatory and inpatient evaluations were comparable across all five specialties in several areas, ambulatory education was rated more favorably in scheduling, clear definition of student roles and responsibilities, appropriate supervision, timely and constructive feedback, being welcomed, and working in a non-threatening environment.Student feedback thus implied that ambulatory experiences were comparable to, or better than, inpatient experiences and suggests areas for improving clinical education in inpatient settings.     

Microburst Teaching and Learning

Amidst changes in the health care field with requirements for increased efficiency and limited time for teaching, there is a need for a teaching-learning model which maximizes the learning process and is exciting, fun, and motivating for both teacher and learner. Microburst Teaching and Learning is one strategy for combining various teaching styles and methods in 'bursts' with different learning styles to enhance the learning process.The model accommodates adult learning theory, adult attention span, learner motivation, the variety of learning styles found in learners, and the need for efficiency. Preliminary reactions to the Microburst Model indicate its appeal and motivating nature as a useful teaching-learning model.The next steps are to more critically evaluate the efficacy of the model for a broader range of clinical preceptors and to examine the variety of specific teaching strategies to determine which methods work best in specific settings. Because there are many potential teaching methods and teaching styles from which medical teachers can choose, a companion article outlining these specific methods and styles is currently in preparation.Weanticipate the article's publication within the next year.     

AMEE Education Guide no. 29: evaluating educational programmes

Evaluation has become an applied science in its own right in the last 40 years. This guide reviews the history of programme evaluation through its initial concern with methodology, giving way to concern with the context of evaluation practice and into the challenge of fitting evaluation results into highly politicized and decentralized systems. It provides a framework for potential evaluators considering undertaking evaluation. The role of the evaluator; the ethics of evaluation; choosing the questions to be asked; evaluation design, including the dimensions of evaluation and the range of evaluation approaches available to guide evaluators; interpreting and disseminating the findings; and influencing decision making are covered.     

Psychometric properties of the Dundee Ready Educational Environment Measure (DREEM) applied to medical residents

The Dundee Ready Educational Environment Measure (DREEM) is a measure of students' perceptions of the educational environment, but its performance in evaluating the educational environment in the residency setting has not yet been described. This study aimed at describing the psychometric performance of DREEM applied to medical residents. DREEM was applied to 97 residents from 12 training programs on four specialties in six institutions in three Brazilian cities. Psychometric measures included factor analysis, Cronbach's alpha coefficients, item-to-total correlations, t-test comparisons of scores between genders, institutions, specialties, and programs, correlations with the global score of the Quality of School Life Scale (concurrent validity), and test-retest reliability. Generalizability theory procedures were applied to a random subset of data. Programs (8) were the objects of measure, while institutions (6), specialties (4), raters-within-programs (40), and items-on-the-scale (50) were facets. Variance components, generalizability (G) and dependability (D) coefficients were calculated. Cronbach's alpha was 0.93. DREEM showed high discriminant and concurrent validities. Test-retest reliability was moderate. Interactions between programs, raters and items accounted for 68% of the total variance. G and D coefficients were 0.95 and 0.67, respectively. The instrument proved to be useful for relative comparisons at both resident and program level.     

Development of culture-sensitive clinical teacher evaluation sheet in the Japanese context

Aim: Many instruments for evaluating clinical teaching have been developed, albeit most in Western countries. This study aims to develop a validated cultural and local context sensitive instrument for clinical teachers in an East Asian setting (Japan), Japanese Clinical Teacher Evaluation Sheet (JaCTES).

Methods: A multicenter, cross-sectional evaluation study was conducted. We collected a total of 1368 questionnaires on 304 clinical teachers, completed by residents in 16 teaching hospitals. The construct validity was examined by conducting a factor analysis and using structural equation modeling (SEM). We also assessed the reliability using generalizability analysis and decision study.

Results: Exploratory factor analysis resulted in three-factor (role model, teaching activities, and accessibility) model including 18 items. Confirmatory factor analysis was performed, using SEM. The comparative fit index was 0.931 and the root mean square error of approximation was 0.087, meaning an acceptable goodness of fit for this model. To obtain a reliable dependability-coefficient of at least 0.70 or higher, 5–8 resident responses are necessary.

Discussion and conclusion: JaCTES is the first reported instrument with validity evidence of content and internal structure and high feasibility in Japan, an East Asian setting. Medical educators should be aware of the local context and cultural aspects in evaluating clinical teachers.     

Peer review improves psychometric characteristics of multiple choice questions

Abstract

Purpose: For new and emerging medical schools, developing a system to peer-review and evaluate the assessment processes through faculty development programs can be a challenge. This study evaluates the impact of peer-review practices on item analysis, reliability, and the standard error of measurement of multiple-choice questions for summative final examinations.

Methods: This study used a retrospective cohort design of two consecutive academic years in 2012 and in 2013. Psychometric analyses of multiple-choice questions of three summative final examinations in Medicine, Pediatrics, and Surgery for sixth year medical students at the College of Medicine Taif University were used. Formal peer review of multiple-choice questions began in 2013, using guidelines from the National Board of Medical Examiners. Psychometric analyses of multiple-choice questions included item analysis (item difficulty and item discrimination) and calculation of internal-consistency reliability and the standard error of measurement. Data analyses were conducted using Stata.

Results: Results showed significant improvement in psychometric indices, particularly item discrimination and reliability by .14 and .12 points, respectively, following the implementation of the peer review process across the three exams. Item difficulty remained unchanged for Pediatrics and Surgery.

Conclusion: Peer-review practices of multiple-choice questions using guidelines can lead to improved psychometric characteristics of items; these findings have implications for faculty development programs in improving item quality, particularly for medical schools in early stages of transforming assessment practices.     

Clinical teaching performance improvement of faculty in residency training: A prospective cohort study

Purpose: The purpose of this study is to investigate how aspects of a teaching performance evaluation system may affect faculty’s teaching performance improvement as perceived by residents over time.

Methods: Prospective multicenter cohort study conducted in The Netherlands between 1 September 2008 and 1 February 2013. Nine hundred and one residents and 1068 faculty of 65 teaching programs in 16 hospitals were invited to annually (self-) evaluate teaching performance using the validated, specialty-specific System for Evaluation of Teaching Qualities (SETQ). We used multivariable adjusted generalized estimating equations to analyze the effects of (i) residents’ numerical feedback, (ii) narrative feedback, and (iii) faculty’s participation in self-evaluation on residents’ perception of faculty’s teaching performance improvement.

Results: The average response rate over three years was 69% for faculty and 81% for residents. Higher numerical feedback scores were associated with residents rating faculty as having improved their teaching performance one year following the first measurement (regression coefficient, b: 0.077; 95% CI: 0.002–0.151; p?=?0.045), but not after the second wave of receiving feedback and evaluating improvement. Receiving more suggestions for improvement was associated with improved teaching performance in subsequent years.

Conclusions: Evaluation systems on clinical teaching performance appear helpful in enhancing teaching performance in residency training programs. High performing teachers also appear to improve in the perception of the residents.     

The versatility of progress testing assessed in an international context: a start for benchmarking global standardization?

Sharing and collaboration relating to progress testing already takes place on a national level and allows for quality control and comparisons of the participating institutions. This study explores the possibilities of international sharing of the progress test after correction for cultural bias and translation problems. Three progress tests were reviewed and administered to 3043 Pretoria and 3001 Maastricht medical students. In total, 16% of the items were potentially biased and removed from the test items administered to the Pretoria students (9% due to translation problems; 7% due to cultural differences). Of the three clusters (basic, clinical and social sciences) the social sciences contained most bias (32%), basic sciences least (11%). The differences that were found, comparing the student results of both schools, seem a reflection of the deliberate accentuations that both curricula pursue. The results suggest that the progress test methodology provides a versatile instrument that can be used to assess medical schools across the world. Sharing of test material is a viable strategy and test outcomes are interesting and can be used in international quality control.