Conceptualization and measurement of clinical competence of residents: a brief rating form and its psychometric properties

Conceptualization and measurement of clinical competence of residents are of interest to medical educators. Yet there is a scarcity of operational tools with satisfactory psychometric support for measuring clinical competence. In this study, we investigated the underlying structure, criterion-related validity and alpha reliability of a brief rating form (20 items) developed to assess clinical competence of residents. The study sample consisted of 882 physicians (654 men, 228 women) in postgraduate training at Thomas Jefferson University Hospital between 1998 and 2000. Construct validity of the form was supported by factor analysis. Two relevant factors emerged: 'Knowledge, Data-Gathering and Processing Skills', and 'Interpersonal Skills and Attitudes'. Criterion-related validity was supported by significant linear associations between factor scores and performance on the medical licensing examinations. Alpha reliability coefficients for the two factors were 0.98 and 0.97, respectively. This brief rating form can be employed as one measure to evaluate clinical competence of residents with reasonable confidence in its measurement properties.     

Primary care physicians in public and private sectors perceive different learning needs

We compared the perceived learning needs of primary care physicians from the public and private sectors who responded to a questionnaire before taking educational courses in Family Medicine. They rated their perceived learning needs on 71 items of clinical practices and practice management on a scale of 1-10. The ratings of their learning needs were closely related to the perceived needs of their daily work. The private physicians gave higher ratings to most items. Both groups of physicians shared similar least-preferred items (e.g. suturing, plastering, taking Pap smears) but had very different most-preferred ones. Public physicians wished to improve their care of individual patients (e.g. skin, eye, ear-nose-throat problems). Private physicians were more concerned with professional development to improve their practice (e.g. audits, counselling, adult learning). Organizers of educational programmes should assess and discuss with physicians their expected learning needs at the planning stage of a programme.     

Alternative learning environments: what do they contribute to professional development of medical students?

Special study modules provide opportunities for students to develop lifelong learning skills and develop areas of interest. The GMC also recommends opportunities to study topics not included in the core medical curriculum. This paper reports the evaluation of modules based within alternative learning environments developed to provide students with experiences outside traditional medical, scientific or academic cultures. The attachment was highly rated as contributing to professional development, provided novel learning experiences, and was stimulating for both students and supervisors. All students achieved new skills that they felt were pertinent to their personal development. Assessment was by standardized pro forma, including generic transferable skills and module specific outcomes. Whilst overall assessment grades were comparable to similar course components, unease amongst both students and supervisors was expressed reflecting anxieties in the diversity of workloads and assessment. Attempts to standardize assessment across the diversity of modules did not reduce anxieties and potentially detracted from the learning experiences.     

Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees

As a result of changes to surgical training in the United Kingdom, there has been a significant reduction in the number of hours worked by surgical trainees per week and in the total duration of surgical training. For trainees to achieve an acceptable level of surgical operative competence, the learning environment in the operating theatre needs to be optimized. In this context, an instrument to measure the learning environment in the operating theatre has been developed. Through a review of the literature and exploratory interviews with trainers and trainees, items that contribute to the learning environment in the operating theatre were identified. Based on these items a questionnaire was drawn up. For validation purposes, the questionnaire was circulated amongst basic surgical trainees on the Aberdeen rotation. This instrument was found to have high reliability and validity. Its potential usefulness for informing committees responsible for overseeing surgical training is discussed.     

A short questionnaire to evaluate the effectiveness of tutors in PBL: validity and reliability

The tutor plays a central role in problem-based learning (PBL). In many PBL-curricula the effectiveness of the tutor is evaluated in order to provide tutors with feedback. In the literature, several tutor evaluation instruments have been described. The problem with many of these instruments is that they are quite long, due to which students become 'tired' of filling them out. Using a short questionnaire is more convenient for students, but the question is whether such a short instrument can be valid and reliable. The purpose of this article is to demonstrate the validity and reliability of a short questionnaire (11 items), representing five underlying factors. A confirmatory factor analysis was carried out to assess the adequacy of the five factors. The results demonstrated that the five factor model had a good fit to the data. The alpha coefficients also demonstrated acceptable levels. In conclusion, the short tutor evaluation instrument (11 items) is reliable and valid and can be used for formative and summative purposes.     

Measuring exposure to bullying and harassment in health professional students in a clinical workplace environment: Evaluating the psychometric properties of the clinical workplace learning NAQ-R scale

Abstract

Background: Instruments that measure exposure to bullying and harassment of students learning in a clinical workplace environment (CWE) that contain validity evidence are scarce. The aim of this study was to develop such a measure and provide some validity evidence for its use.

Method: We took an instrument for detecting bullying of employees in the workplace, called the Negative Acts Questionnaire – Revised (NAQ-R). Items on the NAQ-R were adapted to align with our context of health professional students learning in a CWE and added two new factors of sexual and ethnic harassment. This new instrument, named the Clinical Workplace Learning NAQ-R, was distributed to 540 medical and nursing undergraduate students and we undertook a Confirmatory Factor Analysis (CFA) to investigate its construct validity and factorial structure.

Results: The results provided support for the construct validity and factorial structure of the new scale comprising five factors: workplace learning-related bullying (WLRB), person-related bullying (PRB), physically intimidating bullying (PIB), sexual harassment (SH), and ethnic harassment (EH). The reliability estimates for all factors ranged from 0.79 to 0.94.

Conclusion: This study provides a tool to measure the exposure to bullying and harassment in health professional students learning in a CWE.     

A comparison of clinical teaching evaluations by resident and peer physicians

The purpose of this study was to compare the reliability of inpatient teaching evaluations by resident and peer physicians on Mayo internal medicine hospital services. Three resident and three peer evaluators observed 10 consecutively chosen attending physicians on the Mayo hospital services. Evaluations by resident and peer physicians were compared in terms of mean scores. Kendall's coefficient of concordance (KCC) was used to summarize inter-rater reliabilities and Cronbach's coefficient alpha was used to determine internal consistencies of evaluations by residents and peers. Results of this study revealed that mean scores of the 13 evaluation items were generally higher for resident than peer physicians. None of the items completed by residents had KCC scores >0.5, whereas 10 of the items completed by peers had KCC scores >0.5. Likewise, none of the residents' items had KCC p-values <0.05, whereas nine of the peers' items had KCC p-values <0.05. The overall internal consistency was higher for peers (alpha = 0.76) than for residents (alpha = 0.71). In conclusion, resident physicians uniformly rate faculty highly. Furthermore, peer evaluations yield higher inter-rater and internal reliabilities than resident evaluations, indicating that peer physicians are more reliable than residents for assessing bedside teaching.     

Attitudes among students and teachers on vertical integration between clinical medicine and basic science within a problem-based undergraduate medical curriculum

Important elements in the curriculum at the Faculty of Health Sciences in Link?ping are vertical integration, i.e. integration between the clinical and basic science sections of the curriculum, and horizontal integration between different subject areas. Integration throughout the whole curriculum is time-consuming for both teachers and students and hard work is required for planning, organization and execution. The aim was to assess the importance of vertical and horizontal integration in an undergraduate medical curriculum, according to opinions among students and teachers. In a questionnaire 102 faculty teachers and 106 students were asked about the importance of 14 different components of the undergraduate medical curriculum including vertical and horizontal integration. They were asked to assign between one and six points to each component (6 points = extremely important for the quality of the curriculum; 1 point = unimportant). Students as well as teachers appreciated highly both forms of integration. Students scored horizontal integration slightly but significantly higher than the teachers (median 6 vs 5 points; p=0.009, Mann-Whitney U-test), whereas teachers scored vertical integration higher than students (6 vs 5; p=0.019, Mann-Whitney U-test). Both students and teachers considered horizontal and vertical integration to be highly important components of the undergraduate medical programme. We believe both kinds of integration support problem-based learning and stimulate deep and lifelong learning and suggest that integration should always be considered deeply when a new curriculum is planned for undergraduate medical education.     

Evaluating an instrument for the peer review of inpatient teaching

The purpose of this study was to assess an instrument for the peer review of inpatient teaching at Mayo. The Mayo Teaching Evaluation Form (MTEF) is an instrument, based on the Stanford seven-category educational framework, which was developed for the peer review of inpatient teaching. The MTEF has 28 Likert-scaled items derived from the Stanford Faculty Development Program form (SFDP-26), the Mayo electronic evaluation form and three additional items. In this study three physician-evaluators used the MTEF to evaluate 10 attending physicians on the Mayo general internal medicine hospital services. Cronbach's alphas were used to assess the internal consistency of the MTEF, and Kendall's coefficient of concordance was used to summarize the inter-rater reliability. Results of this study reveal that the MTEF is internally consistent, based on average ratings across all evaluators (Cronbach's alpha=0.894). Stanford categories with the highest alphas are Self-Directed Learning, Learning Climate, Communication of Goals, and Evaluation. Categories with lower alphas are Feedback, Understanding and Retention, and Control of Teaching Session. Additionally, the majority of items on the MTEF show significant agreement across all evaluators, and teacher enthusiasm was among the most reliable items. In conclusion, the MTEF is overall internally consistent for the peer review of inpatient teaching at Mayo. Hence, the MTEF may be a useful element in the peer evaluation of teaching at our institution.     

How effective is a clerkship as a learning environment?

To assess the educational quality of a traditional clerkship, a questionnaire was administered to 28 students at the end of their 10-week surgical clerkship. The questionnaire contained 116 items on learning objectives, patient problems encountered by students, clinical skills performed, feedback received and amounts of time spent on various activities. The students indicated that they had adequate ability to correctly analyse and manage patient problems. However, the standard deviations and ranges show that individual students' exposure to various patient problems was insufficient. Students generally saw too few emergency patients. The frequencies of performance of diagnostic and therapeutic skills varied widely among students. Most of the feedback received by students was given by residents rather than faculty physicians. Students spent considerable time on activities of limited educational value. It appears that learning during a clerkship occurs rather haphazardly. There are indications that the 'educational exposure' varies substantially from student to student.     

AMEE Medical Education Guide No. 23 (Part 2): Curriculum,environment, climate,quality and change in medical education - a unifying perspective

This paper looks at five focal terms in education - curriculum, environment, climate, quality and change - and the interrelationships and dynamics bemeen and among them. It emphasizes the power and utility of the concept of climate as an operationalization or manifetation of the curriculum and the other three concepts. Ideas pertaining w the theory of climate and its measurement can provide a greater understanding of the medical cumadurn. The environment is an impoltant detemzinant of behaviour. Environment is perceived by students and it is perceptions of environment that are related w behaviour. The environment, as perceived, may be designated as climate. It is argued that the climate is the soul and spirit of the medical school environment and curriculum. Students' experiences of the climate of their medical education environment are related w their achievements, sangaction and success. Measures of educational climate are reviewed and the possibilities of new climate measures for medical education are discussed. These should take account of current trends in medical education and curricula. Measures of the climate may subdivide it inw dzfferent components giving, for example, separate assessment of so-called Faculty Press, Student Press, Administration Press and Physical or Material Environmental Press. Climate measures can be used in different modes with the same stakeholders. For example, students may be asked to report, first, their perceptions of the actual environment they have experienced and, second, w report on their ideal or preferred environment. The same climate index can be used with different stakeholders giving, for example, staff and student comparisons. The climate is important for staff as well as for students. The organizational climate that teaching staff experience in the work environment that they inhabit is important for their well-being, and that of their students. The medical school is a learning organization evolving and changing in the illuminative evaluation it makes of its environment and its curriculum through the action research studies of its climate. Consderations of climate in the medical school along the lines of continuous quality improvement and innovation are likely to further the medical school as a learning organization with the attendant benefits. Unless medical schools become such learning organizations their quality of health and their longevity may be threatened.     

The process of developing a research questionnaire to measure attitudes to shared learning

This article discusses the process by which a questionnaire was developed specifically to measure attitude to shared learning. Over a three-year period the attitude questionnaire was developed and tested using pre-registration students who were engaged in a shared learning programme. These pre-registration students were from three professional groups, namely occupational therapy (OT), diagnostic radiography (DR), and therapeutic radiography (TR). While the study hypothesis was that shared learning would improve attitudes over the three-year period, the focus of this article is to discuss the process by which the research instrument was developed. The rationale for focusing on the process is a simple one. After a diligent literature search it was clear that there were no attitude questionnaires specific to attitude measurement in relation to shared learning. Thus without an already validated attitude to a shared learning questionnaire, it became imperative to create systematically a credible attitude measure of shared learning. The process centred on understanding how monadic scales are constructed and used to measure attitudes and perceptions of participants. Amongst a number of processes to be followed was the need also to check the reliability of the developed questionnaire. This was done using the Cronbach alpha coefficient. A Cronbach alpha coefficient of 0.7 was set as a minimum reliability requirement, which was duly satisfied. The outcome of this process was that it very useful to learn how to produce a suitable instrument to measure attitude to shared learning and to add in a small way to the range of attitude questionnaires available.     

AMEE Medical Education Guide No. 23 (Part 1): Curriculum,environment, climate,quality and change in medical education-a unifying perspective

This paper looks at five focal terms in education - curriculum, environment, climate, quality and change - and the interrelationships and dynamics between and among them. It emphasizes the power and utility of the concept of climate as an operationalization or manifestation of the curriculum and the other three concepts. Ideas pertaining to the theory of climate and its measurement can provide a greater understanding of the medical curriculum. The learning environment is an important determinant of behaviour. Environment is perceived by students and it is perceptions of environment that are related to behaviour. The environment, as perceived, may be designated as climate. It is argued that the climate is the soul and spirit of the medical school environment and curriculum. Students' experiences of the climate of their medical education environment are related to their achievements, satisfaction and success. Measures of educational climate are reviewed and climate measures for medical education are discussed. These should take account of current trends in medical education and curricula. Measures of the climate may subdivide it into different components giving, for example, a separate assessment of so-called Faculty Press, Student Press, Administration Press and Physical or Material Environmental Press. Climate measures can be used in different modes with the same stakeholders. For example, students may be asked to report, first, their perceptions of the actual environment they have experienced and, second, to report on their ideal or preferred environment. The same climate index can be used with different stakeholders giving, for example, staff and student comparisons. In addition to the educational climate of the environment that students inhabit, it is important to consider the organizational climate of the work environment that staff inhabit. This organizational climate is very significant, not only for staff, but for their students, too. The medical school is a learning organization evolving and changing in the illuminative evaluation it makes of its environment and its curriculum through the action research studies of its climate. Considerations of climate in the medical school, along the lines of continuous quality improvement and innovation, are likely to further the medical school as a learning organization with the attendant benefits. Unless medical schools become such learning organizations, their quality of health and their longevity may be threatened.     

Young physicians' competences from different points of view

This paper describes a baseline evaluation of medical education focusing on the competences of young physicians. It was conducted in the context of a reform mandated for medical programs at Austrian state universities. The study investigated the goals of medical education (ideal situation) as well as the real situation. Furthermore, learning methods used by medical students were analyzed. A multiple-perspective procedure was applied and questionnaire data from students, graduates, university teachers, and clinical supervisors of young graduates were collected at the University of Graz. A high discrepancy was found between the goals of medical teaching and the educational effectiveness of achieving them. Competences of young physicians were rated to be relatively low -- thereby external ratings were systematically lower than self-ratings by students and graduates. Reading passages from required textbooks is the dominant learning behavior among students. Based on the results, a gap between the ideal and the actual medical competences imparted at university can be identified. With regard to the generally low-rated competences of young physicians, the precondition for practical requirements hardly seems to be optimal.     

South Cheshire Local Multi-disciplinary Evidence Centre: an evaluation

The South Cheshire Local Multidisciplinary Evidence Centre (LMEC) was a two-year project commenced in March 1998 and completed in April 2000. The project aimed to develop an information service to enable all primary and community care staff in South Cheshire to access high-quality evidence and thus to improve patient care. The LMEC gave access from the workplace to both physical and electronic resources to support clinical governance and lifelong learning, with a strong emphasis on evidence-based material. Automation of the library catalogue enabled its inclusion on the website. The project developed enquiry and document delivery services and provided training on using the LMEC and on critical appraisal. An evaluation carried out at the end of the project showed that over 120 primary and community care staff had used the LMEC and were positive about the service. As Clinical Governance, the NHSnet, and plans for continuing professional development are implemented, the LMEC is one model for a Local Health Information Service outlined in the NHS IT strategy.     

Lessons learned about empathy during our interview seasons

Abstract

Empathy is one of the most valuable skills sought by patients from their physicians as well as by educators from their trainees. However, in medical education there is a general concern that, if not cultivated among students and residents, empathy may decline. We have assessed empathy self-perception among potential candidates who were applying to our internal medicine residency program. Interestingly, we observed that they have a good understanding and great appreciation of empathy values. Our candidates expressed that as medical students they were sympathetic and with experience, they are becoming more empathetic. They also depicted the different ways in which they learned about empathy. The lessons we learned from them lead us to conclude that the development of empathy is a multifaceted and a lifelong process, which must be cultivated during training years through many different modalities. We suggest that the patient’s story, coaching skills and self-reflection, might be the most powerful ways of learning empathy.     

Contextual attributes promote or hinder self-regulated learning: A qualitative study contrasting rural physicians with undergraduate learners in Japan

Background and objectives: Previous studies support the notion that East Asian medical students do not possess sufficient self-regulation for postgraduate clinical training. However, some East Asian physicians who are employed in geographically isolated and educationally underserved rural settings can self-regulate their study during the early phase of their postgraduate career. To explore the contextual attributes that contribute to self-regulated learning (SRL), we examined the differences in self-regulation between learning as an undergraduate and in a rural context in East Asia.

Methods: We conducted interviews and diary data collection among rural physicians (n?=?10) and undergraduates (n?=?11) in Japan who undertook self-study of unfamiliar diseases. We analyzed three domains of Zimmerman’s definition of SRL: learning behaviors, motivation, and metacognition using constructivist grounded theory.

Results: Rural physicians recognized their identity as unique, and as professionals with a central role of handling diseases in the local community by conducting self-study. They simultaneously found themselves being at risk of providing inappropriate aid if their self-study was insufficient. They developed strategic learning strategies to cope with this high-stakes task. Undergraduates had a fear of being left behind and preferred to remain as one of the crowd with students in the same school year. Accordingly, they copied the methods of other students for self-study and used monotonous and homogeneous strategies.

Conclusions: Different learning contexts do not keep East Asian learners from being self-regulated. Awareness of their unique identity leads them to view learning tasks as high-stakes, and to initiate learning strategies in a self-regulated manner. Teacher-centered education systems cause students to identify themselves as one of the crowd, and tasks as low-stakes, and to accordingly employ non-self-regulated strategies.     

Demonstration of the effectiveness and acceptability of self-study module use in residency education

Educators face increasing challenges to promote lifelong learning skills, to include new content areas in an already full curriculum and to maximize limited resources for curriculum implementation. Self-study modules (hereafter modules) offer potential solutions. Three modules on preventive medicine topics were evaluated in Family Medicine residencies. A retrospective pre-/post-test of a resident's ability to meet the module's objectives was used for evaluation. Additionally, residents rated the appropriateness and acceptability of the modules, their preference for 13 methods of learning, and completed a multiple-choice knowledge test. This study demonstrates the effectiveness of modules at multiple levels of evaluation in accordance with a modified version of Kirkpatrick's hierarchy of levels of evaluation. Residents found the modules to be acceptable and useful. Significant gains were seen in residents' abilities to meet objectives. The multiple-choice knowledge test was used to demonstrate mastery of the module materials at an appropriate performance level for future practitioners. Module use was in the top five choices of preferred learning methods. No correlation was seen between residents' preference for learning using modules and educational outcomes. Modules are an effective and acceptable learning method for residents. Even those who prefer other learning methods show improved educational outcomes.