CanMEDS evaluation in Canadian postgraduate training programmes: tools used and programme director satisfaction

Context The Royal College of Physicians and Surgeons of Canada (RCPSC) CanMEDS framework is being incorporated into specialty education worldwide. However, the literature on how to evaluate trainees in the CanMEDS competencies remains sparse. Objectives The goals of this study were to examine the assessment tools used and programme directors’ perceptions of how well they evaluate performance of the CanMEDS roles in Canadian postgraduate training programmes. Methods We conducted a web‐based survey of programme directors of RCPSC‐accredited training programmes. The survey consisted of two questions. Question 1 was designed to establish which assessment tools were used to assess each of the CanMEDS roles. Question 2 was intended to assess programme directors’ perceived satisfaction with CanMEDS evaluation in their programmes. Results A total of 149 of the eligible 280 programme directors participated in the survey. Programme directors used a variety of assessment tools to evaluate trainees in CanMEDS competencies. Programmes used more tools to evaluate the Medical Expert (mean = 4.03, standard deviation [SD] = 1.59) and Communicator (mean = 2.36, SD = 1.02) roles. Programme directors used the fewest tools for the Collaborator (mean = 1.75, SD = 1.10) and Manager (mean = 1.75, SD = 1.18) roles. More than 92% of the programmes used in‐training evaluation reports to evaluate all the CanMEDS roles. Programme directors were satisfied with their evaluation of the Medical Expert role, but less so with assessment of the other CanMEDS competencies. Conclusions This study demonstrates that Canadian postgraduate training programmes use a variety of assessment tools to evaluate the CanMEDS competencies. Programme directors are neutral or concerned about how the CanMEDS roles other than that of Medical Expert are evaluated in their programmes. Further efforts are required to establish best practice in CanMEDS evaluation.     

Defining characteristics of educational competencies

Context  Doctor competencies have become an increasing focus of medical education at all levels. However, confusion exists regarding what constitutes a competency versus a goal, objective or outcome.
Objectives  This article attempts to identify the characteristics that define a competency and proposes criteria that can be applied to distinguish between competencies, goals, objectives and outcomes.
Methods  We provide a brief overview of the history of competencies and compare competencies identified by international medical education organisations (CanMEDS 2005, Institute for International Medical Education, Dundee Outcome Model, Accreditation Council for Graduate Medical Education/American Board of Medical Specialties). Based upon this review and comparisons, as well as on definitions of competencies from the literature and theoretical and conceptual analyses of the underpinnings of competencies, the authors develop criteria that can serve to distinguish competencies from goals, objectives and outcomes.
Results  We propose 5 criteria which can be used to define a competency: it focuses on the performance of the end-product or goal-state of instruction; it reflects expectations that are external to the immediate instructional programme; it is expressible in terms of measurable behaviour; it uses a standard for judging competence that is not dependent upon the performance of other learners, and it informs learners, as well as other stakeholders, about what is expected of them.
Conclusions  Competency-based medical education is likely to be here for the foreseeable future. Whether or not these 5 criteria, or some variation of them, become the ultimate defining criteria for what constitutes a competency, they represent an essential step towards clearing the confusion that reigns.     

Communication skills competencies: definitions and a teaching toolbox

BACKGROUND: Doctors' interpersonal and communication skills correlate with improved health care outcomes. International medical organisations require competency in communication skills. The Accreditation Council for Graduate Medical Education (ACGME) developed a toolbox for assessing this competency and 5 others, yet none initially for teaching these skills. PURPOSE AND METHODS: The original focus in the development of the ACGME competencies was evaluation. This paper represents a significant step toward defining methods for teaching communication skills competencies. A total of 16 medical education leaders from medical schools worldwide, participating in the 2003 Harvard Macy Institute Program for Physician Educators, worked together to: (1) further define the ACGME competency in interpersonal and communication skills; (2) delineate teaching strategies for each level of medical education; and (3) create a teaching toolbox to integrate communication skills competencies into medical curricula. Four subgroups defined subcompetencies, identified teaching strategies for undergraduate, graduate and postgraduate medical training and brought their work to the larger group. The expanded communication competencies and teaching strategies were determined by a consensus of the larger group, presented to 80 Harvard Macy Scholars and Faculty for further discussion, then finalised by consensus. CONCLUSION: The teaching toolbox expands the ACGME core communication competencies, adds 20 subcompetencies and connects these competencies to teaching strategies at each level of medical training. It represents the collaboration and consensus of a diverse international group of medical education leaders in a variety of medical specialities and institutions, all involved in teaching communication skills. The toolbox is applicable globally across different settings and specialities, and is sensitive to different definitions of health care.     

The effectiveness of a computer-assisted instruction programme on communication skills of medical specialists in oncology

BACKGROUND: Although doctor--patient communication is important in health care, medical specialists are generally not well trained in communication skills. Conventional training programmes are generally time consuming and hard to fit into busy working schedules of medical specialists. A computer-assisted instruction (CAI) programme was developed -- 'Interact-Cancer' -- which is a time-efficient learning method and easily accessible at the workplace. OBJECTIVE: To investigate the effect of the CAI training, 'Interact-Cancer', on the communication behaviour of medical specialists, and on satisfaction of patients about their physician interaction. DESIGN: Consultations of medical specialists with cancer outpatients were videotaped at 4 specific stages, 2 before and 2 after Interact-Cancer, with intervals of 4 weeks. PATIENTS/PARTICIPANTS: Participants were 21 medical specialists, mainly internists, working in 7 hospitals, and 385 cancer outpatients. METHODS: Communication behaviour was assessed on 23 observation categories derived from the course content. Frequencies were rated as well as judgements about the quality of the performance of each target skill. Satisfaction was measured by the Medical Interview Satisfaction Scale. Data were analyzed by means of multilevel statistical methods. RESULTS: The behavioural assessment showed course effects on ratings of the physicians' quality of performance. No course effects were found on the frequencies of physicians' behaviours and on the patient satisfaction ratings. CONCLUSIONS: CAI is a promising method to supply medical specialists with postgraduate training of communication skills. The application of judgement ratings of communication behaviour proved to be valuable to evaluate course effects in real-life patient encounters.     

The rating of pre-clerkship examination questions by postgraduate medical students: an assessment of quality and relevancy to medical practice

OBJECTIVE: To determine whether postgraduate students are able to assess the quality of undergraduate medical examinations and to establish whether faculty can use their results to troubleshoot the curriculum in terms of its content and evaluation. SUBJECTS: First and second year family medicine postgraduate students. MATERIALS: A randomly generated sample of undergraduate medical examination questions. METHODS: Postgraduate students were given two undergraduate examinations which included questions with an item difficulty (ID) > 0.60. The students answered and then rated each question on a scale of 1-7. RESULTS: The percentage of postgraduate students answering each question correctly correlated significantly with the average perceived relevance (Examination 1: r=0.372; P < 0.05; Examination 2: r=0.458; P < 0.05). Questions plotted for average postgraduate/undergraduate performance ratio versus the average perceived relevance were significantly correlated (Examination 1: r=0.462; P < 0.01; Examination 2: r=0.458; P < 0.05). CONCLUSIONS: This study offers a method of validating question appropriateness prior to examination administration. The design has the potential to be used as a model for determining the relevancy of a medical curriculum.     

Components of postgraduate competence: analyses of thirty years of longitudinal data

CONTEXT: The conceptualisation and measurement of competence in patient care are critical to the design of medical education programmes and outcome assessment. OBJECTIVE: We aimed to examine the major components and correlates of postgraduate competence in patient care. METHODS: A 24-item rating form with additional questions about resident doctors' performance and future residency offers was used. Study participants comprised 4560 subjects who graduated from Jefferson Medical College between 1975 and 2004. They pursued their graduate medical education in 508 hospitals. We used a longitudinal study design in which the rating form was completed by programme directors to evaluate residents at the end of the first postgraduate year. Factor analysis was used to identify the underlying components of postgraduate ratings. Multiple regression, t-test and correlational analyses were used to study the validity of the components that emerged. RESULTS: Two major components emerged, which we labelled 'Knowledge and Clinical Capabilities' and 'Professionalism', and which addressed the science and art of medicine, respectively. Performance measures during medical school, scores on medical licensing examinations, and global assessment of Medical Knowledge, Clinical Judgement and Data-gathering Skills showed higher correlations with scores on the Knowledge and Clinical Capabilities component. Global assessments of Professional Attitudes and ratings of Empathic Behaviour showed higher correlations with scores on the Professionalism component. Offers of continued residency and evaluations of desirable qualities were associated with both components. CONCLUSIONS: Psychometric support for measuring the components of Knowledge and Clinical Capabilities, and Professionalism provides an instrument to empirically evaluate educational outcomes to medical educators who are in search of such a tool.     

Faculty attitudes towards medical communication and their perceptions of students' communication skills training at Dalhousie University

SETTING: Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada. OBJECTIVES: (1) To assess the attitudes of full-time clinical faculty members towards medical communication using the newly developed Attitudes Towards Medical Communication Scale; (2) to determine faculty members' perceptions of communications training for students and residents. METHODS: An anonymous self-completion survey was sent to 233 full-time clinical faculty members. The questionnaire asked about faculty attitudes towards medical communication, and assessed faculty members' views of student and resident training in communication. RESULTS: Faculty scored highly in the Attitudes Towards Medical Communication Scale, with a mean score of 51.5 (SD 4.1) out of a possible 60. In univariate analysis, rating of personal enjoyment of teaching, rating of the importance of teaching, and having attended at least one faculty communications workshop in the previous 5 years were significantly associated with higher scale scores. When these factors were assessed using linear regression, only having attended a workshop and higher rating of the importance of teaching remained significant. Faculty assessed student training in communications skills poorly overall. When assessing seven specific communications areas, more than 20% rated this training as poor for six of the areas for third- and fourth-year students and for five of the areas for residents. CONCLUSIONS: Clinical faculty at Dalhousie have very positive attitudes towards medical communication, and more highly positive attitudes are found in those who have attended a communications workshop. Despite this evidence that faculty appreciate the importance of medical communication skills, many assessed students' training in this curriculum area as poor.     

Effects of conventional and problem-based learning on clinical and general competencies and career development

Objective  To test hypotheses regarding the longitudinal effects of problem-based learning (PBL) and conventional learning relating to students' appreciation of the curriculum, self-assessment of general competencies, summative assessment of clinical competence and indicators of career development.
Methods  The study group included 2 complete cohorts of graduates who were admitted to the medical curriculum in 1992 (conventional curriculum, n  =   175) and 1993 (PBL curriculum, n  =   169) at the Faculty of Medicine, University of Groningen, the Netherlands. Data were obtained from student records, graduates' self-ratings and a literature search. Gender and secondary school grade point average (GPA) scores were included as moderator variables. Data were analysed by a stepwise multiple and logistic regression analysis.
Results  Graduates of the PBL curriculum scored higher on self-rated competencies. Contrary to expectations, graduates of the PBL curriculum did not show more appreciation of their curriculum than graduates of the conventional curriculum and no differences were found on clinical competence. Graduates of the conventional curriculum needed less time to find a postgraduate training place. No differences were found for scientific activities such as reading scientific articles and publishing in peer- reviewed journals. Women performed better on clinical competence than did men. Grade point average did not affect any of the variables.
Conclusions  The results suggest that PBL affects self-rated competencies. These outcomes confirm earlier findings. However, clinical competence measures did not support this finding.     

Professionalism and the basic sciences: an untapped resource

CONTEXT: Medical, technological and societal developments influence doctors' professional responsibilities and present challenges to educating medical students about professionalism. Medical education about professionalism generally focuses on behaviours and competencies which are taught primarily by clinicians in clinical courses and settings. DISCUSSION: Many professional competencies in medicine parallel those in science. We consider here whether medical professionalism can also be taught through the basic science courses which often initiate medical education, and which are typically taught by scientists. CONCLUSIONS: Like doctors, basic science faculty staff can teach professional competencies to medical students. Science faculty are well situated to teach professional competencies and should do so. They can model how to pursue evidence and manage conflicting information. They can also provide explicit messages to students about professional competencies and their value, and create learning objectives that reinforce those messages.     

Refugees and medical student training: results of a programme in primary care

CONTEXT: Medical schools have responded to the increasing diversity of the population of the USA by incorporating cultural competency training into their curricula. This paper presents results from pre- and post-programme surveys of medical students who participated in a training programme that included evening clinical sessions for refugee patients and related educational workshops. METHODS: A self-assessment survey was administered at the beginning and end of the academic year to measure the cultural awareness of participating medical students. RESULTS: Over the 3 years of the programme, over 133 students participated and 95 (73%) completed pre- and post-programme surveys. Participants rated themselves significantly higher in all 3 domains of the cultural awareness survey after completion of the programme. CONCLUSIONS: The opportunity for medical students to work with refugees in the provision of health care presents many opportunities for students, including lessons in communication, and scope to learn about other cultures and practise basic health care skills. An important issue to consider is the power differential between those working in medicine and patients who are refugees. To avoid reinforcing stereotypes, medical programmes and medical school curricula can incorporate efforts to promote reflection on provider attitudes, beliefs and biases.     

Medical students' attitudes towards and perception of the basic sciences: a comparison between students in the old and the new curriculum at the University Medical Center Utrecht,The Netherlands

OBJECTIVES: The attitudes towards the basic sciences of medical students enrolled in either of 2 different curricula at the University of Utrecht Medical School in The Netherlands were investigated. The purpose of this study was threefold: first, to compare students (beginning clerks) in a conventional and an innovative curriculum; second, to compare beginning clerks with advanced clerks; and third to compare the present results with those of 2 previous American and Canadian studies in which the same questionnaire was used. SETTING: Beginning clerks in the old and in the innovative curriculum, and advanced clerks in the old curriculum, rated 9 statements on a 5-point (disagree - agree) Likert scale. The statements assessed students' attitudes toward the basic sciences. RESULTS: The results showed that beginning clerks in our innovative curriculum, unlike those in a conventional curriculum, consider the basic sciences as somewhat less important for medical practice and do not think that as many biomedical facts as possible should be learned before entering clinical practice. On the other hand, students in the innovative curriculum are more excited by the faculty's teaching of the basic sciences. This latter result confirms the findings in a previous Canadian study. No significant differences were found between beginning and advanced clerks in the conventional curriculum. CONCLUSION: Students experience teaching of the basic sciences as more exciting when they are integrated in organ system blocks with clinical bearings, though they are somewhat less positive about the actual importance of these sciences.     

Information Management and Technology (IM&T): training provision in postgraduate medical education

Information Management and Technology (IM&T) is assuming a greater role within the modern NHS and there is an increasing need for members of the medical, and other health care, professions to receive appropriate training and education in these areas. Over half the Postgraduate Deans, Regional Advisers in General Practice and the Medical Royal Colleges have made a conscious decision to supply training in IM&T-related areas. These courses are open to a wide range of health care professionals. However, the number of reported places is not adequate for the large number of people who require the training. Less than half the organizations employed staff with a remit to provide training but over half provided courses in collaboration with educational establishments. As the medical undergraduate curriculum changes and incorporates training in computer skills, the links between the postgraduate institutions and medical schools must be fostered and developed. This will lead to increasing opportunities for health professionals both in the range of subjects and the number of places available.     

Learning in groups: Teamshaping in the teaching of medical psychology

OBJECTIVES: Although it is of growing importance in modern medicine, the ability to work in teams usually plays no explicit role in medical education. We compared the method of Teamshaping with 'traditional' teaching. DESIGN: Ninety-six students in 6 groups pursued a course in medical psychology with traditional teaching methods, while 46 students in 4 groups participated in Teamshaping. To evaluate the team approach, we used the Group Climate Questionnaire, Short Version (GCQ-S). SETTING: University of Vienna, Medical School. SUBJECTS: Third-year medical students. RESULTS: In terms of group climate both groups benefited from teaching, regardless of the method. Teamshaping, however, initially imposed a greater strain on the students than traditional teaching. This changed towards the end of the seminar. Teamshaping produced a stronger sense of commitment and a reduction of perceived conflicts. CONCLUSION: The model is recommended for teaching medical students because it not only extends traditional teaching towards small group learning but adds a competency in teamwork. It introduces students to a better working style and gives them an additional perspective for use in their future profession.     

Assessment of professional behaviour in undergraduate medical education: peer assessment enhances performance

OBJECTIVES: To examine whether peer assessment can enhance scores on professional behaviour, with the expectation that students who assess peers score more highly on professional behaviour than students who do not assess peers. METHODS: Undergraduate medical students in their first and second trimesters were randomly assigned to conditions with or without peer assessment. Of the total group of 336 students, 278 students participated in the first trimester, distributed over 31 tutorial groups, 17 of which assessed peers. The second trimester involved 272 students distributed over 32 groups, 15 of which assessed peers. Professional behaviour was rated by tutors on 3 dimensions: Task Performance; Aspects of Communication, and Personal Performance. The rating scale ranged from 1 (poor) to 10 (excellent). Data were analysed using multivariate repeated measures multilevel analysis. RESULTS: Assessment scores were found to have generally increased in the second trimester, especially the personal performance scores of students who assessed peers. In addition, female students were found to have significantly higher scores than male students. Conclusions: In undergraduate medical education, peer assessment has a positive influence on professional behaviour. However, the results imply that peer assessment is only effective after students have become adjusted to the complex learning environment.     

Developing a Korean communication skills attitude scale: comparing attitudes between Korea and the West

Objectives  We aimed to develop a Korean version of the Communication Skills Attitude Scale (CSAS) created by Rees et al . in order to elucidate the positive and negative aspects of Korean pre-medical and medical students' attitudes towards communication skills (CS) learning.
Methods  We performed two surveys. In the first of these, 325 pre-medical and medical students completed a translated version of the CSAS. In the second survey, 257 medical students and doctors-in-training answered five open-ended questions to obtain more qualitative data about their attitudes.
Results  Principal component analysis with direct oblimin rotation performed with the data from the first survey produced the following five factors: facilitation of interpersonal skills; doubts about the importance of CS learning in medicine; motivation; negative attitudes towards assessment, and overconfidence. Results from the second survey indicated that facilitation and importance within a medical context were two core attitudinal factors and suggested some modification to the CSAS to improve its fit for Korean pre-medical and medical students.
Conclusions  Using a Korean version of the CSAS (CSAS-K), we determined five factors that revealed a somewhat complex attitude structure among students towards CS learning. The CSAS required some modification, possibly because CS teaching and learning in Korea are in the development stage. Finally, the educational implications of the results are discussed.     

Attachment theory in health care: the influence of relationship style on medical students' specialty choice

INTRODUCTION: Converging sources suggest that patient-provider relationships in primary care are generally of greater intensity and duration than those in non-primary care specialties. In this study, we hypothesised that Year 2 medical students whose close relationships were characterised by security and flexibility would be more likely than students who were less comfortable in close relationships to plan to pursue primary over non-primary care postgraduate training. METHODS: We determined the relationship styles and demographic characteristics of 144 Year 2 medical students. We also gathered information regarding their predicted choices of postgraduate training, which were clustered into primary or non-primary care categories. We compared student choices with respect to their interpersonal relationship styles based on attachment theory. RESULTS: Prevalences of attachment styles were similar to those found in the general population, with 56% of students rating themselves as having a secure relationship style. Students with a secure style were more likely to choose primary care (61%) over non-primary care compared to those whose styles were characterised by self-reliance, support-seeking or caution (41% chose primary care). Compared to those with a secure relationship style, students with a cautious style [OR = 5.9 (1.9, 18.7)] and students with a self-reliant style [OR = 2.4 (0.96, 5.9)] were more likely to choose non-primary over primary care, after controlling for gender. CONCLUSIONS: Assessing relationship styles using attachment theory is a potentially useful way to understand and counsel medical students about specialty choice.     

Are some of the challenging aspects of the CanMEDS roles valid outside Canada?

CONTEXT: Many countries have adopted the CanMEDS roles. However, there is limited information on how these apply in an international context and in different specialties. OBJECTIVES: To survey trainee and specialist ratings of the importance of the CanMEDS roles and perceived ability to perform tasks within the roles. METHODS: We surveyed 8749 doctors within a defined region (eastern Denmark) via a single-issue, mailed questionnaire. Each of the 7 roles was represented by 3 questionnaire items to be rated for perceived importance and confidence in ability to perform the role. RESULTS: Responses were received from 3476 doctors (42.8%), including 190 interns, 201 doctors in the introductory year of specialist training, 529 residents and 2152 specialists. The overall mean rating of importance (on a scale of 1-5) of the aspects of competence described in the CanMEDS roles was 4.2 (standard deviation 0.6) and did not differ between trainee groups and specialists. Mean ratings of confidence were lower than ratings of importance and increased across the groups from interns to specialists. Differences between specialty groups were evident in both importance and confidence for many of the roles. For laboratory, technical and, to a lesser extent, cognitive specialties, the role of Health Advocate scored the lowest in importance. For general medicine specialties, the roles of Medical Expert, Collaborator, Manager and Scholar all scored lower for importance and confidence. CONCLUSIONS: This study provides a sketch of the content and construct validity of the CanMEDS roles in a non-Canadian setting. More research is needed in how these aspects of competence can be best taught and applied across specialties in different jurisdictions.     

Medical students' perceptions of identity in communication skills training: a qualitative study

Objectives  Communication skills training in undergraduate medical education is considered to play an important role in medical students' formation of their professional identity. This qualitative study explores Year 1 students' perceptions of their identities when practising communication skills with real patients.
Methods  A total of 23 individual semi-structured interviews and two focus group discussions were conducted with 10 students during their first year of communication skills training. All interviews and discussions were audio-recorded, transcribed and analysed for emergent themes relating to identity.
Results  Students struggled to communicate professionally with patients because of a lack of clinical knowledge and skills. Consequently, students enacted other identities, yet patients perceived them differently, causing conversational ambiguities.
Discussion  Students' perceptions challenge educational goals, suggesting that there is limited potential for the formation of professional identity through early training. Teacher-doctors must acknowledge how students' low levels of clinical competence and patients' behaviour complicate students' identity formation.     

The relationship between medical students' attitudes towards communication skills learning and their demographic and education-related characteristics

INTRODUCTION: The General Medical Council (GMC) has stressed the importance of medical students' attitudes towards learning. However, few studies have explored medical students' attitudes towards communication skills learning. This study explores the relationship between the attitudes of medical students at two different schools and their demographic and education-related characteristics. METHODS: A total of 490 medical students from the Universities of Nottingham (Years 1 and 2) and Leicester (Year 1) completed the 26-item Communication Skills Attitude Scale (CSAS) and a personal details questionnaire satisfactorily. The relationships between students' attitudes and their demographic and education-related characteristics were analysed separately for Nottingham and Leicester students using both univariate and multivariate statistics. RESULTS: The attitudes of Nottingham and Leicester medical students towards communication skills learning were significantly associated with a number of demographic and education-related characteristics. Both Nottingham and Leicester students with more positive attitudes towards communication skills learning tended to be female, tended to think their communication skills needed improving and tended not to have parents who were doctors. Both Nottingham and Leicester students with more negative attitudes towards communication skills learning tended to think their communication skills did not need improving. DISCUSSION: The results indicate that medical students' attitudes towards communication skills learning are associated with their demographic and education-related characteristics. These findings have a number of implications for educational practice and further research and these are discussed in this paper.