Senior medical students' appraisal of CanMEDS competencies

CONTEXT: In 2003 the Dutch Central College of Medical Specialties presented guidelines for the modernisation of all medical specialty training programmes in the Netherlands. These guidelines are based to a large extent on the CanMEDS (Canadian Medical Education Directives for Specialists) 2000 model, which defines 7 roles for medical specialists. This model was adjusted to the Dutch situation. The roles were converted to 7 fields of competency: Medical Performance; Communication; Collaboration; Knowledge and Science; Community Performance; Management, and Professionalism. OBJECTIVE: As changes in postgraduate training will probably be most effective if future trainees recognise their value, we set out to determine how senior medical students rated these fields of competency in terms of their importance. METHODS: We carried out a study at University Medical Centre (UMC) Utrecht, the Netherlands, in which 80 Year 6 medical students answered a questionnaire in which they rated the importance of each of 28 key competencies within the 7 competency fields. RESULTS: Although all key competencies were regarded as important (averages > or = 3.8), Professionalism and Communication scored highest on the student ratings. Management was assessed as least important. CONCLUSIONS: It is interesting that medical students acknowledged the importance of competencies other than those involving medical expertise and performance. It confirms the opinion that educating doctors is currently viewed as much more than providing theoretical and clinical knowledge and skills. The CanMEDS framework is appreciated by Dutch medical students. The fact that all competencies are seen as important adds to their face validity and therefore to their usefulness as a basis for postgraduate training.     

The sum of the parts detracts from the intended whole: competencies and in-training assessments

Objectives  Despite the fact that Canadian residency programmes are required to assess trainees' performance within the context of the CanMEDS Roles Framework, there has been no inquiry into the potential relationship between residents' perceptions of the framework and their in-training assessments (ITA). Using data collected during the study of ITA, we explored residents' perceptions of these competencies.
Methods  From May 2006–07, a purposive sample of 20 resident doctors from internal medicine, paediatrics, and surgery were interviewed about their ITA experiences. Data collection and analysis proceeded in an iterative fashion consistent with grounded theory. In April 2008, a summary of recurrent themes was presented during a focus group interview of another five residents to afford further elaboration and refinement of thematic findings.
Results  The in-training assessment report (ITAR) was perceived as a primary source of residents' information on CanMEDS. Residents' familiarity with the set of competencies appeared to be quite limited and they possessed narrow definitions of the roles. Several trainees questioned the framework's relevance and some appeared confused about the overlapping nature of the roles. Although residents viewed the central Medical Expert role as the most relevant and important competency, they incorrectly perceived it as only involving the acquisition of medical and scientific knowledge. A visual rhetorical analysis of a typical ITAR suggests that the visual features found within this assessment tool may be misrepresenting the framework and the centrality of the Medical Expert role.
Conclusions  Resident doctors' knowledge of CanMEDS was found to be limited. The visual structure of the ITAR appears to be a factor in residents' apparent distortion of the CanMEDS construct from its original holistic philosophy.     

Differences in curriculum emphasis in US undergraduate and generalist residency education programmes

OBJECTIVES: Little attention has been paid to the differential emphasis undergraduate and graduate medical education programmes place on the broad competencies that will be needed for practice in an increasingly managed health care environment. The purpose of this study was to determine differences in emphasis that undergraduate and primary care graduate medical education programmes are currently placing on 33 broad practice competencies, compared with the emphasis they ideally would like to give them, and the barriers they perceive to curriculum change. DESIGN: Subjects were surveyed by mailed questionnaire. A reminder postcard and follow-up mailing were sent to non-respondents. SETTING: US allopathic medical schools. SUBJECTS: Academic deans identified by the Association of American Medical Colleges (AAMC) and generalist (family medicine, internal medicine, paediatrics and obstetrics-gynaecology) residency programme directors identified by the American Council on Graduate Medical Education (ACGME). RESULTS: Findings revealed that residency programmes placed greater emphasis on the study's broad curriculum topics than did undergraduate medical education programmes. Statistically significant differences were found in current emphasis for 12 topics and ideal emphasis for six topics. Both groups identified an already crowded curriculum and inadequate funding as the top two barriers to curriculum change. CONCLUSIONS: The differences in curriculum emphases and perceived barriers to curriculum change most probably reflect the different realities of undergraduate and graduate medical education programmes, i.e. academics vs. a focus on immediate practice realities.     

The accountability of clinical education: its definition and assessment

BACKGROUND: Medical education is not exempt from increasing societal expectations of accountability. Competition for financial resources requires medical educators to demonstrate cost-effective educational practice; health care practitioners, the products of medical education programmes, must meet increasing standards of professionalism; the culture of evidence-based medicine demands an evaluation of the effect educational programmes have on health care and service delivery. Educators cannot demonstrate that graduates possess the required attributes, or that their programmes have the desired impact on health care without appropriate assessment tools and measures of outcome. OBJECTIVE: To determine to what extent currently available assessment approaches can measure potentially relevant medical education outcomes addressing practitioner performance, health care delivery and population health, in order to highlight areas in need of research and development. METHODS: Illustrative publications about desirable professional behaviour were synthesized to obtain examples of required competencies and health outcomes. A MEDLINE search for available assessment tools and measures of health outcome was performed. RESULTS: There are extensive tools for assessing clinical skills and knowledge. Some work has been done on the use of professional judgement for assessing professional behaviours; scholarship; and multiprofessional team working; but much more is needed. Very little literature exists on assessing group attributes of professionals, such as clinical governance, evidence-based practice and workforce allocation, and even less on examining individual patient or population health indices. CONCLUSIONS: The challenge facing medical educators is to develop new tools, many of which will rely on professional judgement, for assessing these broader competencies and outcomes.     

Skills and attributes of directors of educational programmes

OBJECTIVES: The purpose of this study was to obtain a list of desirable skills and attributes of prospective educational programme directors as judged by potential employers of future directors. DESIGN: An international mail survey was conducted worldwide, using a one-page, open-ended questionnaire. SETTING: University of Illinois at Chicago, USA. SUBJECTS: Deans, department heads and chief executive officers from the various health professions. RESULTS: A total of 139 respondents completed the questionnaire (22. 3% response rate). Responses were remarkably similar across health professions. The nine leading skills were: oral communication skills, interpersonal skills, competent practitioner, educational goal-definition skills, educational design skills, problem-solving and decision-making skills, team worker and building skills, written communication skills and fiscal manager and budgeting skills. The three leading personal attributes were: visionary, flexible and open-minded and trustworthy and value-driven individual. CONCLUSIONS: The complete list of skills and attributes can be used by employers or administrators to build checklist items when hiring or reviewing programme directors or to plan and evaluate training programmes. The importance of leadership compared to managerial responsibilities is discussed as well as the need to train future educational programme directors.     

Defining characteristics of educational competencies

Context  Doctor competencies have become an increasing focus of medical education at all levels. However, confusion exists regarding what constitutes a competency versus a goal, objective or outcome.
Objectives  This article attempts to identify the characteristics that define a competency and proposes criteria that can be applied to distinguish between competencies, goals, objectives and outcomes.
Methods  We provide a brief overview of the history of competencies and compare competencies identified by international medical education organisations (CanMEDS 2005, Institute for International Medical Education, Dundee Outcome Model, Accreditation Council for Graduate Medical Education/American Board of Medical Specialties). Based upon this review and comparisons, as well as on definitions of competencies from the literature and theoretical and conceptual analyses of the underpinnings of competencies, the authors develop criteria that can serve to distinguish competencies from goals, objectives and outcomes.
Results  We propose 5 criteria which can be used to define a competency: it focuses on the performance of the end-product or goal-state of instruction; it reflects expectations that are external to the immediate instructional programme; it is expressible in terms of measurable behaviour; it uses a standard for judging competence that is not dependent upon the performance of other learners, and it informs learners, as well as other stakeholders, about what is expected of them.
Conclusions  Competency-based medical education is likely to be here for the foreseeable future. Whether or not these 5 criteria, or some variation of them, become the ultimate defining criteria for what constitutes a competency, they represent an essential step towards clearing the confusion that reigns.     

Assessment of multiple physician competencies in postgraduate training: utility of the structured oral examination

Competency based medical education involves assessing physicians-in-training in multiple roles. Training programs are challenged by the need to introduce appropriate yet feasible assessment methods. We therefore examined the utility of a structured oral examination (SOE) in the assessment of the 7 CanMEDS roles (Medical Expert, Communicator, Collaborator, Manager, Health Advocate, Scholar and Professional) in a Neonatal-Perinatal Medicine subspecialty training program. Between 2004 and 2008, 68 trainees participated in an annual SOE. Each SOE consisted of 4 1st year and 4 2nd year clinical scenarios with standardized questions written by neonatologists that addressed all 7 CanMEDS roles. Examiners assigned a checklist score and global rating scores for knowledge and organization. A satisfaction survey was completed at the end of 3 examinations. Across the 5 SOEs, questions related to each competency were tabulated and an overall checklist score for each competency calculated. Inter-station reliability was determined for each CanMEDS role. Values for Cronbach’s alpha were 0.62 ± 0.2 (Medical Expert), 0.43 (Communicator), 0.40 ± 0.34 (Collaborator), 0.19 ± 0.17 (Manager), 0.46 ± 0.32 (Advocate), 0.67 ± 0.18 (Scholar) and 0.79 (Professional). Inter-rater reliability, determined for the first examination when there were 2 examiners per case, was moderate to substantial for 67% of cases. Participant satisfaction was high. Electronic communication facilitated exam development and costs were minimal. The SOE demonstrates psychometric properties suitable for a formative, in-training assessment as well as low cost, ease of administration and acceptability. It may be a useful way to assess physician competencies in training programs.     

A review of the results of the examinations for Fellowship of the Nigerian Postgraduate Medical College (1972–84)

Efforts at the local training of postgraduate doctors in Nigeria started formally in 1970, following the establishment of the Nigerian Fellowship Programme, now called the National Postgraduate Medical College of Nigeria. The programme consists of three sets of examinations termed Primary, Part I and Part II examinations. This report is an analysis of the performance at examinations conducted by the National Postgraduate Medical College of Nigeria. Since its inception in 1970, a total of 4388 attempts at the various examinations of the College had been made by November 1984. The mean pass rate was 38.8%. Pass rates in the Primary and Part I examinations were similar to each other and to pass rates in similar examinations in other countries. Pass rates were much higher in the Part II or final examinations. The College has produced a total of 145 Fellows during the period under review. Of these 36, 34, 27 and 13 respectively were in internal medicine, obstetrics and gynaecology, general surgery and paediatrics. All other faculties accounted for 35 of the 145 Fellows. Applications for these examinations have increased tremendously in recent years and this trend is likely to continue in the future. The pace at which candidates complete the programme appears slow. The possible causes of the apparent poor performance of the College and ways of improving it are discussed.     

Training in assessment and appraisal: who needs it?

The aim of this project was to develop a training programme to help consultants supervising postgraduate medical trainees to fulfil their role as Educational Supervisors. A training needs analysis was conducted which revealed that consultants did not appreciate the importance of some of their new duties. This was mainly because of the lack of clear perception of their new roles. Hence training needs to be provided.     

Assessing the mini-Clinical Evaluation Exercise in comparison to a national specialty examination

PURPOSE: To evaluate the reliability and validity of the Mini-Clinical Evaluation Exercise (mini-CEX) for postgraduate year 4 (PGY-4) internal medicine trainees compared to a high-stakes assessment of clinical competence, the Royal College of Physicians and Surgeons of Canada Comprehensive Examination in Internal Medicine (RCPSC IM examination). METHODS: Twenty-two PGY-4 residents at the University of British Columbia and the University of Calgary were evaluated, during the 6 months preceding their 2004 RCPSC IM examination, with a mean of 5.5 mini-CEX encounters (range 3-6). Experienced Royal College examiners from each site travelled to the alternate university to assess the encounters. RESULTS: The mini-CEX encounters assessed a broad range of internal medicine patient problems. The inter-encounter reliability for the residents' mean mini-CEX overall clinical competence score was 0.74. The attenuated correlation between residents' mini-CEX overall clinical competence score and their 2004 RCPSC IM oral examination score was 0.59 (P = 0.01). CONCLUSION: By examining multiple sources of validity evidence, this study suggests that the mini-CEX provides a reliable and valid assessment of clinical competence for PGY-4 trainees in internal medicine.     

Mentorship in postgraduate training programmes: views of Canadian programme directors

Objectives  Many academic training programmes have developed mentorship programmes for postgraduate doctors in training, but little is known about the factors that influence their establishment.
Methods  Canadian postgraduate training directors were surveyed to determine views on mentorship and factors associated with the establishment of these programmes.
Results  A total of 199 of 344 (58%) programme directors completed an online survey. Overall, 65% of respondents reported that their training programmes had a mentorship programme and 40% felt there was a need for more structured mentorship in training programmes. Univariate analysis showed that mentorship programmes were present significantly more often in larger programmes, internal medicine-based training programmes, and in programmes where the acting programme director had either been part of a mentorship programme during his or her own training or felt that mentorship had played an important role in his or her professional development. In adjusting for covariates using a logistic regression analysis, only those factors directly attributable to a programme director's personal mentoring experiences remained significantly associated with having a mentorship programme. Those who felt that mentorship had played a role in their own careers ( P  = 0.008, odds ratio [OR] = 3.3, 95% confidence interval [CI] 1.7–6.6) or who had been part of a mentorship programme during their own training ( P  = 0.01, OR = 6.6, 95% CI 1.4–30.1) were more likely to have an active mentorship programme at their institution.
Conclusions  A need for more structured mentorship was identified for many training programmes. Overall, programme directors' previous mentoring experiences were independently associated with having a mentorship programme.     

Developing an education and assessment framework for the Foundation Programme

AIM: To develop an education and assessment framework for the second year of the Foundation Programme (F2). METHODS: A total of 23 PRHOs were recruited to the F2 pilot in August 2003. The training posts included a variety of specialties at 2 hospital trusts plus primary care. Trainee expectations and satisfaction were evaluated using questionnaires administered before and at the end of the pilot. At the end of the pilot, 10 trainees participated in a focus group and 19 trainers participated in a semistructured telephone interview. RESULTS: The majority of trainees (78%) felt that their expectations of the F2 pilot were met and all felt that they had improved their generic skills. Attendance at the generic education programme was 95%. The majority of trainees found the assessment framework useful. The percentage of undecided trainees in terms of career aspirations dropped from 48% to 13%. Trainees valued the breadth of experience provide by the year and the support provided by the programme directors and each other. A need for better communication, administrative support and time for assessment was highlighted by the trainers. CONCLUSIONS: Early, focused education on generic skills will benefit both doctors and their patients. More varied career experience will help to ensure that doctors make appropriate and timely career decisions. Pilots are identifying good practice and areas that need improvement.     

Educational impact of in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice

Objectives To investigate the experiences and opinions of programme directors, clinical supervisors and trainees on an in‐training assessment (ITA) programme on a broad spectrum of competence for first year training in anaesthesiology. How does the programme work in practice and what are the benefits and barriers? What are the users' experiences and thoughts about its effect on training, teaching and learning? What are their attitudes towards this concept of assessment? Methods Semistructured interviews were conducted with programme directors, supervisors and trainees from 3 departments. Interviews were audiotaped and transcribed. The content of the interviews was analysed in a consensus process among the authors. Results The programme was of benefit in making goals and objectives clear, in structuring training, teaching and learning, and in monitoring progress and managing problem trainees. There was a generally positive attitude towards assessment. Trainees especially appreciated the coupling of theory with practice and, in general, the programme inspired an academic dialogue. Issues of uncertainty regarding standards of performance and conflict with service declined over time and experience with the programme, and departments tended to resolve practical problems through structured planning. Discussion Three interrelated factors appeared to influence the perceived value of assessment in postgraduate education: (1) the link between patient safety and individual practice when assessment is used as a licence to practise without supervision rather than as an end‐of‐training examination; (2) its benefits to educators and learners as an educational process rather than as merely a method of documenting competence, and (3) the attitude and rigour of assessment practice.     

Publishing history does not correlate with clinical performance among internal medicine residents

Medical Education 2010: 44 : 468–474 Objectives Selection criteria for applicants to the internal medicine programme at the University of Toronto have included the number and quality of scholarly items published. We sought to determine whether previous publishing record correlated with resident performance as measured by in‐training evaluation reports (ITERs) and global impressions of clinical competency by site programme directors and senior educators (global impression). Methods Data on the total number, quality and type of items published, as well as the timing of publishing with regard to pre‐MD training, were abstracted from the curricula vitae of individuals who applied for residency during 2001–2005. These were correlated with overall, Expert and Scholar role ITER scores, and with global impression, using Spearman rank correlation scores. Results We gathered publishing history data on 181 residents, for 162 of whom ITER data were available. Overall, 68.5% of residents had published, but only 14.9% had published during medical school. There was a weak correlation of borderline significance (ρ = 0.15, P = 0.055) between overall ITER score and number of items published. No such correlation was found with CanMEDS Medical Expert and Scholar role scores. Global impression classified 33.9% of residents as top‐rated. More top‐rated residents had published (76.7% versus 65.1%; P = 0.07), but the number of items published during medical school were similar between top‐rated and non‐top‐rated residents (16.1% versus 12.3%; P = 0.46). Conclusions Our results do not support publishing record as a predictor of residents’ clinical performance. Surprisingly, the correlation between publishing record and Scholar role scores was also weak, possibly indicating an inability of the ITER to capture this competency. Further research is needed to identify predictors and measures of performance in scholarly activities.     

Short-term outcomes and long-term impact of a programme in medical education for medical students

Objectives  There is growing appreciation of the value of early preparation of future medical educators. Staff development programmes, conferences and workshops pertaining to the training of educators may be crucial to the pursuit of a school's larger educational mission to educate students, doctors and scholars and to provide comprehensive knowledge, research, patient care and service. This study examined the efficacy of a 1-week educational intervention aimed at preparing medical students to become effective doctor educators by building skills early in their careers. The study asked whether participation in a 5-day teacher training programme led to increased knowledge of instructional methods, more favourable attitudes towards teaching, and the integration of structured instructional design methods in a student-developed teaching project.
Methods  A mixed methods research design was employed with quantitative data captured through pre- and post-test inventories, qualitative components captured through written comments, and a 2-year post-intervention survey. Quantitative analyses included pre-/post-intervention repeated measures with calculated effect sizes. Qualitative analysis was conducted using constant comparative methods.
Results  Subjects demonstrated improved content knowledge and more positive attitudes towards motivation, teaching confidence, teacher roles, varied pedagogy, and use of assessment, instructional planning, and evaluation. Subjects were able to incorporate the programme's teaching theory and methods into their teaching projects and assessment of peers' and others' teaching in their own institutions 2 years post-training.
Conclusions  This study demonstrates that a well-designed programme for teacher preparation can be pedagogically effective for training medical students to become better educators and that this learning can be incorporated into long-term practice.     

The effects of the introduction of a system of mandatory formative assessment for general practice trainees

A series of questionnaires was used to determine the amount of formative assessment going on within the West of Scotland Region of the UK and its perceived value by trainees (General Practice [GP] Registrars) both at the beginning and end of training. The first survey was carried out in 1989 and the second in 1994. Trainers and course organizers were surveyed in 1994 to determine their attitudes to the use of videotaped consultations for formative assessment. In 1991, the region had initiated a mandatory formative assessment programme which included regular use of videotaped consultations, confidence rating scales and Manchester ratings (RCGP Occasional Paper Number 40). The use of a range of assessment methods for formative assessment of GP Registrars increased considerably between 1989 and 1994. The percentage of trainees using videotaped consultation analysis increased from 76 to 94%, for Manchester ratings from 52 to 68% and for confidence rating scales from 63 to 74%. Video and confidence scales were rated highly by trainees who were assessed by them and by most trainees at the start of the year, but less highly by those who had not been assessed by them by the end of the trainee year. Manchester ratings were not thought to be as useful, and for these there was no difference between users and non-users. Despite the mandatory system, a significant number of trainees (76%) were still not receiving the minimum assessment stipulated. Trainers rated lack of time as the main limiting factor to the greater use of video. We conclude that trainees who are exposed to assessment methods, particularly video, find it useful, but that some trainees are still receiving less than is optimal. We propose increased trainer education and intensified monitoring of the assessment carried out in training Practices.     

Multiple mini-interviews: opinions of candidates and interviewers

OBJECTIVES: To assess candidates' and interviewers' perceptions of the use of a multiple mini-interview (MMI) for selection of senior house officers (SHOs) to a UK regional paediatric training programme. METHODS: Both candidates and interviewers completed anonymous questionnaires (comprising 16 and 25 questions, respectively). Demographic data were recorded for both groups. Data were analysed by frequencies; using Mann-Whitney and Kruskall-Wallis tests for comparisons; and Cronbach's alpha for internal consistency within the data. RESULTS: Both candidates and interviewers were positive about the fairness of the MMI (mean scores of 4.0 and 4.4, respectively). The majority of candidates (83%) had not been to this type of interview before. Gender, age and previous experience of MMIs did not account for differences in candidate responses (P > 0.05). A total of 86% of candidates were international medical graduates who preferred the format more than UK graduates did (P = 0.01). Interviewers were mainly experienced consultants who agreed that the multi-station format was better than the traditional interview (mean score 4.8) and represented a reliable process (mean score 4.4). Interviewers were concerned about the range of competencies covered and the subsequent performance of candidates in post (mean scores 3.6 and 3.2, respectively). CONCLUSIONS: Both candidates and interviewers agreed that the MMI format was reliable, fair and asked appropriate, easy-to-understand questions. In high-stakes interviews such as for specialty training in Modernising Medical Careers programmes, it is vital that all concerned have confidence in the selection process.     

Are some of the challenging aspects of the CanMEDS roles valid outside Canada?

CONTEXT: Many countries have adopted the CanMEDS roles. However, there is limited information on how these apply in an international context and in different specialties. OBJECTIVES: To survey trainee and specialist ratings of the importance of the CanMEDS roles and perceived ability to perform tasks within the roles. METHODS: We surveyed 8749 doctors within a defined region (eastern Denmark) via a single-issue, mailed questionnaire. Each of the 7 roles was represented by 3 questionnaire items to be rated for perceived importance and confidence in ability to perform the role. RESULTS: Responses were received from 3476 doctors (42.8%), including 190 interns, 201 doctors in the introductory year of specialist training, 529 residents and 2152 specialists. The overall mean rating of importance (on a scale of 1-5) of the aspects of competence described in the CanMEDS roles was 4.2 (standard deviation 0.6) and did not differ between trainee groups and specialists. Mean ratings of confidence were lower than ratings of importance and increased across the groups from interns to specialists. Differences between specialty groups were evident in both importance and confidence for many of the roles. For laboratory, technical and, to a lesser extent, cognitive specialties, the role of Health Advocate scored the lowest in importance. For general medicine specialties, the roles of Medical Expert, Collaborator, Manager and Scholar all scored lower for importance and confidence. CONCLUSIONS: This study provides a sketch of the content and construct validity of the CanMEDS roles in a non-Canadian setting. More research is needed in how these aspects of competence can be best taught and applied across specialties in different jurisdictions.