Effects of conventional and problem-based learning on clinical and general competencies and career development

Objective  To test hypotheses regarding the longitudinal effects of problem-based learning (PBL) and conventional learning relating to students' appreciation of the curriculum, self-assessment of general competencies, summative assessment of clinical competence and indicators of career development.
Methods  The study group included 2 complete cohorts of graduates who were admitted to the medical curriculum in 1992 (conventional curriculum, n  =   175) and 1993 (PBL curriculum, n  =   169) at the Faculty of Medicine, University of Groningen, the Netherlands. Data were obtained from student records, graduates' self-ratings and a literature search. Gender and secondary school grade point average (GPA) scores were included as moderator variables. Data were analysed by a stepwise multiple and logistic regression analysis.
Results  Graduates of the PBL curriculum scored higher on self-rated competencies. Contrary to expectations, graduates of the PBL curriculum did not show more appreciation of their curriculum than graduates of the conventional curriculum and no differences were found on clinical competence. Graduates of the conventional curriculum needed less time to find a postgraduate training place. No differences were found for scientific activities such as reading scientific articles and publishing in peer- reviewed journals. Women performed better on clinical competence than did men. Grade point average did not affect any of the variables.
Conclusions  The results suggest that PBL affects self-rated competencies. These outcomes confirm earlier findings. However, clinical competence measures did not support this finding.     

Defining characteristics of educational competencies

Context  Doctor competencies have become an increasing focus of medical education at all levels. However, confusion exists regarding what constitutes a competency versus a goal, objective or outcome.
Objectives  This article attempts to identify the characteristics that define a competency and proposes criteria that can be applied to distinguish between competencies, goals, objectives and outcomes.
Methods  We provide a brief overview of the history of competencies and compare competencies identified by international medical education organisations (CanMEDS 2005, Institute for International Medical Education, Dundee Outcome Model, Accreditation Council for Graduate Medical Education/American Board of Medical Specialties). Based upon this review and comparisons, as well as on definitions of competencies from the literature and theoretical and conceptual analyses of the underpinnings of competencies, the authors develop criteria that can serve to distinguish competencies from goals, objectives and outcomes.
Results  We propose 5 criteria which can be used to define a competency: it focuses on the performance of the end-product or goal-state of instruction; it reflects expectations that are external to the immediate instructional programme; it is expressible in terms of measurable behaviour; it uses a standard for judging competence that is not dependent upon the performance of other learners, and it informs learners, as well as other stakeholders, about what is expected of them.
Conclusions  Competency-based medical education is likely to be here for the foreseeable future. Whether or not these 5 criteria, or some variation of them, become the ultimate defining criteria for what constitutes a competency, they represent an essential step towards clearing the confusion that reigns.     

Reliability and efficiency of components of clinical competence assessed with five performance-based examinations using standardized patients

The present study was conducted to provide in-depth information on the reliabilities of measures of the separate components of clinical competence (e.g. data collection, test interpretation, diagnosis, etc.) assessed by a performance-based examination consisting of standardized-patient cases administered to five classes of senior medical students at Southern Illinois University School of Medicine. In general, the reliabilities of the competencies as they were actually measured on the examination (using the number of cases on which each competency was actually measured) were small, with 54% less than 0.30 and 75% less than 0.40. For generalizability coefficients pooled across the five classes and projected to a common number of k = 10 cases, two of the nine competencies had reliabilities in the 40s, a third was close to 0.40, and the remaining six were in the low 20s. The number of cases needed for the competencies to achieve the recommended reliability of 0.80 ranged from 45 to 170 cases, with six of the nine competencies requiring over 100 cases to reach the 0.80 level. The low reliabilities of these measures of the components of clinical competence raise serious questions about using the scores as indicators of student performance.     

The process of defining a profile of student competencies at the University of Vienna Medical School

CONTEXT: A new law enabling curriculum reform led a small circle of interested faculty members at the University of Vienna medical school to launch the Medizin Curriculum Wien project (MCW) to plan a new curriculum under the supervision of the Committee on Educational Affairs (Studienkommission). OBJECTIVE: The first stage of the project dealt with the definition of a profile of competencies (Qualifikationsprofil), which describes all the competencies we want our students to possess upon graduation. METHOD: This paper presents the approach to and process of defining this profile at one of Europe's largest medical schools, currently enrolling over 1500 new students per year. The procedure involved: preparing materials; communication with stakeholders; information dissemination; feedback gathering; information structuring, and formulating the profile of competencies. RESULTS: Application of the outlined steps in two rounds proved successful for the development of a profile of student competencies for Vienna. General acceptance and awareness of the new profile as well as commitment for a further reform process was engendered. The distinct characteristics of the profile are discussed. CONCLUSION AND UTILIZATION: A profile of student competencies must balance the demanding objectives of curriculum planners and the actual challenges presented by many local parameters; this can be done by employing the process outlined in this paper. The defined profile serves as an accepted reference point in the further development of the medical curriculum.     

Assessing validity in written tests of general practice - exploration by factor analysis of candidate response patterns to paper 1 of the MRCGP examination

OBJECTIVE: To investigate the content validity of Paper 1 of the MRCGP examination. METHOD: Exploratory factor analysis was carried out on candidate responses to Paper 1 of the May and October MRCGP examination in 1998. Contribution of each test question across factors was assessed using a pattern matrix of the oblique rotation. Common dimensions and variations between factor sets were identified. Key testing areas were then matched against the 'domains of competence' intended to be assessed by Paper 1 (as defined within the examination blueprint matrix). RESULTS: Whilst critical appraisal, disease prevention/evidence-based medicine and clinical management emerged as areas tested consistently, content variation was observed between factor sets extracted from both sittings. In addition, some overlap, in terms of domains tested, was seen among other assessment instruments used within the examination. CONCLUSION: Paper 1 conforms to the majority of its stated intentions. However, further development of techniques for investigating validity will be required in order to minimize content variation between both sittings of the examination as well as to help more closely define areas of competence to be tested by Paper 1 of the MRCGP examination.     

Direct, standardized assessment of clinical competence

Doctor ratings of clerkship performance are often discounted as not accurately reflecting clinical competence. Such ratings are influenced by the following uncontrolled variables: case difficulty; differing rater focus and standards; lack of agreement on what constitutes acceptable performance; and collective patient care responsibility masks individual contributions. Standardized direct measures of clinical competence were developed to control these factors and allow direct comparisons of student performance. Students saw 18 patients representing frequently occurring and important patient problems. Student actions and decisions were recorded and subsequent responses to questions revealed knowledge of pathophysiology, basis for actions, use and interpretation of laboratory investigations, and management. Actions and responses were graded using a pre-set key. The examination covered 73% of designated clinical competencies. Examinations scores corresponded with independent measures of clinical competence. Reliability studies indicated that new cases can be substituted in subsequent years with confidence that scores will maintain similar meaning. Costs are +6.95 per student per case, which is modest considering the quality and quantity of information acquired. Methods described are practical for evaluation of clerks and residents and for licensing and specialty certification examinations.     

Evaluating the social competence of young children: A review of the literature

Recent longitudinal studies of established federally funded educational programs designed for young children lacking educational experiences suggest that positive long-term effects were acquired through programs that focused on the development of the social competencies of the children. Although researchers differ on a specific definition and method of evaluation of social competence, the competencies generally accepted as social competence appear to be necessary for success in school and are advantageous to the child as he recognizes and becomes a part of the “culture of literacy”. Researchers agree on the complexity of the social competence construct. Research confirms cognition influences from family, community, and biological factors. Despite the widely accepted importance of social competence in programs for young children, measurement instruments vary. More research is being conducted in federal, state, and private programs on the effects of social competence on success in school. Organizations associated with early childhood programs as well as educational institutions are bringing into focus the need to establish present and future guidelines for developing curricula that enhance the development of social competence and appropriate assessment of social competence of young children.     

Is clinical competence perceived differently for student daily performance on the wards versus clerkship grading?

Clinical rotations play an important role in the medical curriculum and are considered crucial for student learning. However, competencies that should be learned can differ from those that are assessed. In order to explore which competencies are considered important for daily performance of student on the wards and to what extent clinical teachers consider the same competencies important for clerkship grading, a survey that consisted of 21 different student characteristics was administered to clinical teachers. Two independent factor analyses using structural equation modeling were conducted to abstract underlying latent relationships among the different student characteristics and to define a clinical competence profile for daily performance of students on the wards and clerkship grading. Differences between the degree of importance for student daily ward performance and clerkship grading are considered and discussed. The results of the survey indicate that the degree of importance of competencies are rated different for daily performance of students on the wards and clerkship grades. Competencies related to the diagnostic process are more important for clerkship grading, whereas interpersonal skills, professional qualities, and motivation are more important for daily ward performance. It is concluded that the components of clinical competence considered important for adequate performance are not necessarily in alignment with what is required for grading. Future research should focus on an explanation why clinical educators think differently about the importance of competencies for student examination in contrast to what is required for adequate daily performance on the wards.     

Clinical competence of interns

A clinical supervisors rating form addressing 13 competencies was used to assess the clinical competence of graduates one year after qualification in New South Wales (NSW), Australia. Data from 485 interns (97.2%) showed that graduates from the problem-based medical school were rated significantly better than their peers with respect to their interpersonal relationships, 'reliability' and 'self-directed learning'. Interns from one of the two traditional NSW medical schools had significantly higher ratings on 'teaching', 'diagnostic skills' and 'understanding of basic mechanisms'. Graduates from international medical schools performed worse than their peers on all competencies. These results were adjusted for age and gender. Additionally, women graduates and younger interns tended to have better ratings. Junior doctors have differing educational and other background experiences and their performance should be monitored.     

A written test as an alternative to performance testing

Performance tests are logistically complex and time consuming. To reach adequate reliability long tests are imperative. Additionally, they are very difficult to adapt to the individual learning paths of students, which is necessary in problem-based learning. This study investigates a written alternative to performance-based tests. A Knowledge Test of Skills (KTS) was developed and administered to 380 subjects of various educational levels, including both first-year students and recently graduated doctors. By comparing KTS scores with scores on performance tests strong convergent validity was demonstrated. The KTS failed discriminant validity when compared with a general medical knowledge test. Also the identification of sub-tests discriminating between behavioural and cognitive aspects was not successful. This was due to the interdependence of the constructs measured. The KTS was able to demonstrate differences in ability level and showed subtle changes in response patterns over items, indicating construct validity. It was concluded that the KTS is a valid instrument for predicting performance scores and could very well be applied as supplementary information to performance testing. The relative ease of construction and efficiency makes the KTS a suitable substitute instrument for research purposes. The study also showed that in higher ability levels the concepts which were meant to be measured were highly related, giving evidence to the general factor theory of competence. However, it appeared that this general factor was originally non-existent in first-year students and that these competencies integrate as the educational process develops.