MOR: a simulation-based assessment centre for evaluating the personal and interpersonal qualities of medical school candidates

Context Medical school admissions traditionally rely heavily on cognitive variables, with non‐cognitive measures assessed through interviews only. In recognition of the unsatisfactory reliability and validity of traditional interviews, medical schools are increasingly exploring alternative approaches that can provide improved measures of candidates’ personal and interpersonal qualities. Methods An innovative assessment centre (MOR [Hebrew acronym for ‘selection for medicine’]) was designed to measure candidates’ personal and interpersonal attributes. Three assessment tools were developed: behavioural stations, including encounters with simulated patients and group tasks; an autobiographical questionnaire, and a judgement and decision‐making questionnaire. Candidates were evaluated by trained raters on four qualities: interpersonal communication; ability to handle stress; initiative and responsibility, and self‐awareness. Results In the years 2004–05, the 588 medical school candidates with the highest cognitive scores were tested; this resulted in a change of approximately 20% in the cohort of accepted students compared with previous admission criteria. Internal consistency ranged from 0.80 to 0.88; inter‐rater reliability ranged from 0.62 to 0.77 for the behavioural stations and from 0.72 to 0.95 for the questionnaires; test–retest score correlation was 0.7. The correlation between candidates’ MOR scores and cognitive scores approached zero, reflecting the value of MOR in the screening process. Feedback from participants indicated that MOR was perceived as fair and appropriate for medical school screening. Discussion MOR is a reliable tool for measuring non‐cognitive attributes in medical school candidates. It has high content and face validity. Furthermore, its implementation conveys the importance of maintaining humanist characteristics in the medical profession to students and faculty staff.     

A needs-based study and examination skills course improves students' performance

BACKGROUND: Adult learning theory suggests that learning is most effective when related to need, when driven by the learner and when it is flexible. We describe the effect of an educational intervention that was driven by student need, and largely designed by students. METHODS: We undertook a needs assessment of fifth year medical students' study needs. Based on this, we helped them design a course to meet these needs. This was predominantly related to study skills and a practice objective structured clinical examination (OSCE). We evaluated the course by asking for student opinion and by measuring the effect on student performance in a high stakes medical school examination (written examination and OSCE). FINDINGS: Despite the course being run voluntarily and in after-hours sessions, 80-90% of the medical student class attended each session. Student performance on the end of year examinations was significantly enhanced in the year of the intervention, compared with previous years and with students from other schools sitting identical examinations in the same year. INTERPRETATION: Learning activities that are directly based on student needs, that focus on study and examination techniques, and that are largely student-driven, result in effective and valuable outcomes.     

Medical students' personality characteristics and academic performance: a five-factor model perspective

OBJECTIVES: This study investigates: (1) which personality traits are typical of medical students as compared to other students, and (2) which personality traits predict medical student performance in pre-clinical years. DESIGN: This paper reports a cross-sectional inventory study of students in nine academic majors and a prospective longitudinal study of one cohort of medical students assessed by inventory during their first preclinical year and by university examination at the end of each pre-clinical year. SUBJECTS AND METHODS: In 1997, a combined total of 785 students entered medical studies courses in five Flemish universities. Of these, 631 (80.4%) completed the NEO-PI-R (i.e. a measure of the Five-Factor Model of Personality). This was also completed by 914 Year 1 students of seven other academic majors at Ghent University. Year end scores for medical students were obtained for 607 students in Year 1, for 413 in Year 2, and for 341 in Year 3. RESULTS: Medical studies falls into the group of majors where students score highest on extraversion and agreeableness. Conscientiousness (i.e. self-achievement and self-discipline) significantly predicts final scores in each pre-clinical year. Medical students who score low on conscientiousness and high on gregariousness and excitement-seeking are significantly less likely to sit examinations successfully. CONCLUSIONS: The higher scores for extraversion and agreeableness, two dimensions defining the interpersonal dynamic, may be beneficial for doctors' collaboration and communication skills in future professional practice. Because conscientiousness affects examination results and can be reliably assessed at the start of a medical study career, personality assessment may be a useful tool in student counselling and guidance.     

Self-marking in written examination: a way of feedback and learning

A new form of written examination, involving self- and teacher assessment, immediate feedback and discussion time with the course leader has been tried and evaluated at seven undergraduate courses in obstetrics and gynaecology at Umeå University, Sweden. Out of 112 students (59 women, 53 men), 101 (90%) rated this method as a better or far better learning situation than the traditional written examination. Twenty-seven (24%) students regarded the examination to be equal to, and 78 students (70%) thought it better or far better than, the conventional assessment as method to check the knowledge. According to the students, the method of self-marking increases the fairness of the examination and reduces the chances of cheating. No significant differences related to gender, age or test results were found. The examination time required of students was the same as with the traditional method, while the amount of time required of teachers was reduced.     

An oral practical examination in emergency clinical surgery

A body of students were asked about their views regarding an oral practical examination in clinical surgery that had been in practice for over 12 years. This is an examination with the planned objective to test problem-solving abilities and professional attitudes in emergency surgery. Verbal emergency simulation aided by appropriate pathological specimens, surgical instruments and X-rays are put forward by the examiners, and the candidate is asked to respond. One hundred and ninety-four students returned their questionnaire out of a total of 196 students (99% response rate). Candidates who were satisfied with their answers in the examination numbered 127 (65%), while 67 candidates were dissatisfied (35%). Reasons for dissatisfaction were cited as difficult and complex emergency simulation questions, expressed by 44 candidates (23%), and anxiety generated by confrontation of the examiners, expressed by 23 candidates (12%). One hundred and thirteen students (58%) thought a change in the format of the examination was not needed, while 47 students (24%) suggested a change mostly towards standardization of the examination. Emergency clinical surgery is an important area of the surgical curriculum that needs to be assessed properly. A more standardized version of this oral practical examination could perhaps provide a useful tool of assessment.     

The interpersonal, cognitive and efficiency domains of clinical teaching: construct validity of a multi-dimensional scale

BACKGROUND: We are unaware of any hypothesis-driven studies showing that teaching assessments are comprised solely of interpersonal and cognitive domains. Moreover, previous teaching assessments have been biased by heterogeneous samples of evaluators. Consequently, we investigated the construct validity of faculty assessments comprised of interpersonal and cognitive domains, utilising evaluations obtained from resident doctors on an internal medicine hospital service. METHODS: A total of 1000 inpatient evaluations were completed on 60 general internal medicine faculty members. Education theory supported a 2-dimensional, 14-item scale. Principal factor analysis was used to explore the scale's dimensionality. Internal reliability and interobserver agreement were determined. Relationships between domains and instructor characteristics were also examined. RESULTS: Principal factor analysis revealed interpersonal, clinical teaching and efficiency domains. Internal reliabilities of all domains are high (alpha > 0.90). Interobserver agreement is good (range 0.64-0.83). In the interpersonal domain there is a trend towards higher scores for lower ranking faculty. Significant findings are higher overall scores in the interpersonal domain (P < 0.001), higher scores for assistant professors in the interpersonal domain (P = 0.008) and higher scores for male than female faculty in the interpersonal (P = 0.041) and clinical teaching (P = 0.008) domains. CONCLUSIONS: Clinical teaching evaluations are reducible to interpersonal, clinical teaching and efficiency domains. Evidence for construct validity includes predicted domains and high internal and interobserver reliabilities. Utilising a homogenous sample of evaluators minimised variance. Interestingly, lower ranking faculty scored higher in the interpersonal domain, suggesting that lower ranking faculty may focus more attention on teaching activities than full professors do.     

The objective structured clinical examination in undergraduate psychiatry

Inadequate attention has been given to verifying the psychometric attributes of the objective structured clinical examination (OSCE), yet its popularity has been increasing in recent years. Our 6 years' experience in Nigeria showed that OSCE is practicable in undergraduate psychiatry assessment and there is evidence over consecutive years that it has satisfactory reliability and criterion-based validity. The importance of students' feedback in assessing the quality of examination is reinforced, and subtle, less tangible elements which determine students' performance, such as social interactional mystique and some personality traits, are worthy of evaluative research.     

An assessment of academic performance and personality

This study examines whether personality profiles, using personality factors, or clusters of personality factors, are associated with academic success. One hundred and forty medical students of the University of Wales College of Medicine were invited to complete a personality questionnaire (Cattell 16 PF) as they sat their final examinations in June 1988. A total of 129 usable forms were obtained. The students were divided into four groups dependent on their academic performance, which had been monitored throughout the course. The majority (62%) had no academic problems, but 16 (12%) students had serious difficulties, which entailed delaying qualification by at least 6 months. There was no relationship between the scores obtained for the students' first attempt at A-level and their subsequent medical school academic performance. However, students who obtained a degree either before or during their medical course were significantly less likely to have academic problems. Academic success was not associated with any of Cattell's personality factors. This was true of previously reported groups of factors associated with the poor student performance, and regardless of first or second order factors. We conclude that this personality profile is unlikely to be helpful in selecting future intakes of medical students, although a prospective study would be required for a definite answer to this question.     

Achieving acceptable reliability in oral examinations: an analysis of the Royal College of General Practitioners membership examination's oral component

BACKGROUND: The membership examination of the Royal College of General Practitioners (RCGP) uses structured oral examinations to assess candidates' decision making skills and professional values. AIM: To estimate three indices of reliability for these oral examinations. METHODS: In summer 1998, a revised system was introduced for the oral examinations. Candidates took two 20-minute (five topic) oral examinations with two examiner pairs. Areas for oral topics had been identified. Examiners set their own topics in three competency areas (communication, professional values and personal development) and four contexts (patient, teamwork, personal, society). They worked in two pairs (a quartet) to preplan questions on 10 topics. The results were analysed in detail. Generalisability theory was used to estimate three indices of reliability: (A) intercase (B) pass/fail decision and (C) standard error of measurement (SEM). For each index, a benchmark requirement was preset at (A) 0.8 (B) 0.9 and (C) 0.5. RESULTS: There were 896 candidates in total. Of these, 87 candidates (9.7%) failed. Total score variance was attributed to: 41% candidates, 32% oral content, 27% examiners and general error. Reliability coefficients were: (A) intercase 0.65; (B) pass/fail 0.85. The SEM was 0.52 (i.e. precise enough to distinguish within one unit on the rating scale). Extending testing time to four 20-minute oral examinations, each with two examiners, or five orals, each with one examiner, would improve intercase and pass/fail reliabilities to 0.78 and 0.94, respectively. CONCLUSION: Structured oral examinations can achieve reliabilities appropriate to high stakes examinations if sufficient resources are available.     

Are medical students socially exclusive? A comparison with economics students

Context Medical students have long been perceived as socially segregated from other students. However, the evidence for this is mainly anecdotal. This study investigates this issue by comparing medical students with economics students. Methods Questionnaires measuring objective and subjective social life patterns were completed by 149 medical and 149 economics students at a campus‐based university in the UK. Results Medical students drew significantly more of their close friends and housemates from among students on the same course than economics students. Significantly more medical than economics students had a partner on the same course, participated in departmental sports clubs and societies, and felt separated from the rest of university life. Commonly reported reasons for this separation among medical students were high workloads, that the medical school was located outside the main campus, and high numbers of contact hours, some of which took place at sites outside the university. Discussion Medical students are more socially exclusive than economics students, which may lead to ‘in‐group’ attitudes and behaviours. This is educationally important and may affect their future professionalism. Universities should be aware of the issues involved when planning curricula, timetables, welfare initiatives and campus design. Prospective medical students should be informed of the social consequences of studying medicine.     

Geriatric medicine: does teaching alter medical students' attitudes to elderly people?

Summary. The development of a questionnaire to assess the attitudes of medical students towards old people is described. Principal components analysis of the responses of 114 first-year medical students revealed two orthogonal factors, named negative attitudes and medical intervention. Scores on these factors were compared among three groups of medical students: first-year students, 64 clinical phase medical students prior to a geriatric medicine course, and 69 medical students who had completed a geriatric medicine course. Negative attitudes scores did not differ between first year and the clinical years, but were reduced after the geriatric medicine course. Scores on the medical intervention factor reduced significantly from first year to the clinical years and were not reduced further by the geriatric medicine course. Women tended to have lower scores on negative attitudes. Medical students appeared to change their attitudes concerning the degree to which medical intervention is appropriate as a result of preclinical or general medical experience. However, their reservations concerning the reward to be gained from working with elderly people were stable over the same periods, but were altered by a course in geriatric medicine.     

The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study

AIM: Because it deals with qualitative information, portfolio assessment inevitably involves some degree of subjectivity. The use of stricter assessment criteria or more structured and prescribed content would improve interrater reliability, but would obliterate the essence of portfolio assessment in terms of flexibility, personal orientation and authenticity. We resolved this dilemma by using qualitative research criteria as opposed to reliability in the evaluation of portfolio assessment. METHODOLOGY/RESEARCH DESIGN: Five qualitative research strategies were used to achieve credibility and dependability of assessment: triangulation, prolonged engagement, member checking, audit trail and dependability audit. Mentors read portfolios at least twice during the year, providing feedback and guidance (prolonged engagement). Their recommendation for the end-of-year grade was discussed with the student (member checking) and submitted to a member of the portfolio committee. Information from different sources was combined (triangulation). Portfolios causing persistent disagreement were submitted to the full portfolio assessment committee. Quality assurance procedures with external auditors were used (dependability audit) and the assessment process was thoroughly documented (audit trail). RESULTS: A total of 233 portfolios were assessed. Students and mentors disagreed on 7 (3%) portfolios and 9 portfolios were submitted to the full committee. The final decision on 29 (12%) portfolios differed from the mentor's recommendation. CONCLUSION: We think we have devised an assessment procedure that safeguards the characteristics of portfolio assessment, with credibility and dependability of assessment built into the judgement procedure. Further support for credibility and dependability might be sought by means of a study involving different assessment committees.     

The Self-Directed Learning Readiness Scale: a factor analysis study

BACKGROUND: The practice of medicine demands that its physician practitioners are self-directed, life-long learners. The Self-Directed Learning Readiness Scale (SDLRS) intends to measure adults' readiness to engage in self-directed learning. PURPOSE: The present study assesses the underlying factor structure of the SDLRS for a sample of entering medical students. METHODS: Over a period of 6 years, 972 first year medical students at the Virginia Commonwealth University School of Medicine completed the SDLRS. To summarise the inter-relationships among variables, a principal axis factor analysis with oblique rotation was used on the 58 SDLRS items. A series of confirmatory factor analyses using LISREL 8.54 was performed to further examine the measurement model underlying the SDLRS. RESULTS: A 4-factor confirmatory model representing 4 correlated substantive factors and a reverse coding method factor fits these data well. CONCLUSIONS: Medical educators should hold limited expectations of the SDLRS to measure medical students' readiness to engage in self-directed learning. The definitions and theoretical assumptions that inform readiness for self-directed learning should be reconsidered. Alternative approaches to studying self-directed learning should be explored.     

Relationship between candidate communication ability and oral certification examination scores

Context Structured case‐based oral examinations are widely used in medical certifying examinations in the USA. These orals assess the candidate’s decision‐making skills using real or realistic patient cases. Frequently mentioned but not empirically evaluated is the potential bias introduced by the candidate’s communication ability. Objective This study aimed to assess the relationship between candidate communication ability and medical certification oral examination scores. Methods Non‐doctor communication observers rated a random sample of 90 candidates on communication ability during a medical oral certification examination. The multi‐facet Rasch model was used to analyse the communication survey and the oral examination data. The multi‐facet model accounts for observer and examiner severity bias. anova was used to measure differences in communication ability between passing and failing candidates and candidates grouped by level of communication ability. Pearson’s correlations were used to compare candidate communication ability and oral certification examination performance. Results Candidate separation reliability values for the communication survey and the oral examination were 0.85 and 0.97, respectively, suggesting accurate candidate measurement. The correlation between communication scores and oral examination scores was 0.10. No significant difference was found between passing and failing candidates for measured communication ability. When candidates were grouped by high, moderate and low communication ability, there was no significant difference in their oral certification examination performance. Conclusions Candidates’ communication ability has little relationship to candidate performance on high‐stakes, case‐based oral examinations. Examiners for this certifying examination focused on assessing candidate decision‐making ability and were not influenced by candidate communication ability.     

Integration of the surgical specialties into an innovative undergraduate curriculum

Summary. A dilemma exists concerning the most effective way to incorporate surgical specialty teaching into an innovative undergraduate curriculum. Should the specialties form part of the core curriculum or should they be available as electives? The aim of this study is to describe an educationally sound and time-effective way to give all students an exposure to the surgical specialties while maintaining the philosophy of an innovative curriculum. The SCORPIO method was used, on a trial basis, in 1992 to teach cardiothoracic surgery, neurosurgery, plastic surgery and urology to fourth-year medical students at a university teaching hospital. The teaching was evaluated by comparing group pretest performance with a posttest assessment given one month after each of the teaching sessions. Student perceptions were assessed by a questionnaire and a time analysis was carried out comparing SCORPIO with the traditional ward tutorial system. Students completed the sequence pretest, teaching and posttest on 169 occasions. Group performance increased from a pretest mean 28% (SD16) to a posttest mean 44% (SD13) P<0.0001 . Student acceptance was favourable, with high ratings given to the structured, problem-based style of teaching. The time to run the programme was one-eighth that to teach by the ward tutorial system. This teaching model is a practical way to incorporate the surgical specialties into the core curriculum of a school moving towards a student centred, problem-based, integrated curriculum.     

Psychological size and distance: emphasising the interpersonal relationship as a pathway to optimal teaching and learning conditions

BACKGROUND: Positive interpersonal relationships between teachers and learners increase the quality of learning. The purpose of this study was to investigate psychological size (perceived status) and psychological distance (perceived emotional connectedness) in medical teaching interactions and their impact on the teaching and learning process. METHOD: A total of 45 paediatric preceptor/resident pairs engaged in longitudinal continuity training experiences at different sites were surveyed about teaching effectiveness, satisfaction with teaching, and the psychological size and distance in the relationship between each pair. RESULTS: Both residents and preceptors perceived the resident as having a smaller psychological size compared to the preceptor. Residents perceived greater psychological distance in the relationship than did preceptors, and this distance was significantly related to both residents' satisfaction with particular preceptors and their perception of the preceptors' effectiveness. CONCLUSIONS: Psychological size and distance contribute to effective and satisfactory teaching. Investigating additional aspects of the teaching-learning relationship should help identify optimal educational conditions.     

The direct clinical examination: an alternative method for the assessment of clinical psychiatry skills in undergraduate medical students

Summary: Because of dissatisfaction with the traditional long case procedure as a method of examining the clinical competence of medical students undertaking a psychiatry term, an alternative 'direct' method whereby two examiners observe the interaction between student and patient has been developed and is described. This method of examining allows the examiners to set and evaluate case-specific tasks. It is demonstrated that the two examiners achieve satisfactory inter-rater reliability both with respect to the mark awarded and the difficulty the patient presents and that, as one would wish, these two measures do not correlate. Students' opinions regarding the examination were assessed pre- and post-examination using visual analogue scales. The students found the examination stressful but rated the method as an appropriate form of clinical assessment both before and after their examination. The method is seen as having several advantages which must be set against the disadvantage of its being relatively expensive of examiners' time.     

Generalisability: a key to unlock professional assessment

CONTEXT: Reliability is defined as the extent to which a result reflects all possible measurements of the same construct. It is an essential measurement characteristic. Unfortunately, there are few objective tests for the most important aspects of the professional role because they are complex and intangible. In addition, professional performance varies markedly from setting to setting and case to case. Both these factors threaten reliability. AIM: This paper describes the classical approach to evaluating reliability and points out the limitations of this approach. It goes on to describe how generalisability theory solves many of these limitations. CONDITIONS: A G-study uses variance component analysis to measure the contributions that all relevant factors make to the result (observer, situation, case, assessee and their interactions). This information can be combined to reflect the reliability of a single observation as a reflection of all possible measurements - a true reflection of reliability. It can also be used to estimate the reliability of a combined sample of several different observations, or to predict how many observations are required with different test formats to achieve a given level of reliability. Worked examples are used to illustrate the concepts.     

The development, validity and reliability of a multimodality objective structured clinical examination in psychiatry

OBJECTIVES: To evaluate the development, validity and reliability of a multimodality objective structured clinical examination (OSCE) in undergraduate psychiatry, integrating interactive face-to-face and telephone history taking and communication skills stations, videotape mental state examinations and problem-oriented written stations. METHODS: The development of the OSCE on a restricted budget is described. This study evaluates the validity and reliability of 4 15-18-station OSCEs for 128 students over 1 year. Face and content validity were assessed by a panel of clinicians and from feedback from OSCE participants. Correlations with consultant clinical 'firm grades' were performed. Interrater reliability and internal consistency (interstation reliability) were assessed using generalisability theory. RESULTS: The OSCE was feasible to conduct and had a high level of high perceived face and content validity. Consultant firm grades correlated moderately with scores on interactive stations and poorly with written and video stations. Overall reliability was moderate to good, with G-coefficients in the range 0.55-0.68 for the 4 OSCEs. CONCLUSIONS: Integrating a range of modalities into an OSCE in psychiatry appears to represent a feasible, generally valid and reliable method of examination on a restricted budget. Different types of stations appear to have different advantages and disadvantages, supporting the integration of both interactive and written components into the OSCE format.