Audit and summative assessment: two years' pilot experience

An audit project has been submitted by all trainees in the West of Scotland since 1992 as part of a pilot process for summative assessment. The impact of 2 consecutive years of audit was assessed on 117 trainees in May 1994. A response rate of 89% was achieved. For 82 trainees (79%) this was their first practical experience of audit and as a result of it 85% felt more confident in introducing change to their next practice. Protected time was still a problem for the majority (53%) and one-third wanted more help from their trainer. Thirty trainees (29%) had attended four or fewer formal practice meetings in their 10 months of training, with 10% never having attended one. The previous trainee's audit project was rarely or never discussed with 70 trainees (87%) and only 12 trainees were evaluating changes recommended. An audit project as part of summative assessment may be encouraging trainees to think about change after they leave their training practices. Few, however, are evaluating change and protected time and support are still required. The marking of the summative assessment audit project may need to address this.     

Assessment of the M.Sc. course in Occupational Medicine at the London School of Hygiene and Tropical Medicine, 1969–74

The M.Sc. course in Occupational Medicine started in September 1969 at the London School of Hygiene and Tropical Medicine. Between 1969–74, ninetyeight doctors attended this 9 month full-time course. Thirty-four were British and sixty-four from overseas.
A questionnaire was mailed to all of them. Eighty-four replied, i.e. a response rate of 86%. Thirty-one of the U.K. doctors and fifty-three of the overseas replied. The difference between the UK and overseas doctors working full-time in occupational medicine was very marked before and after the course. 100% of the U.K. doctors and 89% of the overseas thought their objectives of attending the course were fulfilled. 97% of the U.K. doctors said the course covered the problems of their country as compared with 77% of the overseas. 100% of the U.K. doctors found the knowledge gained of practical value to their work. Suggestions were also made on topics taught and on the practical instruction of the course.
It appears that the M.Sc. course is successful and that the objectives outlined when starting the M.Sc. were to a great extent fulfilled.
The new changes introduced this year 1976 in the organization and the content of the course are also discussed, and their advantages emphasized.     

Setting educational priorities for learning the concepts of population health

Following the World Health Organization's policy of 'Health for All by the Year 2000', doctors are increasingly being seen as health care providers to populations of patients, in addition to their more traditional role as doctors to individuals in a one-to-one encounter. In order for doctors to take on this expanded role, they must learn the knowledge and skills appropriate to population health. In this paper, we propose a method of educational priority-setting which allows educational planners to identify those diseases and adverse health conditions most appropriate for studying the concepts of population health. Using the Measurement Iterative Loop of Tugwell and colleagues as a framework, a table of Priority Illness Conditions was developed and compared with a previous priority list developed from a survey of clinical teachers at the McMaster University Medical School. Discussion of the implications for this approach in setting educational priorities at undergraduate, postgraduate and continuing medical education levels is presented, along with a review of possible shortcomings and caveats in using this approach.     

The urgent need to train orthopaedic surgeons in third world countries

The paper stresses the need to train orthopaedic surgeons for third world countries. The urgent need for such a programme is emphasized. A suitable training scheme is outlined which could be followed in any developing country to train its own orthopaedic surgeons of the future. The importance of training in the local environment is emphasized.     

Training doctors for the year 2000

The World Health Organization's strategy, Health for All by the Year 2000, presents a challenge to those responsible for training doctors. Doctors are needed who are concerned to promote health not just treat disease. A review of the medical undergraduate curriculum is required to achieve this. We describe a small step towards this by the restructuring of a community medicine teaching programme so that students are introduced to health promotion and the principles of Health for All.     

An assessment of vocational training

A retrospective analysis of the opinions of vocationally trained doctors was obtained from a postal questionnaire. Ninety-eight doctors who had trained in the West of Scotland before the introduction of new criteria in 1985 for the appointment and reappointment of training practices were compared with 107 doctors who had trained following the introduction of the criteria, looking at their rating of training, the frequency of tutorials, value of teaching and research encouragement. There was a significant improvement in the rating of training, frequency of tutorials, research encouragement and enjoyment of training in the group who trained after the new criteria were adopted. The study demonstrates the benefits of vocational training for general practice when criteria are set and followed. This training model may be applicable in other branches of medicine.     

Health promotion in the medical curriculum

It has been argued that health promotion should be included in the medical curriculum, if medical education is to match the needs and health goals of communities. In spite of the medical profession's awareness of the importance of health promotion, few universities in the UK have introduced a substantive course on the subject. This paper describes the teaching of health promotion in Nottingham Medical School. It identifies the aims and objectives of the course which are amongst others to develop an understanding of: the factors which influence the perception and experience of health and illness; the concepts of health promotion and health education; the scientific basis of health promotion; the relative effectiveness of approaches used; the measurement of health status and health behaviour; and health policy and planning. An innovative element of the teaching involves a community project which gives students experience of health promotion in practice. Evaluation of the impact of the course suggests that students perceive it to be interesting, stimulating and enjoyable, whilst examination and continuous assessment indicate that it has also influenced students' knowledge and understanding of the subject.     

Factors underlying performance on written tests of knowledge

The study compares two popular forms of written tests; the multiple choice test (MCQ) and the Modified Essay Question (MEQ). Two factors were varied in the experiment: the format of the questions (multiple choice, directed free response, or open-ended free response) and the context of the questions (in a patient problem or in random sequence). Six problems were developed in each version, and administered to a total of 36 medical students at three educational levels using a Latin-square design. The results showed a significant effect of each factor in the design, amounting to a difference of 8.7% between MCQ and directed free response, 4.2% between directed and open-ended free response and 4.3% between problem and random context. However, the correlation of scores based on content across the formats approached unity after correction for attenuation. A process score, based on the style and presentation in the undirected format, correlated more strongly with the free-response questions. The results suggest that, although the MCQ and MEQ may assess different skills, there is a very strong relationship between content scores derived from the two formats. The free response formats may present the opportunity for assessment of other factors related to presentation if scoring procedures are modified. Finally, the effect of randomizing questions is a deterioration of performance when compared to placing questions in the problem context.     

Paediatric option in MRCP examination: a survey into the results of candidates electing to take the paediatrics option in MRCP (UK) Part II

Five surveys on the performance of candidates in the paediatric option at MRCP(UK) Part II examination show the paediatric candidates to be at a disadvantage in the written part. The paediatric candidate is later in his career to obtain his MRCP (UK), as the mean interval from qualification to success at Part II is 41.7 months compared with 34 months for non-paediatric candidates (   P = 0.0026  ).
Lower marks were scored by paediatric candidates in the written section on the whole (   P = 0.055  ), slide identification (   P = 0.015  ), data interpretation (   P = 0.065  ) and non-paediatric case histories (   P = 0.023  ) as seen in those candidates who passed each part at first attempt, the discrepancy being wider in those having more attempts to pass. In the oral examination the paediatric candidate scored higher (   P = 0.07  ) but no significant difference in the clinical marks.
Although it appears that few paediatricians have been severely penalized by the present system, it does require the paediatric candidate to orientate his preparation for Part II towards his books rather than towards his clinical practice, which is contrary to the aim of the Part II examination of the MRCP(UK). Plans have been announced by the Presidents of the three Royal Colleges of Physicians of the United Kingdom for an entirely paediatric Part II.     

Factors affecting future choice of specialty among first-year medical students of the University of the West Indies, Trinidad

BACKGROUND: This study was conducted to determine factors that influence career choice among 1st-year medical students. DESIGN AND METHODS: A cross-sectional survey of 170 1st-year medical students from the University of the West Indies, St Augustine Campus was undertaken with a questionnaire designed to assess their perceptions of careers in various specialties. Likert scales were used to quantify the reasons for their preferences. RESULTS: The response rate was 136/170 (80%). The age of respondents ranged from 16 to 36 years, mean 20.45, SD 2.88. Of the generic factors students considered important in their choice of a specialty, students ranked the ability to help patients the highest (rating of 1.44), along with the diagnosis and treatment of disease second (rating of 1.49); 38 (27.9%) cited medicine, 26 (19.1%) surgery, 13 (9.6%) paediatrics, 10 (7.4%) family practice and 4 (2.9%) psychiatry as their chosen career. Students begin their medical training with the view that a career in psychiatry is less attractive than other specialties surveyed. The average attractiveness was estimated as surgery 1.64, medicine, 1.81, paediatrics 1.95 and psychiatry 2.57. The differences between the averages were highly significant (F = 57.6, P < 0.001). CONCLUSION: The findings suggest that although 1st-year medical students rank the diagnosis and treatment of disease and the ability to help patients as the greatest influence in choosing a specialty, internal medicine was the most popular chosen career, while the surgical specialties were identified as the most attractive. Medical students have serious reservations about psychiatry as a career choice.     

Teaching nutrition to medical students: a community-based problem-solving approach

This paper presents a community-based problem-solving educational programme which aims at teaching medical and other health science students the importance of nutrition and its application. Through community surveys students assess the nutritional status of children under five using different anthropometric methods. They understand the cultural beliefs and customs related to food fads and the reasons for them. They also acquire the skill to educate the community using the information gathered. They use epidemiological methods such as case control study to find associations between malnutrition and other causative factors. Feedback from students has been positive and evaluation of students' knowledge before and after the programme has shown significant improvement.     

Self-planned experimentation (SPEX) practical classes in physiology

A new form of undergraduate physiology practical experience is described. Students make their acquaintance with the physiological apparatus in the usual manner. At the end of the afternoon each group of four students is allocated an item of equipment with which to perform a true investigative experiment of their own choice, in their own time, the results of which are to be presented 2 weeks later by a randomly chosen member of the group. Students may investigate any question that can be answered using the resources available to them without endangering themselves or their subjects. Marks are awarded for the planning of the experiment (i.e. choice of controls, etc.), the correctness of graphical presentation of the results, the appropriateness of the statistical analysis, and for the discussion. There are no penalties for negative or unexpected results.     

Educational assessment of general practice experience for psychiatric trainees

Eighteen psychiatric trainees spent 6 months each as general practice trainees. The educational impact of the experience was assessed by a self-assessment questionnaire, a semi-structured interview and a videotaped interview with a psychiatric patient. Each assessment was conducted at baseline and after 12 months. A control group of 14 trainees was recruited from the same rotation. On the self-assessment questionnaire, the study registrars rated their abilities to solve general medical problems significantly improved compared to controls. They had also acquired greater understanding of the limitations of their knowledge and their legal responsibilities towards their patients. The semi-structured interview failed to distinguish between the two groups. Videotapes for rating at baseline and follow-up were available for only 17 of the trainees. Assessment of the tapes used the Maguire Scale and the Interview Behaviour Scale. Neither scale demonstrated any intervention effect. The interviews were all characterized by a preponderance of 'closed psychological' and 'checking-out' questions. It appears that psychiatric trainees' interviewing styles had not been influenced by the experience. This study suggests that psychiatric trainees gain greater confidence in their role as a doctor and greater understanding of the scope and nature of general practice by such an attachment. It is unclear whether or not supplementary interviewing skills had been acquired which were not utilized in the taped interview, which conforms very much to traditional psychiatric examination behaviour. Trainees were reassured that they had increased their knowledge without losing any of their specific professional skills.     

A combined course of primary health care practice and family medicine at the University of Gezira

The community-based course presented is a longitudinal course running through four semesters in the Faculty of Medicine, University of Gezira, Sudan. Students combine their regular work in primary health care centres with attachments to a number of families in Wad Medani town. They continue to visit these families regularly throughout their entire medical course with the aim of studying them and helping them with some of their medical and psychosocial problems.     

The modified essay question: an evaluation of its use in a family medicine clerkship

This paper describes the extensive experience of the Department of Family Medicine at Jefferson Medical College in utilizing the modified essay question (MEQ) as a final examination for its required third-year clerkship in family medicine. The results of a preliminary study are presented, comparing the MEQ as a method of evaluation with the multiple choice question (MCQ) format on similar content material. MCQ performance was found to be a better predictor of National Board Part I and II scores, but neither format was found to be a good predictor of postgraduate performance. The data were statistically significant (P less than 0.05) that MEQ performance was related to postgraduate performance in the area of professional attitude while the MCQ was not; however, the number of students in the analysis was small. Additional studies are planned to test this hypothesis further.     

Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence

Despite a lack of face validity, there continues to be heavy reliance on objective paper-and-pencil measures of clinical competence. Among these measures, the most common item formats are patient management problems (PMPs) and three types of multiple choice questions (MCQs): one-best-answer (A-types); matching questions (M-types); and multiple true/false questions (X-types). The purpose of this study is to compare the reliability, validity and efficiency of these item formats with particular focus on whether MCQs and PMPs measure different aspects of clinical competence. Analyses revealed reliabilities of 0.72 or better for all item formats; the MCQ formats were most reliable. Similarly, efficiency analyses (reliability per unit of testing time) demonstrated the superiority of MCQs. Evidence for validity obtained through correlations of both programme directors' ratings and criterion group membership with item format scores also favoured MCQs. More important, however, is whether MCQs and PMPs measure the same or different aspects of clinical competence. Regression analyses of the scores on the validity measures (programme directors' ratings and criterion group membership) indicated that MCQs and PMPs seem to be measuring predominantly the same thing. MCQs contribute a small unique variance component over and above PMPs, while PMPs make the smallest unique contribution. As a whole, these results indicate that MCQs are more efficient, reliable and valid than PMPs.     

A computer-based test item-bank for cognitive assessment of medical students during a clinical medicine clerkship

The development and utilization of multiple-choice test items and the utilization of computer technology have significantly altered evaluation practices in medical schools. This paper describes the development and implementation of a computer-based multiple-choice test item-bank for use in a junior medicine clerkship. The computer program, ITEMBANK, is an interactive system that stores and edits test items and constructs and prints an original copy of each test. Tests can be produced almost immediately by the clerical staff and the total developmental costs have been minimal. The program has reduced the amount of teaching time required to develop tests, and has enhanced the learning of essential information by medical students and residents. Teaching staff and student response has been enthusiastic and positive. Statistically significant Pearson correlation coefficients have been obtained between student performance on these tests and clinical evaluation and NBME Part II scores. The program has made a valuable contribution to the improvement of evaluation procedures in the Internal Medicine clerkship.     

A human sexuality course for Oxford University medical students

A human sexuality course for clinical students in Oxford was held based on the format of similar courses held in the U.S.A. with one-and-a-half days of films and lectures followed by discussion in small groups. The course was largely successful and evaluation of attitudes and knowledge revealed that significant preliminary changes resulted. Students who failed to complete the course had more inhibited attitudes towards sexuality and less sexual information compared with those who attended the whole course. Advice is given to those intending to hold such a course elsewhere, including how more inhibited students, who probably are most in need of such teaching, might be encouraged to participate.     

Competence-based summative assessment of a student-directed course: involvement of key stakeholders

The 5-week module in general practice for final-year students at the University of Sheffield is based on practice attachments and student-directed learning in small groups. This paper describes how the summative assessment process of the module was revised to incorporate the notion of competence-based assessment, and how general practitioner tutors, departmental tutors and students were involved in this revision. The question ‘What are students expected to know and be able to do by the end of the module?’ was answered in terms of a statement of the key purpose of the module and a list of intended learning outcomes. The question ‘How can we find out if students have achieved these outcomes?’ was addressed by developing check-lists of criteria for observed behaviours and for the written products of students' actions.