Assessment of patient management skills and clinical skills of practising doctors using computer-based case simulations and standardised patients

CONTEXT: Standardised assessments of practising doctors are receiving growing support, but theoretical and logistical issues pose serious obstacles. OBJECTIVES: To obtain reference performance levels from experienced doctors on computer-based case simulation (CCS) and standardised patient-based (SP) methods, and to evaluate the utility of these methods in diagnostic assessment. SETTING AND PARTICIPANTS: The study was carried out at a military tertiary care facility and involved 54 residents and credentialed staff from the emergency medicine, general surgery and internal medicine departments. MAIN OUTCOME MEASURES: Doctors completed 8 CCS and 8 SP cases targeted at doctors entering the profession. Standardised patient performances were compared to archived Year 4 medical student data. RESULTS: While staff doctors and residents performed well on both CCS and SP cases, a wide range of scores was exhibited on all cases. There were no significant differences between the scores of participants from differing specialties or of varying experience. Among participants who completed both CCS and SP testing (n = 44), a moderate positive correlation between CCS and SP checklist scores was observed. There was a negative correlation between doctor experience and SP checklist scores. Whereas the time students spent with SPs varied little with clinical task, doctors appeared to spend more time on communication/counselling cases than on cases involving acute/chronic medical problems. CONCLUSION: Computer-based case simulations and standardised patient-based assessments may be useful as part of a multimodal programme to evaluate practising doctors. Additional study is needed on SP standard setting and scoring methods. Establishing empirical likelihoods for a range of performances on assessments of this character should receive priority.     

Evaluating family counselling skills training for family practice

Educational programmes for family practice should develop family counselling skills of students to moderate levels of competence. Few specific training programmes are part of the regular curriculum and of these few are evaluated. Twenty-three residents enrolled in a 2-year family practice programme in Toronto, Canada participated in a non-randomized control intervention study to assess a specific training programme. Pre-training counselling skills, and ability were assessed using two types of generally recognized measurement instruments, the Carkhuff Stems of Communication Skills and the Carkhuff Discrimination Index. The treatment group entered the training programme which involved supervised family counselling 8 half-days each week for one month as part of their usual rotations. They completed a second set of instruments following this course to assess immediate within-group change and then both they and the control group completed a set about one year later to measure differences. Initial scores for all residents showed a less than 'minimally facilitative' level of counselling response on both empathy and warmth dimensions. Following the course, the treatment group scored above this level and significantly better than the control group. Furthermore, the former showed a 20% improvement in ability to discriminate between effective and ineffective helping responses which was sustained over one year, while the control group's scores became worse. Teaching of counselling skills can be effective.     

Effects of the vocational training of general practice consultation skills and medical performance

The effects of the vocational training of general practitioners in the Netherlands on the consultation skills and medical performance of junior doctors were studied. Results obtained at a training institute providing systematic training in these skills (Nijmegen) were compared with those at an institute taking a problem-based learning approach (Groningen). Trainees (n = 63) audiotaped consultations and recorded their medical performance at the start and at completion of training. The skills were evaluated with the aid of validated criteria and medical 'protocols'. Data on 631 pre-training and 624 post-training consultations were compared. Changes in consultation skills and medical performance occurred at both institutes and proved more marked at the institute providing systematic training. Improved medical performance was found to be associated with improved consultation skills. Enhanced clinical knowledge was found to be related to improved medical performance and consultation skills. The most profound changes were found in junior doctors who had started at a lower level of consultation skills and medical performance.     

Training general practitioners to teach psychiatric interviewing skills: an evaluation of group training

Group teaching in problem-based interviewing based on video and audiotape feedback of the doctor's own consultations significantly improved the ability of experienced general practitioners to teach psychiatric skills to their trainees. When the GPs were randomly allocated to one of three further training experiences--video feedback of their tutorial sessions, discussion about how to teach and no further teaching, there were very few differences between the groups. The greatest impact on improving teaching skills was brought about by watching their own consultations in a group feedback setting.     

Training of general practitioners in interpreting chest radiographs

Easily accessible chest and skeleton roentgen examinations are needed in primary health care. In Finland the health centres perform about a third of all roentgen examinations. Many of the films are interpreted by health centre doctors at least in the primary phase. About 40% of them are chest radiographs. In 2640 chest radiographs interpreted by general practitioners the final roentgen diagnosis was pathological in 54%. Pathological conditions were totally overlooked in 13%. Of normal cases 19% were interpreted falsely positive. The greatest need for general practitioners' training is in the common findings in the chest, for example cardiac insufficiency and inflammatory as well as atelectatic changes in the lungs. Pulmonary findings in children proved difficult to interpret and were proposed as a subject for training. The role of poor image quality in interpretation errors must be emphasized when less experienced doctors interpret chest films. If regular radiological consultation is not available, postgraduate training in diagnostics of chest radiographs should be considered in the training of specialists in general practice as well as in their supplementary training.     

Reliability of consumer assessment of communication skills in a postgraduate family practice examination

Summary: Examinations of competence which may affect career prospects require measures which are of high reliability, and which can be demonstrated to be valid. In a New Zealand summative postgraduate examination of competence in family practice the doctor-patient communication skills of candidates were assessed by non-medically trained nominees of community organizations. The assessments were based on direct observation of the candidates' encounters with simulated patients. To estimate the reliability of the consumer examiner, after the examination the examiners re-scored a random selection of videotaped candidate encounters. The test-retest correlations of consumer scoring were demonstrated to be at a level consistent with adequate examination reliability (confidence interval 0.59-0-98).
Summary: Consumers may be valuable as a resource for the training and assessment of the communication skills of medical practitioners.     

Impact of a new curriculum on medical students' interpersonal and interviewing skills

OBJECTIVES: In 1998 we reported on the rise and fall of medical student communication skills during the 4 years of medical school. Since then, the University of Connecticut School of Medicine has completed a major curriculum renewal project with an emphasis on early clinical work, lifelong learning and more ambulatory training. The goals of this study were to compare students' interviewing and interpersonal skills in standardised patient (SP) assessments in the old and new curricula and to assess the success of the new curriculum in preventing a decline in student skills in this domain. METHODS: The clinical skills of 202 students were measured longitudinally during encounters with SPs in each of their 4 years of medical school. Students in this study and the earlier study were evaluated using the Arizona Clinical Interviewing Rating (ACIR) Scale. RESULTS: Compared with students from the previous curriculum, students on the new curriculum in this study showed an improvement in ACIR scores. Year 1 mean ACIR scores (1 = poor to 5 = excellent) were, respectively, 3.6 for the old curriculum cohort and 4.0 for the new curriculum group. In Year 4 the mean score for the old curriculum cohort was 3.7 and that for the new curriculum group was 3.8. Students on the new curriculum still showed a decline in ACIR scores from Years 1 to 4, but it was not as severe a decline as it had been previously. CONCLUSIONS: Pre-clinical medical students perform better on measures of interpersonal communication than their clinical counterparts. The students who participated in the new curriculum demonstrated an earlier acquisition of and a less steep decline in interviewing and interpersonal skills during the course of medical school.     

The effects of continuing medical education on family doctor performance in office practice: a randomized control study

A randomized controlled study was conducted to determine if specifically designed continuing medical education in the fields of cardiovascular and cancer medicine could change doctor office behaviour significantly. Thirty-one volunteer family doctors from 25 offices participated. Six (three cardiovascular and three cancer) learning objectives were defined. Two educational formats were selected as the independent variables: (1) group interaction opportunities (face-to-face and teleconference); and (2) concisely written newsletters. Chart measures of doctor performance prior to and 6 and 12 months following education served as the dependent variables. The family doctors receiving education were found to perform the recommended behaviours significantly more than those who did not receive the education (P less than 0.05) at 6 months post-education. This difference was maintained at the 12-month post-educational period for one of the educational programmes offered. A carefully planned programme of continuing medical education will result in favourable changes in the office practice of volunteer doctors. These changes can persist for as long as 12 months. Adherence to several essential learning principles is required.     

Teaching psychiatric interview skills to medical students

A teaching package is described for teaching interview skills to large blocks of medical students whilst on their psychiatric attachment. The aims of this package are to reduce students' concerns about interviewing psychiatric patients, to reinforce students' knowledge of basic interviewing skills and to introduce students to the particular skills required in taking a psychiatric history and mental state examination. The package emphasizes the following teaching methods: 'hands-on' experience of interviewing a patient in front of small groups of peers; peer feedback using check-lists which focus on three major aspects of interviewing; elicitation of facts, elicitation of feelings and control of the interview; facilitation of small-group discussions in the presence of senior psychiatrist. The active involvement of all students in interviewing psychiatric patients engages them in the learning process. Peer involvement increases motivation and was deemed by students as a supportive and constructive exercise. The presence of a senior psychiatrist ensures that discussion is focused on the process of interviewing rather than on patient pathology. Ideally this package would precede focused training throughout the subsequent psychiatric placement.     

Lectures and skills workshops as teaching formats in a history-taking skills course for medical students

The consulting skills acquired by medical students during their training are an important determinant of their ability to conduct adequate and efficient clinical interviews. These skills comprise: the acquisition of medical knowledge and the ability to apply this; and communication skills required to obtain full, accurate clinical histories from patients and to be able to give to patients the information they need to comply with prescribed regimens. Until recently, consulting skills training has certainly not had a high profile in medical curricula, despite evidence that students do not gain sufficient expertise during their medical training. A history-taking skills course within the Austin Hospital Clinical School, utilizing mass lecture and small-group skills workshops is described. Independent evaluation of students' videotaped interviews with patients, completed before training, after mass lectures and following small-group workshops, showed that students trained in consulting skills demonstrated significant improvements in interview skills and techniques, compared with a similar group of students for whom training followed the more traditional model. Whilst there were some improvements after mass lectures, most significant gains in history-taking skills were obtained following skills workshops. Ongoing evaluation of these students will determine if these short-term improvements in consultation skills persist over their clinical training and internship.     

Context-bound communication skills training: development of a new method

OBJECTIVES: To examine how communication skills training might be integrated into everyday clinical practice in a manner that is acceptable to clinicians. DESIGN: General practitioners from 3 group practices agreed to take part, in turn, in a study of how to manage difficult consultations about antibiotic prescribing for acute respiratory infections. This provided the opportunity to conduct communication skills training in which lessons learned from one practice were taken into the next. SETTING: United Kingdom general practices. SUBJECTS: Three groups of general practitioners. FINDINGS: Difficulties with the acceptability of a traditional off-site workshop approach, using role play as the main teaching method, led to the development of a new training method (context-bound training), which proved to be practical and acceptable to experienced clinicians. The main features of the method were the delivery of training in the clinicians' place of work, and the transformation of their reported difficult cases into scenarios which they then encountered with a standardized simulated patient before and after brief seminars. Everyday clinical experience was kept in the foreground and 'communication skills' in the background. CONCLUSIONS: The method is acceptable to clinicians and adaptable to a range of clinical situations. It offers potential for improving the communication skills of clinicians both in hospital and primary care settings.     

Biomedical knowledge in explanations of clinical problems by medical students

This paper was motivated by a controversy concerning the role of basic sciences in medical education. A problem underlying this issue is that it is unknown how basic science is used in clinical reasoning. The experiment was designed to address this issue. Three texts were constructed dealing with basic science knowledge relevant to a clinical problem. Subjects were asked to read and recall the texts. Next, the subjects were required to read and recall the clinical text describing a patient problem. Finally, they were asked to provide a diagnosis and an explanation of the underlying pathophysiology. Subjects were first-, second- and fourth-year medical students. Detailed analysis of subjects' protocols are presented. In general, the results show that when basic science information is given before the clinical problem, the basic science knowledge is used either incorrectly or inconsistently in explaining the clinical problem by all subjects. The authors interpret these results to indicate that the basic sciences and the more practical clinical knowledge form two separate domains with their own individual structures and the clinical information cannot be embedded into the basic science knowledge structure.     

Impact of patient feedback on the interpersonal skills of general practice registrars: results of a longitudinal study

CONTEXT: A general practice vocational training program. OBJECTIVES: To examine the impacts and implications of different models of systematic patient feedback on the development of general practice (GP) registrars' interpersonal skills as they progressed through a GP vocational training program. DESIGN: A longitudinal study in which GP registrars were randomly assigned to three models of patient feedback: a control group and two intervention groups. The major source of data gathering was through the Doctors' Interpersonal Skills Questionnaire (DISQ) which was administered to patients immediately after their consultation. SUBJECTS: 210 GP registrars, 104 GP supervisors and 28 156 patients. RESULTS: Multivariate analysis techniques (including repeated-measures analysis) tested the effectiveness of the interventions. Findings showed that systematic patient feedback at regular intervals throughout GP training resulted in sustained levels of interpersonal skills. The most significant gains in interpersonal skills for both intervention groups occurred in the earlier stages of general practice training. Most registrars found the experience of patient feedback useful for gaining a better understanding of their interpersonal skills and for identifying areas in which they needed to improve. GP supervisors valued the opportunity to receive patient feedback themselves and found the activity a useful adjunct to their preceptor role. CONCLUSIONS: Patients, by providing feedback on doctors' interpersonal skills, have been able to contribute to improving the quality of the patient-doctor interaction. GP registrars and their supervisors value highly the role of patient feedback in interpersonal skill development.     

Psychological tests to measure the effects of medical education on students'' interpersonal skills

The consulting skills required of medical students and practitioners have been categorized into a number of specific skills, two of which are: students' ability to empathize with the patient; and ability to decode non-verbal cues given by the patient in the interview. Training programmes to improve students' consulting skills are usually evaluated using analysis of students' actual interview behaviours with patients. Broad psychological and personality tests have also been used to measure changes in students' interviewing skills, but have generally not been successful. The hypothesis is advanced that more specific tests of the skills of interviewing, such as non-verbal sensitivity and empathy, would detect changes in students' ability to display these skills. As part evaluation of a consulting skills training programme, clinical students completed psychological tests of empathy and non-verbal sensitivity. Subsequent comparisons between trained and control student groups revealed no clear pattern in test results. These data suggest that specific psychological tests of empathy and non-verbal sensitivity may be no more effective in detecting changes in students' interpersonal skills than global personality measures.     

The direct clinical examination: an alternative method for the assessment of clinical psychiatry skills in undergraduate medical students

Summary: Because of dissatisfaction with the traditional long case procedure as a method of examining the clinical competence of medical students undertaking a psychiatry term, an alternative 'direct' method whereby two examiners observe the interaction between student and patient has been developed and is described. This method of examining allows the examiners to set and evaluate case-specific tasks. It is demonstrated that the two examiners achieve satisfactory inter-rater reliability both with respect to the mark awarded and the difficulty the patient presents and that, as one would wish, these two measures do not correlate. Students' opinions regarding the examination were assessed pre- and post-examination using visual analogue scales. The students found the examination stressful but rated the method as an appropriate form of clinical assessment both before and after their examination. The method is seen as having several advantages which must be set against the disadvantage of its being relatively expensive of examiners' time.     

Acquisition of communication skills in postgraduate training for general practice

PURPOSE: The evidence suggests that a longitudinal training of communication skills embedded in a rich clinical context is most effective. In this study we evaluated the acquisition of communication skills under such conditions. METHODS: In a longitudinal design the communication skills of a randomly selected sample of 25 trainees of a three-year postgraduate training programme for general practice were assessed at the start and at the end of training. Eight videotaped real life consultations were rated per measurement and per trainee, using the MAAS-Global scoring list. The results were compared with each other and with those of a reference group of 94 experienced GPs. RESULTS: The mean score of the MAAS-Global was slightly increased at the end of training (2.4) compared with the start (2.2). No significant difference was found between the final results of the trainees and the reference group. According to the criteria of the rating scale the performance of both trainees and GPs was unsatisfactory. CONCLUSION: The results of this study indicate that communication skills do not improve in a three-year postgraduate training comprising both a rich clinical context and a longitudinal training of communication skills, and that an unsatisfactory level still exists at the end of training. Moreover, GPs do not acquire communication skills during independent practice as they perform comparably to the trainees. Further research into the measurement of communication skills, the teaching procedures, the role of the GP-trainer as a model and the influence of rotations through hospitals and the like, is required.     

Assessing the competence of general practitioners in diagnosing generalized anxiety disorder using standardized patients

Summary. The competence of general practitioners (GPs) in diagnosing anxiety neurosis was assessed using standardized patients (SPs) unknown to the doctors. Out of a computer-generated random sample of 100 general practitioners in Kuala Lumpur, 42 volunteered to participate in the study. The results showed that the GPs can be divided into three groups: group A made the correct diagnosis and informed the SPs about their condition (11.9%); group B prescribed tranquillizers and did not inform the SPs of the actual diagnosis but instead said that they were either normal or were suffering from some stress (28.6%); and group C made various diagnoses of physical disorder or did not detect any abnormality at all (59.5%). Thus about 40% of the doctors considered an emotion-related disorder and only 12% of the doctors were confident enough to make and inform the patient of the actual diagnosis. Group A significantly ( P < 0.001) asked higher numbers of relevant questions in the signs and symptoms section of the history than the other two groups. No differences between the three groups were observed in the other two sections of history-taking (personality, family, social and precipitating factors), in the general and specific physical examination and interpersonal skills. Generally, with the exception of the interpersonal skills section, the doctors performed less than 40% of the expected tasks in every section. The study highlighted the lack of competence in making a definite diagnosis of anxiety disorder. Among those who apparently made the diagnosis (group B) or made the diagnosis with certainty (group A), there was no demonstration of appropriate treatment behaviour with respect to pharmacological intervention. One of the contributory factors could be inadequate knowledge about the signs and symptoms of the illness as well as current knowledge about the disorder.     

Improving the psychiatric skills of the general practice trainee: an evaluation of a group training course

Fourteen general practice trainees took part in a course specifically designed to improve their psychiatric interviewing skills. The trainees were instructed in the problem-based model and were taught in a group setting with the use of videotape feedback. A significant improvement was demonstrated in the trainees' ability to identify psychiatric illness accurately, and there were significant changes in their interview behaviours after training. Those who were below average before training showed the greatest improvement. The implications of these findings are discussed. Group video feedback training is as effective as one-to-one video feedback training in improving the psychiatric interviewing skills of GP trainees, and could be more widely employed in general practice vocational training.     

Training doctors for the year 2000

The World Health Organization's strategy, Health for All by the Year 2000, presents a challenge to those responsible for training doctors. Doctors are needed who are concerned to promote health not just treat disease. A review of the medical undergraduate curriculum is required to achieve this. We describe a small step towards this by the restructuring of a community medicine teaching programme so that students are introduced to health promotion and the principles of Health for All.