Can communication skills be assessed independently of their context?

In many examinations, communication skills tend to be treated as if they are a single attribute independent of the context of the communication. However, it is clear that such assessments are confounded by candidates' knowledge or lack of knowledge of the medical issues about which they are communicating. In the 1990 Part One examination for Membership of the Royal New Zealand College of General Practitioners candidates were provided with all the essential knowledge relevant to the problem they were to communicate about. Despite this, performance was still seen to be context specific, demonstrating that such specificity is not purely knowledge related. Candidates completing the examination were observed to share information about the cases with candidates about to commerce. There was no evidence that performance was enhanced by such breaches in examination security.     

Acquisition of communication skills in postgraduate training for general practice

PURPOSE: The evidence suggests that a longitudinal training of communication skills embedded in a rich clinical context is most effective. In this study we evaluated the acquisition of communication skills under such conditions. METHODS: In a longitudinal design the communication skills of a randomly selected sample of 25 trainees of a three-year postgraduate training programme for general practice were assessed at the start and at the end of training. Eight videotaped real life consultations were rated per measurement and per trainee, using the MAAS-Global scoring list. The results were compared with each other and with those of a reference group of 94 experienced GPs. RESULTS: The mean score of the MAAS-Global was slightly increased at the end of training (2.4) compared with the start (2.2). No significant difference was found between the final results of the trainees and the reference group. According to the criteria of the rating scale the performance of both trainees and GPs was unsatisfactory. CONCLUSION: The results of this study indicate that communication skills do not improve in a three-year postgraduate training comprising both a rich clinical context and a longitudinal training of communication skills, and that an unsatisfactory level still exists at the end of training. Moreover, GPs do not acquire communication skills during independent practice as they perform comparably to the trainees. Further research into the measurement of communication skills, the teaching procedures, the role of the GP-trainer as a model and the influence of rotations through hospitals and the like, is required.     

Evaluating family counselling skills training for family practice

Educational programmes for family practice should develop family counselling skills of students to moderate levels of competence. Few specific training programmes are part of the regular curriculum and of these few are evaluated. Twenty-three residents enrolled in a 2-year family practice programme in Toronto, Canada participated in a non-randomized control intervention study to assess a specific training programme. Pre-training counselling skills, and ability were assessed using two types of generally recognized measurement instruments, the Carkhuff Stems of Communication Skills and the Carkhuff Discrimination Index. The treatment group entered the training programme which involved supervised family counselling 8 half-days each week for one month as part of their usual rotations. They completed a second set of instruments following this course to assess immediate within-group change and then both they and the control group completed a set about one year later to measure differences. Initial scores for all residents showed a less than 'minimally facilitative' level of counselling response on both empathy and warmth dimensions. Following the course, the treatment group scored above this level and significantly better than the control group. Furthermore, the former showed a 20% improvement in ability to discriminate between effective and ineffective helping responses which was sustained over one year, while the control group's scores became worse. Teaching of counselling skills can be effective.     

Performance correlates in a postgraduate psychiatric examination

This paper has reported some aspects of an analysis of the correlates of success in candidates'first attempts at the postgraduate psychiatric examinations of the R.A.N.Z.C.P. These are developed to suggest some attributes which selectors of trainees could take into account. Profitable future areas for research are delineated.     

Does performance improve when candidates resit a postgraduate examination?

Summary. Luck plays some role in passing any examination. When candidates pass a postgraduate examination at the second, third or subsequent attempt is it because their knowledge has truly improved or because they have at last been lucky? In this paper a simple model, requiring knowledge only of the pass rates at resits, and of the reliability of the examination, is applied to the MRCGP examination of the Royal College of General Practitioners. Candidates increase their true ability before second and third attempts at the examination, after which ability declines.     

Competence-based summative assessment of a student-directed course: involvement of key stakeholders

The 5-week module in general practice for final-year students at the University of Sheffield is based on practice attachments and student-directed learning in small groups. This paper describes how the summative assessment process of the module was revised to incorporate the notion of competence-based assessment, and how general practitioner tutors, departmental tutors and students were involved in this revision. The question ‘What are students expected to know and be able to do by the end of the module?’ was answered in terms of a statement of the key purpose of the module and a list of intended learning outcomes. The question ‘How can we find out if students have achieved these outcomes?’ was addressed by developing check-lists of criteria for observed behaviours and for the written products of students' actions.     

The direct clinical examination: an alternative method for the assessment of clinical psychiatry skills in undergraduate medical students

Summary: Because of dissatisfaction with the traditional long case procedure as a method of examining the clinical competence of medical students undertaking a psychiatry term, an alternative 'direct' method whereby two examiners observe the interaction between student and patient has been developed and is described. This method of examining allows the examiners to set and evaluate case-specific tasks. It is demonstrated that the two examiners achieve satisfactory inter-rater reliability both with respect to the mark awarded and the difficulty the patient presents and that, as one would wish, these two measures do not correlate. Students' opinions regarding the examination were assessed pre- and post-examination using visual analogue scales. The students found the examination stressful but rated the method as an appropriate form of clinical assessment both before and after their examination. The method is seen as having several advantages which must be set against the disadvantage of its being relatively expensive of examiners' time.     

Educational assessment of general practice experience for psychiatric trainees

Eighteen psychiatric trainees spent 6 months each as general practice trainees. The educational impact of the experience was assessed by a self-assessment questionnaire, a semi-structured interview and a videotaped interview with a psychiatric patient. Each assessment was conducted at baseline and after 12 months. A control group of 14 trainees was recruited from the same rotation. On the self-assessment questionnaire, the study registrars rated their abilities to solve general medical problems significantly improved compared to controls. They had also acquired greater understanding of the limitations of their knowledge and their legal responsibilities towards their patients. The semi-structured interview failed to distinguish between the two groups. Videotapes for rating at baseline and follow-up were available for only 17 of the trainees. Assessment of the tapes used the Maguire Scale and the Interview Behaviour Scale. Neither scale demonstrated any intervention effect. The interviews were all characterized by a preponderance of 'closed psychological' and 'checking-out' questions. It appears that psychiatric trainees' interviewing styles had not been influenced by the experience. This study suggests that psychiatric trainees gain greater confidence in their role as a doctor and greater understanding of the scope and nature of general practice by such an attachment. It is unclear whether or not supplementary interviewing skills had been acquired which were not utilized in the taped interview, which conforms very much to traditional psychiatric examination behaviour. Trainees were reassured that they had increased their knowledge without losing any of their specific professional skills.     

Comparison of two instruments for assessing communication skills in a general practice objective structured clinical examination

OBJECTIVE: In recent decades, there has been increased interest in tools for assessing and improving the communication skills of general practice trainees. Recently, experts in the field rated the older Maas Global (MG) and the newer Common Ground (CG) instruments among the better communication skills assessment tools. This report seeks to establish their cross-validity. METHODS: Eighty trainees were observed by 2 raters for each instrument in 2 standardised patient stations from the final year objective structured clinical examination for Belgian trainee general practitioners. Each instrument was assigned 6 raters. RESULTS: Trainees showed the lowest mean scores for evaluating the consultation (MG7), summarising (MG11), addressing emotions (MG9) and addressing feelings (CG5). Inter-rater kappa statistics revealed fair-to-moderate agreement for the MG and slight-to-fair agreement for the CG. Cronbach's alpha was 0.78 for the MG and 0.89 for the CG. A generalisability study was only feasible for the MG: it was more helpful to increase the number of cases than the number of raters. Agreement between the instruments was examined using kappa statistics, Bland-Altman plots and multi-level analysis. Ranking the trainees for each instrument revealed similar results for the least competent trainees. Variances between and within trainees differed between instruments, whereas case specificity was comparable. Multi-level analysis also revealed a rater-item interaction effect. CONCLUSIONS: The 2 instruments have convergent validity, but the drawbacks of the CG, which has fewer items to be scored, include lower inter-rater reliability and score variance within trainees.     

Effects of the vocational training of general practice consultation skills and medical performance

The effects of the vocational training of general practitioners in the Netherlands on the consultation skills and medical performance of junior doctors were studied. Results obtained at a training institute providing systematic training in these skills (Nijmegen) were compared with those at an institute taking a problem-based learning approach (Groningen). Trainees (n = 63) audiotaped consultations and recorded their medical performance at the start and at completion of training. The skills were evaluated with the aid of validated criteria and medical 'protocols'. Data on 631 pre-training and 624 post-training consultations were compared. Changes in consultation skills and medical performance occurred at both institutes and proved more marked at the institute providing systematic training. Improved medical performance was found to be associated with improved consultation skills. Enhanced clinical knowledge was found to be related to improved medical performance and consultation skills. The most profound changes were found in junior doctors who had started at a lower level of consultation skills and medical performance.     

The modified essay question: an evaluation of its use in a family medicine clerkship

This paper describes the extensive experience of the Department of Family Medicine at Jefferson Medical College in utilizing the modified essay question (MEQ) as a final examination for its required third-year clerkship in family medicine. The results of a preliminary study are presented, comparing the MEQ as a method of evaluation with the multiple choice question (MCQ) format on similar content material. MCQ performance was found to be a better predictor of National Board Part I and II scores, but neither format was found to be a good predictor of postgraduate performance. The data were statistically significant (P less than 0.05) that MEQ performance was related to postgraduate performance in the area of professional attitude while the MCQ was not; however, the number of students in the analysis was small. Additional studies are planned to test this hypothesis further.     

Training psychiatrists and family doctors in evaluating interpersonal skills

Ten psychiatrists and 15 family doctors were asked to score videotapes of patient-doctor encounters before and after each of two training periods. One period focused on the theory of assessment of doctors' interpersonal skills, while the other was purely practical. Results indicate that after one training session in either theory or practice, both groups of doctors achieved a significantly higher interrater reliability. The second session, which crossed over theory and practice of assessment, did not increase the improvement in interrater agreement achieved by the first training period. Although both groups of doctors showed a significant increase in interrater agreement, psychiatrists exhibited greater reliability scores than family doctors before the experiment as well as after the second training session. These results were discussed in terms of their implicatioins for future research on the doctor-patient relationship.     

Predictive validity of a written knowledge test of skills for an OSCE in postgraduate training for general practice

PURPOSE: To examine the validity of a written knowledge test of skills for performance on an OSCE in postgraduate training for general practice. METHODS: A randomly-selected sample of 47 trainees in general practice took a knowledge test of skills, a general knowledge test and an OSCE. The OSCE included technical stations and stations including complete patient encounters. Each station was checklist rated and global rated. RESULTS: The knowledge test of skills was better correlated to the OSCE than the general knowledge test. Technical stations were better correlated to the knowledge test of skills than stations including complete patient encounters. For the technical stations the rating system had no influence on the correlation. For the stations including complete patient encounters the checklist rating correlated better to the knowledge test of skills than the global rating. CONCLUSION: The results of this study support the predictive validity of the knowledge test of skills. In postgraduate training for general practice a written knowledge test of skills can be used as an instrument to estimate the level of clinical skills, especially for group evaluation, such as in studies examining the efficacy of a training programme or as a screening instrument for deciding about courses to be offered. This estimation is more accurate when the content of the test matches the skills under study. However, written testing of skills cannot replace direct observation of performance of skills.     

Standardized patients: a long-station clinical examination format

Summary. Data from the first 20 periods of a long-station clinical performance examination for a 4-week required clerkship in family medicine were examined in order to assess the reliability and validity of the examination. Data from 304 students were examined for station, case scenario and examiner effects and results compared to short-station formats. A significant examiner effect was found but there were no differences in student performance for station or case scenario. These findings reflect examiner specificity cited in the literature for short station examinations, but not case specificity. The source of variability for this examination appears to be primarily examiner effect. There was a significant correlation between student scores on the two cases, and raters tended to rank order students similarly in spite of variability in mean rater score. Scores on the CPE correlated with other measures of clinical performance as well as other methods of student evaluation for the clerkship providing some evidence for construct and criterion-related validity. CPE cases were developed from clerkship objectives but examination of the test blueprint revealed some gaps in the extent to which the CPE covers the course content. CPE developers are working to increase interrater reliability through examiner training and further standardize case scenarios through checklists and patient training. Additional cases are being developed to increase the content validity of the examination.     

The effects of continuing medical education on family doctor performance in office practice: a randomized control study

A randomized controlled study was conducted to determine if specifically designed continuing medical education in the fields of cardiovascular and cancer medicine could change doctor office behaviour significantly. Thirty-one volunteer family doctors from 25 offices participated. Six (three cardiovascular and three cancer) learning objectives were defined. Two educational formats were selected as the independent variables: (1) group interaction opportunities (face-to-face and teleconference); and (2) concisely written newsletters. Chart measures of doctor performance prior to and 6 and 12 months following education served as the dependent variables. The family doctors receiving education were found to perform the recommended behaviours significantly more than those who did not receive the education (P less than 0.05) at 6 months post-education. This difference was maintained at the 12-month post-educational period for one of the educational programmes offered. A carefully planned programme of continuing medical education will result in favourable changes in the office practice of volunteer doctors. These changes can persist for as long as 12 months. Adherence to several essential learning principles is required.     

The rating of pre-clerkship examination questions by postgraduate medical students: an assessment of quality and relevancy to medical practice

OBJECTIVE: To determine whether postgraduate students are able to assess the quality of undergraduate medical examinations and to establish whether faculty can use their results to troubleshoot the curriculum in terms of its content and evaluation. SUBJECTS: First and second year family medicine postgraduate students. MATERIALS: A randomly generated sample of undergraduate medical examination questions. METHODS: Postgraduate students were given two undergraduate examinations which included questions with an item difficulty (ID) > 0.60. The students answered and then rated each question on a scale of 1-7. RESULTS: The percentage of postgraduate students answering each question correctly correlated significantly with the average perceived relevance (Examination 1: r=0.372; P < 0.05; Examination 2: r=0.458; P < 0.05). Questions plotted for average postgraduate/undergraduate performance ratio versus the average perceived relevance were significantly correlated (Examination 1: r=0.462; P < 0.01; Examination 2: r=0.458; P < 0.05). CONCLUSIONS: This study offers a method of validating question appropriateness prior to examination administration. The design has the potential to be used as a model for determining the relevancy of a medical curriculum.     

Longitudinal reliability of the Royal Australian College of General Practitioners Certification Examination

The examination for Fellowship of the Royal Australian College of General Practitioners is a long-standing examination that is regarded as a valid and comprehensive assessment of competence for general practice in Australia. Income and employment opportunities now depend on passing this battery of eight subtests. This paper describes the structures of the examination and reports an analysis of the reliability arid efficiency of the examinations, based on data from five consecutive examinations. Results indicate that the examination achieves acceptable overall reliability, although suggestions for changes to improve reliability are made. A decision on implementing any changes must take into consideration any consequent changes in validity.     

The effects of the introduction of a system of mandatory formative assessment for general practice trainees

A series of questionnaires was used to determine the amount of formative assessment going on within the West of Scotland Region of the UK and its perceived value by trainees (General Practice [GP] Registrars) both at the beginning and end of training. The first survey was carried out in 1989 and the second in 1994. Trainers and course organizers were surveyed in 1994 to determine their attitudes to the use of videotaped consultations for formative assessment. In 1991, the region had initiated a mandatory formative assessment programme which included regular use of videotaped consultations, confidence rating scales and Manchester ratings (RCGP Occasional Paper Number 40). The use of a range of assessment methods for formative assessment of GP Registrars increased considerably between 1989 and 1994. The percentage of trainees using videotaped consultation analysis increased from 76 to 94%, for Manchester ratings from 52 to 68% and for confidence rating scales from 63 to 74%. Video and confidence scales were rated highly by trainees who were assessed by them and by most trainees at the start of the year, but less highly by those who had not been assessed by them by the end of the trainee year. Manchester ratings were not thought to be as useful, and for these there was no difference between users and non-users. Despite the mandatory system, a significant number of trainees (76%) were still not receiving the minimum assessment stipulated. Trainers rated lack of time as the main limiting factor to the greater use of video. We conclude that trainees who are exposed to assessment methods, particularly video, find it useful, but that some trainees are still receiving less than is optimal. We propose increased trainer education and intensified monitoring of the assessment carried out in training Practices.