Content of a trainer''s report for summative assessment in general practice: views of trainers.

BACKGROUND: Summative assessment of doctors completing general practice vocational training will become compulsory for those completing training after 1 September 1996. One component of the assessment will be a trainer's report. AIM: A study set out to consider the content of the trainer's report. It aimed to determine which elements of general practice general practitioner trainers believe are the most important for independent general practice; to seek the views of trainers as to the most appropriate methods for assessing these elements in summative assessment of general practitioner registrars; and to determine how frequently trainers have been sufficiently concerned about the performance of registrars to consider not signing the form denoting satisfactory completion of the training year. METHOD: A questionnaire was sent to 1296 general practitioner trainers in the United Kingdom. The main outcome measures used were: the percentage of respondents indicating that an element was very important/crucial for independent general practice; the percentage of respondents indicating a favoured method of assessment that did not include a trainer's report at all; and the proportion of trainers who had ever considered not signing the form denoting satisfactory completion of the training year. RESULTS: The response rate was 78%. Of 75 elements examined, 31 were considered important for independent general practice by 70% or more of respondents. For 29 of these 31 elements, fewer than 30% of respondents would choose a method of assessment that did not include a trainer's report at all. Twenty-six per cent of respondents had considered not signing the form denoting satisfactory completion of the training year, which represents a trainer considering not signing this form once every 29 years. CONCLUSION: These results provide a basis for the content of a structured trainer's report for summative assessment in general practice; such a report is likely to contain about 30 items. Data from the Joint Committee on Postgraduate Training for General Practice and the present study suggest that, for every form not signed, trainers considered not signing the forms of another 13 general practitioner registrars. This highlights the need for a report that will help trainers to make the difficult decision as to whether or not a registrar is ready for independent practice.     

Clinical skills assessment.

BACKGROUND: From September 1996, all GP registrars completing vocational training in the United Kingdom must demonstrate competence by means of a four-part assessment procedure. AIM: To look at the accuracy of one of the components of vocational training: the trainer's report. METHOD: Seventy-five registrars completing their general practice training at the end of July 1997 were invited to take part in a practical skills workshop. Eight stations were designed to test practical skills and diagnostic interpretations that were included in the trainer's report, and a clinical vignette accompanied each task. The marking schedule used was developed from the minimum standards required in the trainer's report. Twenty-nine registrars (38%) took part in the workshop. RESULTS: Only one registrar passed all eight stations. The maximum number of stations failed by any one individual was five and this doctor was the only one of the sample to ultimately fail summative assessment. The majority of registrars failed by being unable to interpret clinical findings. Twenty-five registrars (86%) responded to the follow-up questionnaire. Of these, only six felt that the stations were unrealistic. All but two registrars had spent at least six months in their hospital training doing obstetrics and gynaecology but, in spite of this, only 31% of registrars were above minimum competence for vaginal and speculum examination. CONCLUSION: With one exception, registrars passed all aspects of the trainer's report. Discrepancy was found between the trainer's report and the doctor's ability to carry out clinical procedures. There is an assumption that many of these clinical skills are being taught and assessed at undergraduate level and during the hospital component, but this cannot be taken for granted. Doubt must also be cast on whether the trainers are using the trainer's report appropriately, and whether this is a valid and reliable tool to identify skills deficient in registrars for summative assessment.     

Stresses, coping mechanisms and job satisfaction in general practitioner registrars.

BACKGROUND: There is concern about the morale of general practitioner registrars. There may be stress-provoking factors that could be avoided or minimized. AIMS: The aims of the study were to assess the sources of stress and job satisfaction of general practitioner registrars, to compare registrars' job satisfaction with that of established principals using a recently published survey and to identify registrars' usual responses to stress. METHOD: A postal questionnaire survey was sent to all 143 general practitioner registrars in the West Midlands Region. The main measures were: self-rating scales of stresses associated with work and training; the Warr, Cook and Wall job satisfaction scale; and self-reported responses to stress. RESULTS: A total of 118 (83%) general practitioner registrars responded. The most potent sources of stress were family-job conflict, working for the Membership of the Royal College of General Practitioners, patients' unrealistic expectations and disruption of social life. Registrars practised good coping responses to stress. Registrars in this study had significantly greater job satisfaction than general practitioner principals in a 1993 survey for three out of 10 items measured (responsibility given, hours of work and the job as a whole) and significantly worse scores for three items (recognition for good work, rate of pay and variety of work). CONCLUSIONS: Registrars have additional stresses to those of established principals because they need to study for examinations, learn new tasks in general practice and carry out their service commitments at a stage in life when many are newly married or have a young family. Training in stress management for general practitioner registrars is recommended.     

Management confidence and decisions to refer to hospital of GP registrars and their trainers working out-of-hours.

BACKGROUND: There is concern about the educational impact and possible stress on registrars of new out-of-hours co-operatives. AIM: To compare the confidence in managing out-of-hours problems of registrars in traditional on-call rotas and co-operatives with that of their trainers. To determine how frequently registrars discussed problems out-of-hours with their trainers, and to compare the referral pattern of registrars with their trainers out-of-hours. METHOD: Analysis of log diaries of out-of-hours experiences kept by registrars and trainers over two, two-month periods in winter and summer. RESULTS: Thirty registrars (out of a possible 51) and 34 (out of a possible 52) trainers took part in the winter, and 18 registrars and 29 trainers in the summer. Registrars were confident in their management, and their confidence increased over the year (59% versus 72% difference = 12%, 95% CI = 6% to 20% for very confident). Registrars varied in their discussion of problems with trainers. When 'a little worried' they discussed their management 30 out of 53 times (57%); if 'very confident', 36 out of 576 times (6%). Registrars during the summer segment of the study referred more frequently to hospital than trainers (20% versus 10% difference = 10%, 95% CI = 3% to 17%. Registrars in traditional rotas recorded a slightly higher but statistically insignificant level of confidence in their management of problems than those registrars in cooperatives. CONCLUSIONS: While many registrars are confident in their work and are using their trainer for information appropriately, some are not. Registrars may be referring to hospital at a much higher rate than their trainers. More research is required to confirm and further explore these findings.     

Use of videotaped consultations in summative assessment of trainees in general practice.

BACKGROUND: There are many different methods by which trainees may be assessed summatively. AIM: The objective of the study was to determine if videotaped consultations could be used to identify reliably those general practitioner trainees who have not yet reached acceptable levels of competence. METHOD: Videotapes of 10 trainees carrying out normal consultations were assessed by 20 assessors for acceptable competence using a rating scale specifically developed for the purpose. RESULTS: A principal components analysis showed a strong correlation among the items in the rating scale used, indicating that a single underlying factor accounted for 76% of the overall scores. Agreement between assessors on the scoring of individual consultations was limited. There was much greater consistency with regard to the decision on overall competence, examined for the first consultation. A non-competent trainee would have a 95% probability of being identified by the process as described using two assessors for each videotape. The assessors had reached firm judgements on each trainee by the time four consultations had been viewed. CONCLUSION: The workload involved in producing and analysing the tapes is discussed. Considerations of patient consent are addressed. It is concluded that the use of videotaped consultations appears to offer a feasible and reliable method of summative assessment of general practitioner trainees.     

Audit and summative assessment: a criterion-referenced marking schedule.

BACKGROUND. 1996 will see the introduction of summative assessment of general practitioner registrars (trainees). One part of this assessment is the written submission of practical work. In the west of Scotland, audit projects have been chosen as the format for practical work. A valid and reliable marking schedule for such projects is needed. AIM. A study was undertaken to develop a criterion-referenced marking schedule for assessing registrars' audit projects for summative assessment. METHOD. Medical and non-medical professionals, in a series of workshops, compiled a list of essential elements of good audit. These features were tested and refined using registrars' audit projects. All trainers in the west of Scotland were then sent a list of 14 criteria and asked to indicate whether each criterion was an essential or desirable component of a registrar's audit project for summative assessment. A final workshop was held to develop an audit marking schedule. RESULTS. Of 155 trainers in the west of Scotland, 135 replied to the list of criteria for registrar audit projects (87%). Ten criteria were deemed essential or desirable by 80% or more of the respondents. Participants in the final workshop selected five criteria which would form the audit project marking schedule for registrars undergoing summative assessment. These were: defined reason for choice of audit project, relevance of criteria chosen, appropriate preparation and planning, appropriate interpretation of relevant data and detailed proposals for change. For an audit project to pass assessment all five criteria must be present. CONCLUSION. A criterion-referenced approach to assessing registrars' audit projects, developed from their trainers' opinions of essential or desirable criteria for good audit, is described. Further evaluation of the criterion-referenced marking schedule is required.     

Comparing trainer and trainee referral rates: implications for education and allocation of resources.

Referral rates and urgency of referral to hospital by six pairs of trainers and trainees were compared over one month in 1986. From 5846 consultations, 327 resulted in referral to hospital. Overall, the referral rates was low at 5.6 per 100 consultations; however, there was considerable variation in individual referral rates with trainees tending to follow their trainer's referral patterns, despite statistical adjustment for sharing the same practice. This paper critically examines the wisdom of using general practitioner referral rates for the purpose of resource allocation and education.     

Training for audit: lessons still to be learned.

BACKGROUND: Audit is a criterion for training in general practice, and registrars are reliant on their trainers' teaching of basic audit methods. Their ability to teach this had been assumed, but registrars' projects submitted as part of summative assessment offered an opportunity to test this. AIM: To test trainers' knowledge of basic audit methods. Their knowledge was based on an ability to recognize key audit criteria using a marking schedule that they had helped to create. METHOD: All 158 trainers in the west of Scotland were asked to mark five general practice registrar audit projects using a marking schedule consisting of five independent criteria. Each project had one criterion that was below a level of minimum competence, as agreed by a group of 'expert' assessors. RESULTS: A total of 114 trainers (72%) completed the marking exercise of five audit projects. Three (3%) correctly identified the five criteria that were below minimum competence. They did this by highlighting many other criteria not below minimum competence. For all trainers, there was a direct relationship between the number of criteria they correctly identified as being below minimum competence and the total number of other criteria that they incorrectly identified. CONCLUSION: Trainers are failing to recognize basic audit methodology using a marking schedule they themselves helped to design. This has implications for their ability to teach audit to their registrars and may explain some of the difficulty in implementing audit.     

Simulated surgery in the summative assessment of general practice training: results of a trial in the Trent and Yorkshire regions.

BACKGROUND: General practice registrars are now required to undertake a summative assessment of their consulting skills. Simulated surgeries have been developed as an alternative to the existing method of assessing video-recorded consultations. AIM: To evaluate the simulated surgery assessment method, developed in the General Practice Postgraduate Education Department in Leicester, for use in assessing general practice consultation skills. METHOD: General practice registrars in Leicester performed two eight-patient simulated surgeries separated by four weeks. Assessment outcomes were compared to demonstrate the consistency of the method. Pilot surgeries in Yorkshire were videotaped, and then rated by video-raters trained for summative assessment. RESULTS: The method consistently identified those registrars who were competent and those who were not yet competent in consulting skills. It proved acceptable to candidate doctors and has fewer resource requirements for both examiners and candidates than other consulting skills assessment methods. CONCLUSION: The method developed in Leicester and successfully transferred to Yorkshire is feasible on a large scale, and offers an acceptable alternative to other consulting skills assessment methods. In this study it consistently identified competent from incompetent candidate doctors.     

Summative assessment of vocational trainees: results of a 3-year study.

BACKGROUND: Summative assessment of general practice trainees will be introduced across the UK for trainees completing vocational training after 1 September 1996. AIMS: A study was set up to assess the reliability of the consulting skills assessment which will be used to develop audit and factual knowledge testing as summative assessment tools and to determine the probable impact of the system on the numbers of trainees identified as not yet competent for independent practice. METHOD: Videotaped consultations, an audit project, a trainer's report and a multiple-choice paper were evaluated for 359 trainees in the West of Scotland completing their training between 31 July 1993 and 31 July 1995. RESULTS: A total of 77 trainees (22%) were identified as being potentially of doubtful competence and 17 (5%) were adjudged to be as yet incompetent for independent practice. Videotaped consultation produced the highest pickup rate (14) followed by trainer's judgement (3). Subsequent analysis of the video tapes identified one trainee who should have been picked up by this method but was not. CONCLUSION: The videotape instrument reliably identified the non-competent trainees. Audit and factual knowledge tests have a distinct role in summative assessment. The proportion of trainees being refused certificates by this assessment is likely to increase considerably but will still represent only about 5% of all trainees.     

A Delphi technique as a method for selecting the content of an electronic patient record for asthma

INTRODUCTION: An electronic patient record (EPR) with disease-specific data may support improvement of the quality of care for patients with chronic diseases. The structure and content of such a record can only be assessed by clinicians in co-operation with IT-specialists, because, the result has to be clinically relevant, easily accessible and adjusted to the information needs of different workers in primary care. METHODS: We applied a modified Delphi-procedure--a method characterized by anonymous written comments by an expert panel. The panel had to agree about the question whether or not an item should be included in the EPR. The questions for the written comments were prepared by a steering committee (general practitioners (GP) and health scientists, either expert in asthma and disease management or IT-specialist) based on the guidelines for diagnosing and treating asthma of the Dutch College of General Practitioners (DCGP). When agreement within the panel was < or = 70%, we sent a modified format to the expert panel for reassessment. RESULTS: Three written commentary rounds were necessary, in which 95 potential items were discussed with the expert panel. In the first round they selected 50 items relating to diagnosing asthma and 22 concerning the treatment of asthma. During the second round 17 items were still under discussion and six were rejected. In subsequent rounds, the expert-panel assessed the best registration format (operationalisation). The written rounds failed to create a full consensus. Therefore the study ended with a consensus meeting of the expert panel. Due to the presence and contribution of nearly all experts, consensus could be reached about the structure and contents of the EPR on asthma. DISCUSSION: The modified Delphi procedure, proved to be a feasible method for selecting the optimal content of an electronic registration protocol. Both, written and verbal commentary rounds were necessary. The existence of a set of guidelines was essential.     

Ensuring best practice in genomics education and evaluation: reporting item standards for education and its evaluation in genomics (RISE2 Genomics)

PurposeWidespread, quality genomics education for health professionals is required to create a competent genomic workforce. A lack of standards for reporting genomics education and evaluation limits the evidence base for replication and comparison. We therefore undertook a consensus process to develop a recommended minimum set of information to support consistent reporting of design, development, delivery, and evaluation of genomics education interventions.MethodsDraft standards were derived from literature (25 items from 21 publications). Thirty-six international experts were purposively recruited for three rounds of a modified Delphi process to reach consensus on relevance, clarity, comprehensiveness, utility, and design.ResultsThe final standards include 18 items relating to development and delivery of genomics education interventions, 12 relating to evaluation, and 1 on stakeholder engagement.ConclusionThese Reporting Item Standards for Education and its Evaluation in Genomics (RISE2 Genomics) are intended to be widely applicable across settings and health professions. Their use by those involved in reporting genomics education interventions and evaluation, as well as adoption by journals and policy makers as the expected standard, will support greater transparency, consistency, and comprehensiveness of reporting. Consequently, the genomics education evidence base will be more robust, enabling high-quality education and evaluation across diverse settings.     

General practitioner reaccreditation: use of performance indicators.

There has been increasing debate about reaccreditation of general practitioners over the last few years with contributions from the General Medical Services Committee, the Royal College of General Practitioners and the National Association of Health Authorities and Trusts. The implications of proposals in terms of cost, logistics and organization are discussed in this paper, in light of experience with the introduction of summative assessment for general practitioner registrars (trainees) and a programme of training practice visits in West Midlands Region. A model for reaccreditation for all general practitioners is proposed which is professionally led and sensitive to the needs of patients and health service managers. The basic proposition is that publicly owned family health services authority data could be used as initial performance indicators for professional competence. The model is dependent on the rebuttal of the null hypothesis: there is no link between the competence of a general practitioner and his or her achievements in the suggested performance indicators. If the performance indicators (educational commitments, prescribing data, health promotion activity and immunization targets, and service elements) can be shown to correlate with possession of the attributes for independent practice as defined by the General Medical Council, then a relatively inexpensive and simple system of reaccreditation could be envisaged. General practitioners who are recorded as achieving set performance indicator targets would be accorded automatic reaccreditation. Only substandard practitioners would be required to be assessed further by a visiting team of local general practitioner peers and, if appropriate, a remedial education strategy introduced. This method would complement the General Medical Council scheme for assessing an individual doctor's persistent poor performance, which could then be invoked as a last resort.     

What determines competence within a general practice consultation? Assessment of consultation skills using simulated surgeries.

A method for assessing the consultation skills required in general practice using patient simulators within a simulated surgery has been developed in Leicester, aimed at registrars coming to the end of their three-year training programme. This paper addresses the issues of validity, describing the technique as it has evolved over the past three years, and paying particular attention to ways in which the consultation was made to feel 'real' for candidates. As well as testing clinical issues during consultations, simulated surgery incorporates the views of the patient into the assessment process. Feedback from registrars has been positive and the method presents an alternative to video-recording the consultation for the purpose of assessment.     

Clinically important differences in health-related quality of life for patients with asthma: an expert consensus panel report.

BACKGROUND: Standards for change in health-related quality of life (HRQoL) measures used among asthmatic patients have been established by considering only patient preferences to determine important differences and may not reflect an informed clinical evaluation of change. OBJECTIVE: To establish clinically important difference (CID) standards through the consensus of an expert physician panel for the Juniper Asthma Quality of Life Questionnaire (AQLQ) and Medical Outcomes Study Short-Form 36-Item Health Survey (SF-36, version 2) when used to measure HRQoL among asthmatic patients. METHODS: We organized an 8-person panel of physicians familiar with measuring HRQoL among asthmatic patients with the AQLQ and the SF-36. This expert panel participated in 2 rounds of a modified Delphi process before attending an in-person consensus meeting to establish the CID standards for small, moderate, and large changes in the AQLQ and the SF-36. After the consensus meeting, the panel used an iterative improvement process to cooperatively complete their final report. RESULTS: The expert panel's thresholds for detecting CIDs in the domains of the AQLQ were much higher than previously established levels using patient-perceived changes. CIDs for asthma were also ascertained on the scales of the SF-36, version 2, which were markedly greater than previously cited cross-sectional differences between patient groups. CONCLUSIONS: The CID standards established by this expert panel elucidate a potential distinction between patient and physician perspectives of important HRQoL changes. The many stakeholders of HRQoL difference standards should consider this distinction when adopting standards to evaluate patient change.     

The Relapsing Polychondritis Disease Activity Index: Development of a disease activity score for relapsing polychondritis

ObjectiveThe rarity of relapsing polychondritis (RP) has hindered the development of standardized tools for clinical assessment. Here, we describe the development of a preliminary score for disease assessing activity in RP, the Relapsing Polychondritis Disease Activity Index (RPDAI).MethodsTwenty-seven RP experts participated in an international collaboration. Selection and definition of items for disease activity were established by consensus during a 4-round internet-based Delphi survey. Twenty-six experts assessed the Physician's Global Assessment (PGA) of disease activity on 43 test cases on a 0–100 scale, yielding a total of 1118 PGA ratings. The weight of each item was estimated by multivariate regression models with generalized estimating equation, using PGA as the dependent variable.ResultsExperts decided in consensus that the RPDAI should consider the 28-day period before each RPDAI assessment. Inter-rater reliability assessed by the intra-class correlation coefficient for the 1118 PGA ratings was 0.51 (CI95%: 0.41–0.64). The final RPDAI score comprised 27 items with individual weights ranging from 1 to 24 and a maximum theoretical RPDAI score of 265. Correlation between the RPDAI scores calculated based on the weights derived from the final multivariate model, and the 1118 PGA ratings was good (r = 0.56, p < 0.0001).ConclusionWe have developed the first consensus scoring system to measure disease activity in relapsing polychondritis (see www.RPDAI.org for online scoring). This tool will be valuable for improving the care of patients with this rare disease.     

Management of Cerebral Concussion in Sports: The Athletic Trainer's Perspective

OBJECTIVE: To present a new approach in the evaluation and management of concussion from the athletic trainer's perspective. BACKGROUND: The evaluation and management of concussion continues to be a controversial topic among sports medicine professionals. Inconsistent management, lack of objective data, and confusion concerning assessment techniques may lead to inappropriate decisions regarding when to return an athlete to competition after concussion. In this article, we provide recommendations and considerations for the certified athletic trainer in the management of concussion. We also present a quantifiable assessment technique that provides more information on which return-to-play decisions can be made; this technique can be used during the initial sideline examination as well as during subsequent follow-up examinations. RECOMMENDATIONS: Certified athletic trainers and team physicians should consistently use appropriate grading scales. Assessment of concussion should include a symptom checklist, the Balance Error Scoring System, and the Standardized Assessment of Concussion, and the results should be compared with the athlete's normal baseline scores. Follow-up neuropsychological and postural stability testing is recommended. Return-to-play decisions should be based on the grade of concussion, scores on objective tests, and presence of concussive symptoms during exertional activities.     

MMPI偏执分量表的计算机自适应测验可行性分析

目的:对MMPI偏执分量表的CAT研究的可行性进行探索,并在普通人群和病人中检验CAT测验的结果。方法:首先采用项目反应理论对Pa分量表的单维性进行检验,其次依据项目反应理论挑选项目,形成用于CAT的删减版Pa-D,最后在病人和普通人中验证CAT的可行性。结果:①Pa和Pa-D都能满足单维性的假设,可以用IRT分析项目。②Pa-D的项目参数在合理可接受的范围内。结论:Pa和Pa-D均满足单维性,且删除题目未对测验的信息造成明显的影响。采用后验CAT模拟可以节省一半的题目。     

Communication breakdown: clinicians disagree on subacromial impingement

“Subacromial impingement syndrome (SIS)” is often used as a diagnostic label, but has become more controversial as such in the literature. We assessed views on SIS in clinical practice using a survey with 63 0–10 VAS items among orthopedic surgeons and physical therapists from the United States and the Netherlands. Multivariate regression and cluster analyses were applied to identify consensus items and to study profession and/or nationality effects on item ratings. Most items received neutral or highly variable ratings. Twenty-nine were considered associated with SIS, including worsening of pain with overhead activities, painful arc and a positive Neer’s test. Seven items were regarded pleading against SIS, including loss of passive motion. Activity modifications and physical therapy are the most important treatments according to therapists, who highly valued motion-related etiologic mechanisms. Surgeons, with higher ratings for intrinsic and anatomic etiologies, appreciated the use of subacromial corticosteroids and surgery. Clinicians from different professional backgrounds have different views on what SIS is, and even within professional groups, variations are substantial. This has to be taken into account when communicating about SIS symptoms, for example, in intercollegial consultation or scientific research. The authors suggest cautious use of (subacromial) impingement syndrome as a diagnostic label.