Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review

Objective  To develop generic criteria for the global assessment of clinical procedural competence and to quantify the extent to which existing checklists allow for holistic assessment of procedural competencies.
Methods  We carried out a systematic review and qualitative analysis of published clinical procedural skills assessment checklists and enumerated the contents of each. Source materials included all English-language papers published from 1990 to June 2005, identified from 18 databases, which described or referred to an assessment document for any clinical procedural skill. A pair of reviewers identified key generic themes and sub-themes through in-depth analysis of a subset of 20 checklists with iterative agreement and independent retesting of a coding framework. The resulting framework was independently applied to all checklists by pairs of reviewers checking for the emergence of new themes and sub-themes. Main outcome measures were identification of generic clinical procedural skills and the frequency of occurrence of each in the identified checklists.
Results  We identified 7 themes ('Procedural competence', represented in 85 [97%] checklists; 'Preparation', 65 [74%]; 'Safety', 45 [51%]; 'Communication and working with the patient', 32 [36%]; 'Infection control', 28 [32%]; 'Post-procedural care', 24 [27%]; 'Team working', 13 [15%]) and 37 sub-themes, which encapsulated all identified checklists. Of the sub-themes, 2 were identified after the initial coding framework had been finalised.
Conclusions  It is possible to develop generic criteria for the global assessment of clinical procedural skills. A third and a half of checklists, respectively, do not enable explicit assessment of the key competencies 'Infection control' and 'Safety'. Their assessment may be inconsistent in assessments which use such checklists.     

Quality of reporting of experimental studies in medical education: a systematic review

OBJECTIVE: Determine the prevalence of essential elements of reporting in experimental studies in medical education. DESIGN: Systematic review. DATA SOURCES: Articles published in 2003 and 2004 in Academic Medicine, Advances in Health Sciences Education, American Journal of Surgery, Journal of General Internal Medicine, Medical Education, and Teaching and Learning in Medicine. REVIEW METHODS: Articles describing education experiments, including evaluation studies with experimental designs, were identified (n = 185) by reviewing titles and abstracts. A random sample (n = 110) was selected for full review. The full text of each article was evaluated for the presence of guideline-based features of quality reporting: a critical literature review, conceptual framework, statement of study intent (e.g. aim, research question, or hypothesis), statement of study design, definition of main intervention and comparison intervention or control group, and consideration of human subject rights. RESULTS: Of the 105 articles suitable for review, 47 (45%) contained a critical literature review and 58 (55%) presented a conceptual framework. A statement of study intent was present in 80 articles (76%), among which the independent and dependent variables were operationally defined in 38 (47%) and 26 articles (32%), respectively. A total of 17 articles (16%) contained an explicit study design statement. Among the 48 studies with a comparison group, 35 (73%) clearly defined the comparison intervention or control group. Institutional review board approval or participant consent was reported in 44 articles (42%). CONCLUSIONS: The quality of reporting of experimental studies in medical education was generally poor. Criteria are proposed as a starting point for establishing reporting standards for medical education research.     

Clinical skills education: outcomes of relationships between junior medical students, senior peers and simulated patients

Objectives  Peer-assisted learning (PAL) has been reported to have educational benefits in cross-year, small-group teaching in other contexts. Accordingly, we explored whether senior medical students are effective tutors for their junior peers in clinical skills education, and how the participants in the learning triad (tutors, learners and simulated patients [SPs]) perceive the learning environment created in PAL.
Methods  Year 2 students were randomly allocated to one of two groups for skills training. Group 1 ( n  = 64) were tutored by volunteer Year 6 students, and Group 2 ( n  = 67) by paid doctors. The results of both groups in a clinical skills examination were compared using an independent samples t -test. Qualitative data, obtained from Year 2 students ( n  = 125) by written questionnaire and Year 6 students ( n  = 11) and SPs ( n  = 3) by focus group interviews, were analysed for themes.
Results  Students receiving PAL did at least as well in the clinical skills examination as students with qualified tutors (difference in mean total score: 0.7 marks out of 112; 95% confidence interval − 3.8 to 2.4). The PAL environment was perceived as 'comfortable' and fostered the development of confidence in all participants. Peer tutors created a more active learning environment than doctor tutors for both learners and SPs and reported personal benefits from teaching.
Conclusions  With appropriate support, volunteer Year 6 student tutors are as effective as graduate doctors for small-group structured tutorials in clinical skills. Educational relationships were forged between all participants in the learning triad.     

Peer-assisted versus faculty staff-led skills laboratory training: a randomised controlled trial

Objectives Although peer‐assisted learning (PAL) is widely employed throughout medical education, its effectiveness for training in technical procedures in skills laboratories has been subject to little systematic investigation. We conducted a prospective, randomised trial to evaluate the hypotheses that PAL is effective in technical skills training in a skills laboratory setting, and PAL is as effective as faculty staff‐led training. Methods Volunteer Year 3 medical students were randomly assigned to one of three groups. Two of these received regular skills training from either cross‐year peer tutors or experienced faculty staff. Following training, both groups were assessed using an objective structured clinical examination (OSCE) (three stations assessing various injection techniques) which was video‐recorded. Two independent video assessors scored the OSCEs using binary checklists and global ranking forms. A third student group was assessed prior to training and served as a control group. Results A total of 89 students (mean age 23.0 ± 0.2 years; 41 male, 48 female) agreed to participate in the trial. Confounding variables including prior training as a paramedic or previous experience in performing the technical procedures did not significantly differ between the three study groups. In the OSCE, PAL (58.1 ± 1 binary points, 4.9 ± 0.1 global ranking points) and faculty‐led groups (58.3 ± 1 binary points, 4.7 ± 0.1 global ranking points) scored significantly higher than the control group (33.3 ± 1 binary points, 2.7 ± 0.1 global ranking points; all P < 0.0001). There was no significant difference between the PAL and faculty‐led groups (P = 0.92 for binary checklists, P = 0.11 for global rankings). Conclusions Peer‐assisted learning is a successful method for learning technical procedures in a skills laboratory setting and can be just as effective as the training provided by experienced faculty staff.     

Approaches to medical student remediation after a comprehensive clinical skills examination

OBJECTIVE: Most US medical schools conduct comprehensive clinical skills assessments during Years 3 and 4. This study explores strategies used to identify and remediate students who perform poorly on these assessments. METHODS: In the academic year 2005-06, we conducted 33 semi-structured interviews with individuals responsible for standard setting in and remediation after their schools' comprehensive clinical skills assessments. We coded interviews to identify major themes. RESULTS: Prior to remediation, some schools employed a 'verification' step to ensure the accuracy of the failing score or need for remediation. Participants described a remediation process that included some or all of 3 steps. Firstly, students' specific learning deficits were diagnosed. Next, students participated in remedial activities such as performance review sessions or practice with standardised or actual patients. Lastly, students were re-tested, usually with a shorter, more formative examination. All participants reported using a diagnostic step, most offered or required remedial activities and many re-tested, although schools varied in the emphasis placed on each step. Many participants cited the individualised attention students received from remediation faculty staff as a strength of their approach, although they raised concerns about the substantial time demands placed on remediation faculty. Most respondents reported some dissatisfaction with their school's remediation process, particularly uncertainty about efficacy or rigour. CONCLUSIONS: Schools vary in the intensity and scope of remediation offered to students who perform poorly on clinical skills assessments. Although many schools invest significant resources in remediation, the effect of these efforts on students' subsequent clinical performance is unknown.     

Development of a tool to support holistic generic assessment of clinical procedure skills

Context  The challenges of maintaining comprehensive banks of valid checklists make context-specific checklists for assessment of clinical procedural skills problematic.
Objectives  This paper reports the development of a tool which supports generic holistic assessment of clinical procedural skills.
Methods  We carried out a literature review, focus groups and non-participant observation of assessments with interview of participants, participant evaluation of a pilot objective structured clinical examination (OSCE), a national modified Delphi study with prior definitions of consensus and an OSCE. Participants were volunteers from a large acute teaching trust, a teaching primary care trust and a national sample of National Health Service staff.
Results  In total, 86 students, trainees and staff took part in the focus groups, observation of assessments and pilot OSCE, 252 in the Delphi study and 46 candidates and 50 assessors in the final OSCE. We developed a prototype tool with 5 broad categories amongst which were distributed 38 component competencies. There was > 70% agreement (our prior definition of consensus) at the first round of the Delphi study for inclusion of all categories and themes and no consensus for inclusion of additional categories or themes. Generalisability was 0.76. An OSCE based on the instrument has a predicted reliability of 0.79 with 12 stations and 1 assessor per station or 10 stations and 2 assessors per station.
Conclusions  This clinical procedural skills assessment tool enables reliable assessment and has content and face validity for the assessment of clinical procedural skills. We have designated it the Leicester Clinical Procedure Assessment Tool (LCAT).     

Peer-assisted learning: a novel approach to clinical skills learning for medical students

OBJECTIVE: This study aimed to determine whether peer-assisted learning (PAL) can enhance clinical examination skills training. METHODS: Three student trainers studied small-group theory and clinical examination and provided PAL as extra tuition for 86 trainees. Trainees watched an examination video, were videotaped practising the examination and, after constructive feedback, repeated the examination. Responses to PAL were evaluated to attain an overview of trainee and trainer performance using visual analogue and Likert scale analyses. Year-group review was undertaken using questionnaires. RESULTS: Trainees evaluated all aspects of PAL highly, including their post-training confidence in examination skills (mean > 7.7 on a 10-cm scale), indicating that the PAL was effective. Written comments confirmed the students perceived the sessions as well structured and of high quality. Compared with trainees in the first groups, those from later groups gave all parameters similar or higher gradings. Those for interest (P = 0.03) and appropriateness (P = 0.01) were significantly higher, suggesting that trainers may improve their technique with time. Students with previous degrees gave similar or lower gradings than standard entry students, with answers about post-training confidence and recommendation to friends being statistically lower (P < 0.006). Six months later, year-group analysis showed that 90% of trainees rated PAL highly, and 86% wished to become trainers. Of the trainers' year group, 79% perceived that PAL training could improve examination skills. CONCLUSIONS: In the context of clinical skills training, PAL was highly evaluated across many parameters, including confidence after training. Student interest and enthusiasm supports suggestions that PAL could be a useful adjunct to clinical skills training.     

A conceptual framework for developing teaching cases: a review and synthesis of the literature across disciplines

Case-based teaching is regarded as a superior instructional method compared with lectures in promoting a learner's critical thinking skills. While much is known about the role a discussion facilitator plays in case-based teaching, the debate on the influence of the format and structure of cases on learning is controversial. We sought to identify strategies for constructing cases based on studies from multiple disciplines, which report the development and use of cases in teaching and learning. The purpose was to offer the medical and other educational communities a conceptual framework that can be examined in future research. Based on a review of 100 studies, we synthesised 17 strategies around 5 core attributes of cases: relevant (level of learner, goals and objectives, setting of case narrative); realistic (authenticity, distractors, gradual disclosure of content); engaging (rich content, multiple perspectives, branching of content); challenging (difficulty, unusual cases, case structure, multiple cases), and instructional (build upon prior knowledge, assessment, feedback, and teaching aids). Despite the wide use of cases in disparate disciplines, there has been no overarching study that synthesises strategies of case development or tests these strategies in research settings. The framework we developed can serve as a menu of case development options that educators and researchers can pilot and evaluate in their local settings.     

Feedback by simulated patients in undergraduate medical education: a systematic review of the literature

Objectives  Although the importance of feedback by simulated patients (SPs) is generally recognised, knowledge is scarce about the most effective ways in which SPs can provide feedback. In addition, little is known about how SPs are trained to provide feedback. This study aimed to provide a systematic overview of the ways in which SPs provide feedback to undergraduate medical students, the domains in which SPs provide feedback and the ways in which SPs are trained to provide feedback.
Methods  We performed a systematic search of the literature using PubMed, PsychINFO and ERIC and searched for additional papers cited in reference lists. Papers were selected on the basis of pre-established inclusion and exclusion criteria and were classified, using a pre-established form, according to three aspects of SP feedback: training in giving feedback; the process of delivering feedback, and the domain(s) in which feedback is given.
Results  A total of 49 studies were included and described in detail on the basis of the three aspects of SP feedback described above. The ways in which SPs were trained to give feedback were largely heterogeneous, as were the processes by which feedback was provided by SPs. Only a few studies described feedback processes that were in accordance with general recommendations for the delivery of effective feedback. Although feedback from the patient's perspective is generally recommended, most SPs provided feedback on clinical skills and communication skills.
Discussion  There appear to be no clear standards with regard to effective feedback training for SPs. Furthermore, the processes by which feedback is provided by SPs and the selection of domain(s) in which SPs give feedback often seem to lack a solid scientific basis. Suggestions for further research are provided.     

Anatomical dissection as a teaching method in medical school: a review of the evidence

CONTEXT: Discussions about dissection as a teaching method in gross anatomy are characterised by a lack of objective evidence. METHODS: A search for such evidence in the literature produced 14 relevant papers. These were reviewed for objective data on the effect of cadaver dissection on cognitive learning outcomes. RESULTS: All reviewed studies compared groups of students exposed to different teaching approaches, including active dissection, learning on prosected material, or a combination with computerised teaching aids. Study and course designs varied substantially and student groups compared were not always homogeneous. In all studies, compared learning experiences differed in more than 1 variable, and assessment of anatomical knowledge was not standardised. DISCUSSION: It is difficult to interpret and generalise from the results of the reviewed studies. Considering the bias that must be assumed for teachers who develop new course designs and compare these with traditional ones, the review shows a slight advantage for traditional dissection over prosection. CONCLUSIONS: More sophisticated research designs may be necessary to solve the general problem of the small measurable impact of educational interventions and to come to scientifically sound conclusions about the best way to teach gross anatomy. Such research will have to include sufficient sample sizes, the use of validated assessment instruments, and a discussion of the educational significance of measured differences. More educational research in anatomy is necessary to counterbalance emotional arguments about dissection with scientific evidence. Anatomical knowledge is too important to future doctors to leave its teaching to the educational fashion of the day.     

Strategies and interventions for the involvement of real patients in medical education: a systematic review

Objectives  There is increasing emphasis on encouraging more active involvement of patients in medical education. This is based on the recognition of patients as 'experts' in their own medical conditions and may help to enhance student experiences of real-world medicine. This systematic review provides a summary of evidence for the role and effectiveness of real patient involvement in medical education.
Methods  MEDLINE, EMBASE, ERIC, PsychINFO, Sociological Abstracts and CINAHL were searched from the start of the databases to July 2007. Three key journals and reference lists of existing reviews were also searched. Articles published in English and reporting primary empirical research on the involvement of real patients in medical education were included. The synthesis of findings is integrated by narrative structured in such a way to address the research questions.
Results  A total of 47 articles were included in the review. The majority of studies reported patients in the role of teachers only; others described patient involvement in assessment or curriculum development or in combined roles. Patient involvement was recommended in order to bring the patient voice into education. There were several examples of how to recruit and train patients to perform an educational role. The effectiveness of patient involvement was measured by evaluation studies and reported improvements in skills.
Conclusions  There was limited evidence of the long-term effectiveness of patient involvement and issues of ethics, psychological impact and influence on education policy were poorly explored. Future studies should address these issues and should explore the practicalities of sustaining such educational programmes within medical schools.     

Using borderline methods to compare passing standards for OSCEs at graduation across three medical schools

CONTEXT: Medical schools in the UK set their own graduating examinations and pass marks. In a previous study we examined the equivalence of passing standards using the Angoff standard-setting method. To address the limitation this imposed on that work, we undertook further research using a standard-setting method specifically designed for objective structured clinical examinations (OSCEs). METHODS: Six OSCE stations were incorporated into the graduating examinations of 3 of the medical schools that took part in the previous study. The borderline group method (BGM) or borderline regression method (BRM) was used to derive the pass marks for all stations in the OSCE. We compared passing standards at the 3 schools. We also compared the results within the schools with their previously generated Angoff pass marks. RESULTS: The pass marks derived using the BGM or BRM were consistent across 2 of the 3 schools, whereas the third school generated pass marks which were (with a single exception) much lower. Within-school comparisons of pass marks revealed that in 2 schools the pass marks generally did not significantly differ using either method, but for 1 school the Angoff mark was consistently and significantly lower than the BRM. DISCUSSION: The pass marks set using the BGM or BRM were more consistent across 2 of the 3 medical schools than pass marks set using the Angoff method. However, 1 medical school set significantly different pass marks from the other 2 schools. Although this study is small, we conclude that passing standards at different medical schools cannot be guaranteed to be equivalent.     

Teaching clinical skills to new medical students: the Oman experience

The College of Medicine at the Sultan Qaboos University (SQU) in Oman took up the challenge of moving away from a didactic emphasis in the teaching of family and community health by actively involving students in health care, requiring some clinical skills, as early as possible. This paper describes the experience of the Department of Family and Community Health at SQU of the feasibility of training first-year medical students in the measurement of blood pressure within a few days. Our experience showed that proper training before starting field-work can teach clinical skills to students who have had no prior exposure to medical subjects. There was a strong correlation between the measurements of blood pressure of 638 individuals by the students and the supervisors. This experience at SQU has encouraged us to implement the idea of family- and community-based clinical exposure for every class. Teaching of clinical skills is feasible in the field for new entrants, provided there is adequate orientation beforehand and skilled supervision of the students in small groups.     

An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation

PURPOSE: This investigation aimed to explore the measurement properties of scores from a patient simulator exercise. METHODS: Analytic and holistic scores were obtained for groups of medical students and residents. Item analysis techniques were used to explore the nature of specific examinee actions. Interrater reliability was calculated. Scores were contrasted for third year medical students, fourth year medical students and emergency department residents. RESULTS: Interrater reliabilities for analytic and holistic scores were 0.92 and 0.81, respectively. Based on item analysis, proper timing and sequencing of actions discriminated between low- and high-ability examinees. In general, examinees with more advanced training obtained higher scores on the simulation exercise. CONCLUSION: Reliable and valid measures of clinical performance can be obtained from a trauma simulation provided that care is taken in the development and scoring of the scenario.     

Transition to clinical training: influence of pre-clinical knowledge and skills, and consequences for clinical performance

Context  Many students experience a tough transition from pre-clinical to clinical training and previous studies suggest that this may constrict students' progress. However, clear empirical evidence of this is lacking. The aim of this study was to determine: whether the perceived difficulty of transition influences student performance during the first 2 weeks of clerkships; whether it influences students' overall performance in their first clerkship, and the degree to which the difficulty of transition is influenced by students' pre-clinical knowledge and skills levels.
Methods  Clerks ( n  = 83) from a university hospital and eight affiliated hospitals completed a questionnaire measuring the perceived difficulty of the transition period. Data collected included student scores on pre-clinical knowledge and skills, their performance during the second week of the first clerkship, and their overall performance in the first clerkship. Univariate and multivariate multiple regression analyses were used to analyse the data.
Results  The perceived difficulty of transition was neither predictive of student performance during the transition period (adjusted R ² = 11.8%, P  = NS), nor of their overall clerkship performance (adjusted R ² = 8.6%, P  = NS). Students' pre-clinical knowledge and skills played a minor role in the perceived difficulty of the transition period.
Conclusions  The negative effect of the transition period on student progress suggested in the literature was not found in this study. A possible explanation for the limited influence of students' knowledge and skills on performance during the transition period is that the workload in this period causes a cognitive overload, interfering with students' abilities to apply their pre-clinical knowledge and skills.     

Evaluation of procedural skills training in an undergraduate curriculum

CONTEXT: A substantial proportion of medical students enter their intern year without any basic skills experience. Lack of experience is a significant source of stress for many junior doctors. OBJECTIVES: To evaluate the effect of a basic procedural skills tutorial for Year 3 medical students on their competence in relevant skills at Year 5. SUBJECTS: The control group consisted of 93 medical students who completed Year 3 in 1996. The intervention group consisted of 92 medical students who completed Year 3 in 1997. The intervention group received a practical skills tutorial in Year 3; the control group did not. Both groups were assessed on their practical skills competence during Year 5. METHODS: A 3-hour practical tutorial on injection and suturing techniques was delivered to the intervention group. The effectiveness of the intervention was assessed by self-reported experience of giving injections, inserting sutures and sustaining needlestick injuries, and by teacher-rated competency in four basic procedural skills. RESULTS: Students who received the Year 3 tutorial were significantly more likely to record a satisfactory assessment for their performance in all four basic skills compared with students who did not receive the tutorial. They were less likely than controls to refuse invitations to give injections, but not invitations to insert a suture, during Years 4 and 5. CONCLUSIONS: A single session of formalised teaching in procedural skills in the early stages of a medical degree can have long-term effectiveness in basic skills competence and may increase students' confidence to practise their skills.     

Growth of analytical thinking skills over time as measured with the MATCH test

Context Ber’s Comprehensive Integrative Puzzle aims to assess analytical clinical thinking in medical students. We developed a paediatric version, the MATCH test, in which we added two irrelevant options to each question in order to reduce guessing behaviour. We tested its construct validity and studied the development of integrative skills over time. Methods We administered a test (MATCH 1) to subjects from two universities, both with a 6‐year medical training course. Subjects included 30 students from university 1 who had completed a paediatric clerkship in Year 4, 23 students from university 2 who had completed a paediatric clerkship in Year 5, 13 students from both universities who had completed an advanced paediatric clerkship in Year 6, 28 paediatric residents and 17 paediatricians. We repeated this procedure using a second test with different domains in a new, comparable group of subjects (MATCH 2). Results Mean MATCH 1 scores for the respective groups were: Year 4 students: 61.2% (standard deviation [SD] 1.3); Year 5 students: 71.3% (SD 1.6); Year 6 students: 76.2% (SD 1.5); paediatric residents: 88.5% (SD 0.7), and paediatricians: 92.2% (SD 1.1) (one‐way anova F = 104.00, P < 0.0001). Students of both universities had comparable scores. MATCH 1 and 2 scores were comparable. Cronbach’s α‐values in MATCH 1 and 2 were 0.92 and 0.91, respectively, for all subjects, and 0.82 and 0.87, respectively, for all students. Conclusions Analytical clinical thinking develops over time, independently of the factual content of the course. This implies that shortened medical training programmes could produce less skilled graduates.     

Trialling a new way to learn clinical skills: systematic clinical appraisal and learning

AIM: To describe and evaluate the effectiveness of a new method of teaching clinical skills designed to increase students' active and self-directed learning as well as tutor feedback. METHODS: A total of 22 consenting Year 4 medical students undertaking general practice and general surgery clinical experience were involved in a pre- and post-test research design. In the initial period of the study, students were taught clinical skills in a traditional manner. In the second phase a clinical teaching strategy called systematic clinical appraisal and learning (SCAL) was utilised. This learning strategy involved active and self-directed learning, holistic care and immediate feedback. Students independently saw a patient and were asked to make judgements about the patient's potential diagnosis, tests required, management, psychosocial needs, preventive health requirements, and any ethical problems. These judgements were then compared with those of the clinical supervisor, who saw the same patient independently. Students recorded details for each consultation. Comparisons were made of the two study periods to examine whether the use of SCAL increased the number of students' independent judgements, perceived student learning, tutor feedback and self-directed learning. RESULTS: During the SCAL learning period, students reported making a greater number of statistically significant independent judgements, and receiving significantly increased tutor feedback in both general practice and general surgery. The number of learning goals set by students was not found to differ between the two periods in surgery but significantly increased in general practice in the SCAL period. Students' perceptions of their learning significantly increased in the SCAL period in surgery but not in general practice. During the traditional learning period in both settings, there was limited student decision-making about most aspects of care, but particularly those relating to prevention, psychosocial issues and ethics. CONCLUSIONS: The SCAL approach appears to offer some advantages over traditional clinical skills teaching. It appears to encourage active and independent decision-making, and to increase tutor feedback. Further exploration of the approach appears warranted.