An analysis of standardised patient checklist errors and their effect on student scores

BACKGROUND: The frequency and nature of standardised patient (SP) recording errors during clinical performance examinations (CPX) have an effect on case scores and ultimately on pass/fail decisions. PURPOSE: To determine the effect of SP recording errors on case scores. METHODS: Standardised patients completed checklists immediately after each encounter. To determine checklist accuracy, multiple reviewers developed a checklist key for each student encounter studied. The total errors, the net errors, the errors of commission and omission and error rates by competency skill were analysed. RESULTS: The frequency of errors in history taking was greater than in physical examination, and the majority of errors were made in the students' favour. Summing the errors of commission and omission decreased the effect of total errors on student scores. CONCLUSIONS: High levels of SP recording accuracy are achievable. When errors occur, the net effect is usually in the students' favour.     

Validation of a checklist to assess ward round performance in internal medicine

BACKGROUND: Ward rounds are an essential responsibility for doctors in hospital settings. Tools for guiding and assessing trainees' performance of ward rounds are needed. A checklist was developed for that purpose for use with trainees in internal medicine. OBJECTIVE: To assess the content and construct validity of the task-specific checklist. METHODS: To determine content validity, a questionnaire was mailed to 295 internists. They were requested to give their opinion on the relevance of each item included on the checklist and to indicate the comprehensiveness of the checklist. To determine construct validity, an observer assessed 4 groups of doctors during performance of a complete ward round (n = 32). The nurse who accompanied the doctor on rounds made a global assessment of the performance. RESULTS: The response rate to the questionnaire was 80.7%. The respondents found that all 10 items on the checklist were relevant to ward round performance and that the item collection was comprehensive. Checklist mean-item scores differed between levels of expertise: junior house officers 1.4 (1.0-1.9); senior house officers 2.0 (1.5-2.9); specialist trainees 2.5 (1.8-2.8), and specialists 2.7 (2.3-3.5); median (range) (P < 0.001). A significant correlation was found between global observer scores and nurse scores (r = 0.56, P < 0.001). CONCLUSION: The checklist, developed for assessing trainees' performance of ward rounds in internal medicine, showed high content validity. Construct validity was supported by the higher scores of experienced doctors compared to those with less experience and the significant correlation between the observer's and nurses' global scores. The developed checklist should be valuable in guiding and assessing trainees on ward round performance.     

Evaluating surgeons' informed decision making skills: pilot test using a videoconferenced standardised patient

BACKGROUND: Standardised patients (SPs) are effective in evaluating communication skills, but not every training site may have the resources to develop and maintain SP programmes. OBJECTIVES: To test whether videoconferencing technology (VT) could enable an interaction between an SP and an orthopaedic surgeon that would allow the SP to accurately evaluate the surgeon's informed decision making (IDM) skills. We also assessed whether this sort of interaction was acceptable to orthopaedic surgeons as a means of learning IDM skills. METHODS: We trained an SP to represent a 75-year-old woman considering hip replacement surgery. Orthopaedic surgeons in Chicago individually consulted with the SP in Philadelphia; each participant could see and hear the other on large television screens. The SP evaluated the surgeons' advice using a 23-item checklist of IDM elements, and gave each surgeon verbal and written feedback on his IDM skills. The surgeons then gave their evaluations of the exercise. RESULTS: Twenty-two surgeons completed the project. The SP was > or = 80% accurate in classifying 20 of the 23 IDM skills when compared to a clinician rater. Although 12 (55%) of the orthopaedic surgeons felt that some aspects of the technology were distracting, most were pleased with it, and 19 of 22 (86%) would recommend the videoconferenced SP interaction to their colleagues as a means of learning IDM skills. CONCLUSIONS: These results suggest that VT allows accurate evaluation of IDM skills in a format that is acceptable to orthopaedic surgeons. Videoconferencing technology may be useful in long-distance SP communication assessment for a variety of learners.     

Assessing performance in final year medical students. Can a postgraduate measure be used in an undergraduate setting?

BACKGROUND: The transition between medical school and graduate performance should be a continuum. This study aimed to evaluate an assessment tool developed for practising doctors when applied to undergraduates. METHODS: A 12-item rating form was developed from that used for practising doctors by the Royal Australasian College of Physicians. Over a 2-year period, senior doctors, junior doctors and nurses completed the rating form on final year medical students. Some students completed self-assessments. We performed factor analysis and correlated scores between raters and attachments. Correlating ratings with concurrent traditional assessment results across the year tested construct validity. RESULTS: Ten forms per student were distributed for all 123 students and 856 were returned (70%). Internal consistency was very high. In all, 71.1% of the variance was accounted for by two factors (clinical skills and humanistic). This factor structure is unchanged when restricted to different raters and is the same as that noted previously when rating practising doctors. There were good correlations between raters (including self) and between attachments. Nurse ratings were reliable but nurses rated students significantly lower on humanistic qualities. Correlations with traditional assessments were high when all traditional assessments were combined. Women scored more favourably than men on humanistic qualities. CONCLUSION: A rating instrument for doctors in practice retains the same factor structure and a high degree of reliability and validity for senior medical students. Reliable ratings by nurses have implications for measures of collegiality and teamwork. We believe the instrument could be a useful outcome measure for medical programmes and employers.     

Using standardised students in faculty development workshops to improve clinical teaching skills

PURPOSE: We describe the use of standardised students (SSs) in interdisciplinary faculty development programmes to improve clinical teaching skills. Standardised students are actual health professions students who are trained to portray a prototypical teaching challenge consistently across many encounters with different faculty participants. METHODS: The faculty development programmes described focused on the skills of providing feedback and brief clinical teaching. At the beginning of each session, each participant was videotaped in encounters with 2 different SSs. Using microteaching (an instructional method in which learners view short segments of their own videotaped performance and discuss the tapes with a facilitator, consultant or other workshop participants), each group of participants and instructors reviewed the tapes and reflected on the encounters, providing immediate feedback to participants and modelling different approaches to the same teaching problem. The same process was repeated with more complicated scenarios after 2 weeks and again after 6 months offering reinforcement, further practice and more sophisticated development of the strategies learned. Participants completed post-session evaluations and a follow-up telephone survey. RESULTS: A total of 36 faculty members from the colleges of medicine, dentistry, pharmacy and nursing participated in workshops in 2000-01. The workshops were rated as highly relevant to participants' teaching, and most participants reported that they had learned a great deal. Participants most appreciated reviewing the videotaped interactions, the feedback they received, the interactions with their colleagues, the interdisciplinary nature of the groups and the practical focus of the workshops. CONCLUSIONS: Standardised students provide a high fidelity, low risk, simulated environment in which faculty can reflect on and experiment with new teaching behaviours. Such encounters can enhance the effectiveness and impact of faculty development programmes to improve clinical teaching skills.     

Assessing cardiac physical examination skills using simulation technology and real patients: a comparison study

Objective High‐stakes assessments of doctors’ physical examination skills often employ standardised patients (SPs) who lack physical abnormalities. Simulation technology provides additional opportunities to assess these skills by mimicking physical abnormalities. The current study examined the relationship between internists’ cardiac physical examination competence as assessed with simulation technology compared with that assessed with real patients (RPs). Methods The cardiac physical examination skills and bedside diagnostic accuracy of 28 internists were assessed during an objective structured clinical examination (OSCE). The OSCE included 3 modalities of cardiac patients: RPs with cardiac abnormalities; SPs combined with computer‐based, audio‐video simulations of auscultatory abnormalities, and a cardiac patient simulator (CPS) manikin. Four cardiac diagnoses and their associated cardiac findings were matched across modalities. At each station, 2 examiners independently rated a participant’s physical examination technique and global clinical competence. Two investigators separately scored diagnostic accuracy. Results Inter‐rater reliability between examiners for global ratings (GRs) ranged from 0.75–0.78 for the different modalities. Although there was no significant difference between participants’ mean GRs for each modality, the correlations between participants’ performances on each modality were low to modest: RP versus SP, r = 0.19; RP versus CPS, r = 0.22; SP versus CPS, r = 0.57 (P < 0.01). Conclusions Methodological limitations included variability between modalities in the components contributing to examiners’ GRs, a paucity of objective outcome measures and restricted case sampling. No modality provided a clear ‘gold standard’ for the assessment of cardiac physical examination competence. These limitations need to be addressed before determining the optimal patient modality for high‐stakes assessment purposes.     

Clinical skills in junior medical officers: a comparison of self-reported confidence and observed competence

BACKGROUND: The intern year is a key time for the acquisition of clinical skills, both procedural and cognitive. We have previously described self-reported confidence and experience for a number of clinical skills, finding high levels of confidence among Australian junior doctors. This has never been correlated with an objective measure of competence. AIMS AND HYPOTHESIS: We aimed to determine the relationship between self-reported confidence and observed competence for a number of routine, procedural clinical skills. METHODS: A group of 30 junior medical officers in their first postgraduate year (PGY1) was studied. All subjects completed a questionnaire concerning their confidence and experience in the performance of clinical skills. A competency-based assessment instrument concerning 7 common, practical, clinical skills was developed, piloted and refined. All 30 PGY1s then completed an assessment using this instrument. Comparisons were then made between the PGY1s' self-reported levels of confidence and tutors' assessments of their competence. RESULTS: A broad range of competence levels was revealed by the clinical skills assessments. There was no correlation between the PGY1s' self-ratings of confidence and their measured competencies. CONCLUSIONS: Junior medical officers in PGY1 demonstrate a broad range of competence levels for several common, practical, clinical skills, with some performing at an inadequate level. There is no relationship between their self-reported level of confidence and their formally assessed performance. This observation raises important caveats about the use of self-assessment in this group.     

Degree of concurrency among experts in data collection and diagnostic hypothesis generation during clinical encounters

BACKGROUND: Given that there are variations in clinicians' reasoning, methods to elaborate scoring checklists for standardised patient-based assessment need to be valid. The use of data elicited by experts solving problems independently has been advocated as a method of setting performance standards. AIMS: To determine the degree of concurrence and common characteristics among items independently elicited by doctors during patient encounters and to assess the number of experts needed to derive reliable performance standards. METHODS: Six experienced internists worked-up the same 7 chief complaints with standardised patients (SPs). A stimulated recall of the recorded encounter was then performed. The degree of concurrence of the collected history and physical examination information and the generated diagnostic hypotheses was computed. Reliability was derived from generalisability analyses. RESULTS: By case, experts elicited a mean of 114 information items (SD = 15) and generated 30 diagnostic hypotheses (SD = 6). A high concurrence (80-100%) was observed for a mean of 22 information items (20%; SD = 6) and 7 diagnostic hypotheses (24%; SD = 2). More than a third of the 153 highly concurrent information items were clarification questions. At least 3 doctors were needed to obtain a reliability of 0.80 or higher when deriving the scoring checklists. CONCLUSION: The limited concurrency in data elicited by clinicians during a patient encounter supports the use of high-fidelity methods to develop performance checklists used in SP-based assessment. It also suggests that relying only on information collected to assess clinical competence may not be sufficient. Additional criteria, such as structure and style of work-up, should be further explored.     

Further challenges in measuring communication skills: accounting for actor effects in standardised patient assessments

Context  Subjective rating scales for communication skills may yield more personally meaningful responses than more standardised rating schemes. It is unclear, however, whether such evaluations may be overly biased by respondents' rating styles, which may lead to unreliable measurement of examinees' communication skills.
Methods  Our study involved 212 students from the classes of 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All students were rated by actors depicting standardised patients (SPs) on the same seven cases using the 19-item Rochester Communication Rating Scale (RCRS). Different students were assigned to different actors playing the same SP. We assessed the extent to which actors' personal rating styles influenced the scores they assigned to students. Main outcome measures were: between-actor variability in responses; the degree to which actors' response styles contribute to overall scores, and improvements in reliability achieved by standardising actors' ratings.
Results  There were statistically significant differences between actors in their mean assigned scores. Scores aggregated over 18 separate SP cases have an expected generalisability coefficient of 0.79. If raw RCRS scores are used, a total of 27 replications of the RCRS are required to achieve a Cronbach's alpha of 0.8; standardisation reduces this number to 18.
Conclusions  Although actors are variable in their use of a standardised subjective scale of communication, such differences contribute to an acceptably small proportion of the total variance if scores are combined across a large number of cases. Reliability can be markedly improved by standardising scores across raters.     

Evaluation of procedural skills training in an undergraduate curriculum

CONTEXT: A substantial proportion of medical students enter their intern year without any basic skills experience. Lack of experience is a significant source of stress for many junior doctors. OBJECTIVES: To evaluate the effect of a basic procedural skills tutorial for Year 3 medical students on their competence in relevant skills at Year 5. SUBJECTS: The control group consisted of 93 medical students who completed Year 3 in 1996. The intervention group consisted of 92 medical students who completed Year 3 in 1997. The intervention group received a practical skills tutorial in Year 3; the control group did not. Both groups were assessed on their practical skills competence during Year 5. METHODS: A 3-hour practical tutorial on injection and suturing techniques was delivered to the intervention group. The effectiveness of the intervention was assessed by self-reported experience of giving injections, inserting sutures and sustaining needlestick injuries, and by teacher-rated competency in four basic procedural skills. RESULTS: Students who received the Year 3 tutorial were significantly more likely to record a satisfactory assessment for their performance in all four basic skills compared with students who did not receive the tutorial. They were less likely than controls to refuse invitations to give injections, but not invitations to insert a suture, during Years 4 and 5. CONCLUSIONS: A single session of formalised teaching in procedural skills in the early stages of a medical degree can have long-term effectiveness in basic skills competence and may increase students' confidence to practise their skills.     

The interpersonal, cognitive and efficiency domains of clinical teaching: construct validity of a multi-dimensional scale

BACKGROUND: We are unaware of any hypothesis-driven studies showing that teaching assessments are comprised solely of interpersonal and cognitive domains. Moreover, previous teaching assessments have been biased by heterogeneous samples of evaluators. Consequently, we investigated the construct validity of faculty assessments comprised of interpersonal and cognitive domains, utilising evaluations obtained from resident doctors on an internal medicine hospital service. METHODS: A total of 1000 inpatient evaluations were completed on 60 general internal medicine faculty members. Education theory supported a 2-dimensional, 14-item scale. Principal factor analysis was used to explore the scale's dimensionality. Internal reliability and interobserver agreement were determined. Relationships between domains and instructor characteristics were also examined. RESULTS: Principal factor analysis revealed interpersonal, clinical teaching and efficiency domains. Internal reliabilities of all domains are high (alpha > 0.90). Interobserver agreement is good (range 0.64-0.83). In the interpersonal domain there is a trend towards higher scores for lower ranking faculty. Significant findings are higher overall scores in the interpersonal domain (P < 0.001), higher scores for assistant professors in the interpersonal domain (P = 0.008) and higher scores for male than female faculty in the interpersonal (P = 0.041) and clinical teaching (P = 0.008) domains. CONCLUSIONS: Clinical teaching evaluations are reducible to interpersonal, clinical teaching and efficiency domains. Evidence for construct validity includes predicted domains and high internal and interobserver reliabilities. Utilising a homogenous sample of evaluators minimised variance. Interestingly, lower ranking faculty scored higher in the interpersonal domain, suggesting that lower ranking faculty may focus more attention on teaching activities than full professors do.     

Martin's Map: a conceptual framework for teaching and learning the medical interview using a patient-centred approach

OBJECTIVE: At the end of training, students seem to lack a basic understanding of how to take an organised, relevant medical and social history using a patient-centred approach. The aim of developing the map described in this paper was to provide a framework for such an approach. METHODS: Action research was used to continuously modify and refine an interview map that was used by medical clerks, family medicine residents, international medical graduates and practising doctors for teaching and learning purposes over a 10-year period. CONCLUSION: 'Martin's Map' provides a realistic framework for flexibly organising and integrating medical content with process that did not previously exist. The map provides medical educators with a standardised framework for talking about the medical interview, which helps learners understand how to use their medical knowledge with a patient-centred approach. Learners are able to visually see how they can take a focused medical and social history using a patient-centred approach, which subsequently seems to help them organise their thinking and approach during the medical encounter.     

The effect of an extended hospital induction on perceived confidence and assessed clinical skills of newly qualified pre-registration house officers

BACKGROUND: Recent studies raise concerns over the preparedness of newly qualified doctors for the role of the pre-registration house officer (PRHO). This study aimed to assess self-perception of preparedness, objective assessment of core clinical skills and the effect of an extended clinical induction programme prior to commencing full duties. METHODS: A group of 26 newly qualified doctors from 1 district general hospital underwent an extended 5-day, ward-based induction programme. The participants completed questionnaires on their own perceptions of their preparedness for PRHO duties and underwent an objective structured clinical examination (OSCE) of 4 core clinical skills prior to induction, on completion of induction and 1 month into working life. RESULTS: At the outset PRHOs had low perceptions of their own capabilities in all clinical scenarios and skills. Most perceptions improved after induction, although in 2 clinical areas they felt even less confident. One month into post there were significant improvements in all areas. Only 1 PRHO passed all 4 clinical skills assessments at the pre-induction assessment. Seven (26%) failed on 1 or more skills at the post-induction assessment. However, all participants were deemed competent in all skills at the 1-month assessment. CONCLUSION: Newly qualified doctors do not feel prepared for PRHO duties and objectively are not competent in basic clinical skills. An extended induction improves preparedness in some but not all clinical areas and improves performance of objectively assessed clinical skills.     

Approaches to medical student remediation after a comprehensive clinical skills examination

OBJECTIVE: Most US medical schools conduct comprehensive clinical skills assessments during Years 3 and 4. This study explores strategies used to identify and remediate students who perform poorly on these assessments. METHODS: In the academic year 2005-06, we conducted 33 semi-structured interviews with individuals responsible for standard setting in and remediation after their schools' comprehensive clinical skills assessments. We coded interviews to identify major themes. RESULTS: Prior to remediation, some schools employed a 'verification' step to ensure the accuracy of the failing score or need for remediation. Participants described a remediation process that included some or all of 3 steps. Firstly, students' specific learning deficits were diagnosed. Next, students participated in remedial activities such as performance review sessions or practice with standardised or actual patients. Lastly, students were re-tested, usually with a shorter, more formative examination. All participants reported using a diagnostic step, most offered or required remedial activities and many re-tested, although schools varied in the emphasis placed on each step. Many participants cited the individualised attention students received from remediation faculty staff as a strength of their approach, although they raised concerns about the substantial time demands placed on remediation faculty. Most respondents reported some dissatisfaction with their school's remediation process, particularly uncertainty about efficacy or rigour. CONCLUSIONS: Schools vary in the intensity and scope of remediation offered to students who perform poorly on clinical skills assessments. Although many schools invest significant resources in remediation, the effect of these efforts on students' subsequent clinical performance is unknown.     

Avoidable breaches of confidentiality: a study among students of medicine and of law

OBJECTIVES: Avoidable breaches of patient confidentiality due to inadvertence or ignorance occur frequently. The aim of this study was to explore whether participants were able to identify violations of confidentiality and how serious these violations were. DESIGN: After a 2-hour theoretical education session, participants filled out a standardised questionnaire presenting 6 hypothetical cases (e.g. politician's illness mentioned to the doctor's family, violence inflicted by the police mentioned to a lawyer at a dinner party, both without patient consent) and asking whether confidentiality had been violated. Answers were given on a score of 0-3, where 0 = no violation and 3=serious violation. SETTING: University of Geneva, Switzerland. PARTICIPANTS: Three professors of criminal law (representing the gold standard) and convenience samples of 311 law, medical and dental students took part. Main outcome measures Students' answers were compared to the gold standard (answers of criminal law professors). RESULTS: According to our gold standard, a violation of confidentiality took place in all 6 cases (scores: 3, 2, 3, 2, 3, 3; means of students: 1.7, 0.6, 1.0, 0.4, 1.5, 2.6). Most students (88%) mistakenly believed in 1-3 cases that no violation took place. In only 2 cases did more than 90% of participants recognise the violation. Law students attributed significantly lower scores than medical or dental students in 3 cases and higher scores in 1 case. CONCLUSIONS: Despite theoretical education, participants did not fully understand obligations towards patient confidentiality when it came to practical situations, especially when colleagues and authorities (police, those in a judicial context) asked for information. Teaching should address the risk of patient identification and the seriousness of violations in order to motivate doctors to avoid unjustified violations of confidentiality.     

Impact of a new curriculum on medical students' interpersonal and interviewing skills

OBJECTIVES: In 1998 we reported on the rise and fall of medical student communication skills during the 4 years of medical school. Since then, the University of Connecticut School of Medicine has completed a major curriculum renewal project with an emphasis on early clinical work, lifelong learning and more ambulatory training. The goals of this study were to compare students' interviewing and interpersonal skills in standardised patient (SP) assessments in the old and new curricula and to assess the success of the new curriculum in preventing a decline in student skills in this domain. METHODS: The clinical skills of 202 students were measured longitudinally during encounters with SPs in each of their 4 years of medical school. Students in this study and the earlier study were evaluated using the Arizona Clinical Interviewing Rating (ACIR) Scale. RESULTS: Compared with students from the previous curriculum, students on the new curriculum in this study showed an improvement in ACIR scores. Year 1 mean ACIR scores (1 = poor to 5 = excellent) were, respectively, 3.6 for the old curriculum cohort and 4.0 for the new curriculum group. In Year 4 the mean score for the old curriculum cohort was 3.7 and that for the new curriculum group was 3.8. Students on the new curriculum still showed a decline in ACIR scores from Years 1 to 4, but it was not as severe a decline as it had been previously. CONCLUSIONS: Pre-clinical medical students perform better on measures of interpersonal communication than their clinical counterparts. The students who participated in the new curriculum demonstrated an earlier acquisition of and a less steep decline in interviewing and interpersonal skills during the course of medical school.     

The effect of candidate familiarity on examiner OSCE scores

CONTEXT: Although examiners are a large source of variability in the objective structured clinical examination (OSCE), the exact causes of examiner variance remain understudied. OBJECTIVE: This study aimed to determine whether examiner familiarity with candidates influences candidate scores. METHODS: A total of 24 candidates from 4 neonatal-perinatal training programmes participated in a 10-station OSCE. Sixteen trainees and 7 examiners came from a single centre (site A) and 8 candidates and 5 examiners came from the other 3 centres. Examiners completed station-specific binary checklists and an overall global rating; standardised patients (SPs) and standardised health professionals (SHPs) completed 4 process ratings and the overall rating. A fixed-effect, 2-way analysis of variance was performed to ascertain whether there was interaction between examiner site and candidate site. RESULTS: Interstation Cronbach's alpha was 0.80 for the examiner checklist, 0.88 for the examiner global rating and 0.88 for the SP or SHP global rating. Although the checklist scores awarded by site A examiners were significantly higher than those awarded by non-site A examiners, there was no significant interaction between examiner and candidate site (P = 0.124). Similarly, the interaction between examiner and candidate site for the global rating was not significant (P = 0.207). Global ratings awarded by SPs and SHPs were also higher in stations where site A faculty examined site A candidates, suggesting the observed differences may have been related to performance. CONCLUSIONS: Results from this small dataset suggest that examiner familiarity with candidates does not influence how examiners score candidates, confirming the objective nature of the OSCE. Confirmation with a larger study is required.     

Observed versus reported precepting skills: teaching behaviours in a community ambulatory clerkship

PURPOSE: Community-based doctors are increasingly utilised for the clinical education of medical students. Faculty development programmes are frequently provided. However, data about the community faculty-student teaching interaction and the longterm impact of such programmes are limited. This study observes community faculty-student teaching interactions and assesses the use of clinical teaching methods taught during faculty development workshops. METHODS: Between March and August 2002, 13 rural, urban and suburban community-based faculty who completed at least 1 faculty development workshop were observed during faculty-student-patient encounters and interviewed. Observer-interviewers utilised a checklist to record teaching styles, methods and skills, and administered a questionnaire to obtain community faculty's self-perceptions. RESULTS: On observation, the assertive precepting style and didactic teaching predominated in 90% and 86% of teaching encounters, respectively. Interactive discussion and direct observation occurred less often, in 71% and 18% of encounters, respectively. Feedback was provided in 25% of encounters. On interview, community faculty reported that faculty development workshops resulted in improved precepting effectiveness and professional satisfaction with moderate to considerable improvement in feedback, interactive teaching and use of multiple precepting styles for 100%, 77% and 77% of community faculty, respectively. Community faculty reported that their skills in addressing problem students, direct observation and didactic teaching improved moderately. There was little statistical correlation between self-assessed and observed skills. CONCLUSIONS: Community-based faculty rate their teaching skills as improved following faculty development. Observation reveals that interactive discussion, direct observation, feedback and problem management skills are still lacking.     

Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel

The Script Concordance Test (SCT) assesses reasoning in the context of uncertainty. Because there is no single correct answer, scoring is based on the comparison of answers provided by examinees with those provided by members of a reference panel made up of experienced practitioners. The study aimed to assess the discriminatory power of the SCT based on the variability of the reference panel's answers. Items from a bank covering different family medicine domains were classified into 3 groups according to the degree of variability of answers provided by a pool of experienced doctors. A variability index (mean squared error) was used to select items in the low, moderate and high variability categories. A 102-item test (Cronbach's alpha 0.70), made up of 3 subtests of each category, was administered to 3 contrasting groups in family medicine: 157 clerkship students, 30 residents and 30 practising doctors. anova and effect size (ES) were used to quantify and test the discrimination power of the 3 subtests. The high variability subtest showed high effect size for discrimination between extreme groups (ES = 1.5; F = 16.3, P < 0.001), whereas the moderate variability subtest showed less effect size (ES = 0.56; F = 57, P = 0.041). The low variability subtest did not discriminate significantly (ES = 0.31; F = 2.9, P = 0.06). Variability of answers within the reference panel is a key component of the discriminatory power of the SCT. In accordance with theory, the presence of variability ensures discrimination between levels of clinical experience. These results imply important considerations for the construction of efficient SCTs.     

Early identification of 'at-risk' students by the parents of paediatric patients

INTRODUCTION: Assessment of medical student clinical skills is best carried out using multiple assessment methods. A programme was developed to obtain parent evaluations of medical student paediatric interview skills for feedback and to identify students at risk of poor performance in summative assessments. METHOD: A total of 130 parent evaluations were obtained for 67 students (parent participation 72%, student participation 58%). Parents completed a 13-item questionnaire [Interpersonal Skills Rating Scale (IPS) maximum score 91, higher scores = higher student skill level]. Students received their individual parent scores and de-identified class mean scores as feedback, and participants were surveyed regarding the programme. Parent evaluation scores were compared with student performance in formative and summative faculty assessments of clinical interview skills. RESULTS: Parents supported the programme and participating students valued parent feedback. Students with a parent score that was less than 1 standard deviation (SD) below the class mean (low IPS score students) obtained lower faculty summative assessment scores than did other students (mean +/- SD, 59% +/- 5 versus 64% +/- 7; P < 0.05). Obtaining 1 low IPS score was associated with a subsequent faculty summative assessment score below the class mean (sensitivity 0.38, specificity 0.88). Parent evaluations combined with faculty formative assessments identified 50% of students who subsequently performed below the class mean in summative assessments. CONCLUSIONS: Parent evaluations provided useful feedback to students and identified 1 group of students at increased risk of weaker performance in summative assessments. They could be combined with other methods of formative assessment to enhance screening procedures for clinically weak students.