Done or Almost Done? Improving OSCE Checklists to Better Capture Performance in Progress Tests

Construct: The impact of using nonbinary checklists for scoring residents from different levels of training participating in objective structured clinical examination (OSCE) progress tests was explored. Background: OSCE progress tests typically employ similar rating instruments as traditional OSCEs. However, progress tests differ from other assessment modalities because learners from different stages of training participate in the same examination, which can pose challenges when deciding how to assign scores. In an attempt to better capture performance, nonbinary checklists were introduced in two OSCE progress tests. The purposes of this study were (a) to identify differences in the use of checklist options (e.g., done satisfactorily, attempted, or not done) by task type, (b) to analyze the impact of different scoring methods using nonbinary checklists for two OSCE progress tests (nonprocedural and procedural) for Internal Medicine residents, and (c) to determine which scoring method is better suited for a given task. Approach: A retrospective analysis examined differences in scores (n = 119) for two OSCE progress tests (procedural and nonprocedural). Scoring methods (hawk, dove, and hybrid) varied in stringency in how they awarded marks for nonbinary checklist items that were rated as done satisfactorily, attempted, or not done. Difficulty, reliability (internal consistency), item-total correlations and pass rates were compared for each OSCE using the three scoring methods. Results: Mean OSCE scores were highest using the dove method and lowest using the hawk method. The hawk method resulted in higher item-total correlations for most stations, but there were differences by task type. Overall score reliability calculated using the three methods did not differ significantly. Pass–fail status differed as a function of scoring methods and exam type, with the hawk and hybrid methods resulting in higher failure rates for the nonprocedural OSCE and the dove method resulting in a higher failure rate for the procedural OSCE. Conclusion: The use of different scoring methods for nonbinary OSCE checklists resulted in differences in mean scores and pass–fail status. The results varied with procedural and nonprocedural OSCEs.     

Home trials of a speech synthesizer in severe dysarthria: patterns of use, satisfaction and utility of word prediction.

OBJECTIVE: The aim of this study was to evaluate a speech synthesizer with respect to patterns of use and satisfaction, during a 2-month trial at home, and the usefulness of the word prediction function. DESIGN: Prospective study. PARTICIPANTS: Of the 24 patients with severe dysarthria recruited, 10 completed the study. Five patients had cerebral palsy, 3 amyotrophic lateral sclerosis, one locked-in syndrome, and one anoxic brain damage. Mean age was 32 (standard deviation 21) years (range 9-66 years). METHODS: Each participant received 10 hours of training with the device (Dialo) and then used it at home for 2 months. The main outcome measures were: level of use recorded by the device, Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST) satisfaction score (maximum = 5), and time needed to take dictations of standard-dictionary and personal-dictionary words with and without word prediction. RESULTS: Level of use varied widely across participants. Overall satisfaction at the end of the home trial was high, with a mean QUEST score of 3.4 (SD 1) and was related to the level of use of the device. Level of satisfaction at the end of the training session could not predict the level of use at home. No significant differences were found in dictation-taking times with and without word prediction. However, 6 of the 10 patients took dictation faster with than without word prediction. CONCLUSION: This study provides the first evidence supporting the benefits of a speech synthesizer used at home for several weeks. Word prediction is useful for some patients even if increase in dictation speed did not reach significance.     

Potential role for psychological skills training in emergency medicine: Part 1 ‐ Introduction and background

Psychological skills training (PST) is the systematic acquisition and practice of different psychological techniques to improve cognitive and technical performance. This training consists of three phases: education, skills acquisition and practice. Some of the psychological skills developed in this training include relaxation techniques, focusing and concentration skills, positive ‘self‐suggestion’ and visualisation exercises. Since the middle of the 20th century, PST has been successfully applied by athletes, performing artists, business executives, military personnel and other professionals in high‐risk occupations. Research in these areas has demonstrated the breadth and depth of the training's effectiveness. Despite the benefits realised in other professions, medicine has only recently begun to explore certain elements of PST. The present paper reviews the history and evidence behind the concept of PST. In addition, it presents some aspects of PST that have already been incorporated into medical training as well as implications for developing more comprehensive programmes to improve delivery of emergency medical care.     

Scales to assess sleep impairment in Parkinson's disease: Critique and recommendations

There is a broad spectrum of sleep disturbances observed in Parkinson's disease (PD). A variety of scales have been applied to the evaluation of PD sleep and wakefulness, but only a small number have been assessed specifically for clinimetric properties in the PD population. The movement disorder society has commissioned this task force to examine these scales and to assess their use in PD. A systematic literature review was conducted to explore the use of sleep scales in PD and to determine which scales qualified for a detailed critique. The task force members, all of whom have extensive experience in assessing sleep in PD reviewed each of the scales using a structured proforma. Scales were categorized into recommended, suggested and listed according to predefined criteria. A total of 48 potential scales were identified from the search and reviewed. Twenty‐nine were excluded because they did not meet review criteria or were variations of scales already included, leaving 19 scales that were critiqued and rated by the task force based on the rating criteria. Only six were found to meet criteria for recommendation or suggestion by the task force: the PD sleep scale (PDSS) and the Pittsburgh sleep quality index (PSQI) are recommended for rating overall sleep problems to screen and to measure severity, the SCOPA‐sleep (SCOPA) is recommended for rating overall sleep problems both to screen and to measure severity, and for rating daytime sleepiness; the Epworth sleepiness scale (ESS) is recommended for rating daytime sleepiness to screen and to measure severity; the inappropriate sleep composite score (ISCS) is suggested for rating severe daytime sleepiness or sleep attacks to screen and to measure severity; and the Stanford sleepiness scale (SSS) is suggested for rating sleepiness and to measure severity at a specific moment. The task force does not recommend the development of new scales, but emphasizes the need for educational efforts to train physicians in sleep interview techniques and polysomnography. © 2010 Movement Disorder Society     

Parkinson's disease dementia can be easily detected in routine clinical practice

Parkinson's disease (PD) is mainly characterized by its motor manifestations, but it is also frequently associated with dementia. Early diagnosis of PD dementia (PDD) is particularly important because effective cholinesterase inhibitor treatments are available. This study aimed at validating a short procedure for screening for PDD in routine clinical practice and which adopts recently published diagnostic criteria. One hundred eighty‐eight patients with PD participated in the study. The examination procedure comprised three steps: standard clinical examination, a short cognitive function assessment fulfilling the requirements of the Movement Disorders Society (Mini Mental State Examination, five‐word test, word generation task, and impact on daily life, including a questionnaire on compliance with medication) and an extensive evaluation of cognitive functions and behavior. After each step, the suspected presence or absence of dementia was recorded. After the short cognitive function assessment, PDD was suspected in 18.62% of the patients [95% confidence interval (CI): 13.32‐24.93%]. After the extensive assessment, 21.81% (95% CI: 16.13‐28.40%) met the criteria for probable PDD. The short battery's sensitivity and specificity were 65.85% (95% CI = 49.41‐79.92%) and 94.56% (95% CI = 89.56‐97.62%), respectively. A stepwise logistic regression analysis showed that use of a specific cut‐off considerably enhanced the short battery's sensitivity (85.37%, 95% CI = 70.83‐94.43%) without decreasing its specificity (83.67%, 95% CI = 76.69‐89.25%). With an easy‐to‐use, short battery of tests that are commonly used in routine clinical practice, it is possible to diagnose PDD in accordance with reference criteria and with the same sensitivity and specificity as in a more extensive evaluation. © 2010 Movement Disorder Society