Standard Setting Methods for Pass/Fail Decisions on High-Stakes Objective Structured Clinical Examinations: A Validity Study

Construct: Authentic standard setting methods will demonstrate high convergent validity evidence of their outcomes, that is, cutoff scores and pass/fail decisions, with most other methods when compared with each other. Background: The objective structured clinical examination (OSCE) was established for valid, reliable, and objective assessment of clinical skills in health professions education. Various standard setting methods have been proposed to identify objective, reliable, and valid cutoff scores on OSCEs. These methods may identify different cutoff scores for the same examinations. Identification of valid and reliable cutoff scores for OSCEs remains an important issue and a challenge. Approach: Thirty OSCE stations administered at least twice in the years 2010–2012 to 393 medical students in Years 2 and 3 at Aga Khan University are included. Psychometric properties of the scores are determined. Cutoff scores and pass/fail decisions of Wijnen, Cohen, Mean–1.5SD, Mean–1SD, Angoff, borderline group and borderline regression (BL-R) methods are compared with each other and with three variants of cluster analysis using repeated measures analysis of variance and Cohen's kappa. Results: The mean psychometric indices on the 30 OSCE stations are reliability coefficient = 0.76 (SD = 0.12); standard error of measurement = 5.66 (SD = 1.38); coefficient of determination = 0.47 (SD = 0.19), and intergrade discrimination = 7.19 (SD = 1.89). BL-R and Wijnen methods show the highest convergent validity evidence among other methods on the defined criteria. Angoff and Mean–1.5SD demonstrated least convergent validity evidence. The three cluster variants showed substantial convergent validity with borderline methods. Conclusions: Although there was a high level of convergent validity of Wijnen method, it lacks the theoretical strength to be used for competency-based assessments. The BL-R method is found to show the highest convergent validity evidences for OSCEs with other standard setting methods used in the present study. We also found that cluster analysis using mean method can be used for quality assurance of borderline methods. These findings should be further confirmed by studies in other settings.     

Test-retest reliability of the Xhosa version of the Diagnostic Interview Schedule for Children

Background The Diagnostic Interview Schedule for Children (DISC‐IV) is a widely used structured psychiatric diagnostic instrument for children that is designed to be administered by non‐clinicians. Although reliability data on several versions of the DISC are available, none have assessed the Xhosa version for use in the South African context. The objective of this study was to examine the test–retest reliability of the Xhosa version of the DISC‐IV. Methods Test–retest reliabilities were documented for a sample of 105 parent/care giver and youth pairs. The research assistants worked in pairs, one interviewed the parent and the other interviewed the youth. The same researchers returned after approximately 2 weeks to conduct the second interviews. Results The DISC‐IV was reliable across informants for many psychiatric disorders. Kappa coefficients ranged from 0.448 (any anxiety disorder) to 0.662 (major depressive disorder) for parent reports and from 0.145 (anxiety) to 0.661 (major depressive disorder) for child reports. Conclusion The results of this study suggest that the test–retest reliability of the Xhosa version of the DISC‐IV is similar to the reliability reported in other translated versions of the instrument. The satisfactory reliability and straightforward application make this instrument suitable for use in South Africa.     

影响军校新学员意志力和理性的心理社会因素路径分析

目的调查和分析影响军校新学员意志和理性的心理社会因素,为心理咨询和治疗提供科学依据。方法采用人性的哲学量表的意志与理性分量表、艾森克个性问卷、生活事件量表、症状自评量表对某军校562名新学员进行心理测评。统计学处理采用秩变换单因素方差分析、SPEARMAN相关分析及AMOS路径分析。结果新学员意志力和理性与新学员的个性因素具有显著相关性(r=0.128~0.267,P<0.01),负性意志力与理性与SCL-90的总分及各因素具有显著相关性(r=0.14~0.22,P<0.01)。AMOS路径分析显示,个性因素在心理健康与正、负性意志力与理性之间具有较强的中介效应,对正性意志力与理性的总调节效应为9.72%〔(0.54×0.18)×100〕,对负性意志力与理性的总调节效应为18.4%〔(0.54×0.34)×100〕。结论影响新学员意志力与理性的主要因素是新学员的个性因素及心理健康状态。     

Psychometric properties of the 15-item geriatric depression scale in functionally impaired, cognitively intact, community-dwelling elderly primary care patients

OBJECTIVES: To examine the psychometric properties of the 15-item Geriatric Depression Scale (GDS-15), a brief depression screening measure. DESIGN: Cross-sectional. SETTING: Nineteen counties in western New York, West Virginia, and Ohio. PARTICIPANTS: Nine hundred sixty functionally impaired, cognitively intact, community-dwelling primary care patients aged 65 and older. MEASUREMENTS: The GDS-15, major depression as measured using the Mini-International Neuropsychiatric Interview, depressed mood, life satisfaction, suicidal ideation, and reported suicide attempts. RESULTS: Exploratory factor analyses suggested a two-factor structure for the GDS-15 in this category of patients, with component subscales assessing depression and positive affect. Cronbach alpha coefficients provide evidence for moderate, although acceptable, internal consistency reliability. Significant associations between the GDS-15 and measures of depressed mood, life satisfaction, and suicidal ideation demonstrated construct validity, whereas acceptable sensitivity and specificity to discriminate between depressed and nondepressed patients demonstrated criterion validity. Internal consistency reliability and construct validity did not differ significantly between patients with low and high functional impairment. A significant weakness of the scale is its low correlation with suicide attempt status. CONCLUSION: In general, this study provides evidence of impressive psychometric properties of the GDS-15 when administered to a sample of functionally impaired, cognitively intact, community-dwelling primary care patients.     

Validation of the translation and cross-cultural adaptation into Finnish of the Abdominal Symptom Questionnaire,the Hospital Anxiety and Depression Scale and the Complaint Score Questionnaire

Background: Epidemiological surveys require questionnaires to be validated in the native language of the participants. The aim of this study was to validate the Finnish translations of the Abdominal Symptom Questionnaire (ASQ), the Hospital Anxiety and Depression Scale (HAD) and the Complaint Score Questionnaire (CSQ). Methods: A random sample of adults (n?=?3000) in a northern Swedish bilingual district was surveyed using a mailed ASQ offered in both Swedish and Finnish, and 2122 responded (239 in Finnish). A random subsample of the responders (n?=?1001, 123 preferring Finish) was then surveyed once more using the ASQ, the HAD and the CSQ. The first 50 responders of the latter survey were then given the three questionnaires again within two weeks. The Finnish versions had been put through a comprehensive translation procedure Results: A factor analysis comparison between the responders using either language in the mailed survey gave a comparable factor construction, and this was also comparable with an earlier analysis of the Swedish version. The Finnish responses to the second survey were further evaluated by testing internal consistency reliability, convergent validity towards previously validated relevant instruments (ShortForm-36 and the Gastrointestinal Symptom Rating Scale) and the test/re-test accuracy of the three questionnaires. These were found to be reliable, as was the correlation between the ASQ and the CSQ, for relevant domains. Conclusion: The Finnish translations of the ASQ, HAD and CSQ questionnaires all seem to be robust and usable for population-based surveys among Finnish adults.     

A Rasch-based validation of a short version of ABILHAND as a measure of manual ability in adults with unilateral upper limb amputation

Purpose.?To evaluate the measurement properties of ABILHAND (a generic measure developed to assess functioning in people with upper limb impairments) when used in adults with unilateral upper limb amputation (ULA).

Methods.?A convenience sample of 72 adults who had unilateral ULA and completed rehabilitation at the Institute for Rehabilitation in Ljubljana at least 1 year prior to the study. They filled in the ABILHAND questionnaire. Rating scale analysis (Rasch model) was used to evaluate functioning of the rating scale categories, the validity of the measure by examining fit of the items to the latent trait and the hierarchy of item difficulties compared with expectations of the construct.

Results.?Rasch analysis allowed us to improve ABILHAND by rescoring to reduce the response categories from 5 to 4, and identifying 22 of 46 items that are useful to measure upper limb function in people with ULA. The results indicate that high confidence can be placed in the consistency of both person-ability and item-difficulty estimates.

Conclusions.?This revised ABILHAND for people with unilateral ULA (ABILHAND-ULA 1.0) is a promising instrument for measuring their degree of manual functioning.     

Brain electrical activity (quantitative EEG and bit-mapping neurocognitive CNV components), psychometrics and clinical findings in presenile subjects with initial mild cognitive decline or probable Alzheimer-type dementia

Clinical, neuropsychological and neuropsychophysiological data (Q-EEG, ERPs and CNV/RT activity) were obtained from 24 patients who had more or less severe presenile primary cognitive decline without depression, and compared with similar data from 10 age-matched healthy volunteers (mean age, 59.4 years). All of the patients (15 M and 9 F; mean age 59.6 years) were selected according to the DSM III-R, ICD-10 and NINCDS-ADRDA criteria and underwent CT and MRI scanning, in addition to a standard clinical examination, a battery of psychometric tests, spectral EEG, and bit-mapped CNV complex and RT to S2 analyses. Twelve of the 24 patients presented an initial presenile idiopathic cognitive decline (PICD) but did not wholly fulfil the clinical and neuropsychological criteria for primary dementia or for a diagnosis of probable AD; the remaining 12 patients showed characteristic clinical signs and symptoms of a very probable early stage of presenile Alzheimer-type dementia (PAD). ANOVA, correlational and discriminant analyses of the neuropsychological test scores, and the neurophysiological and RT to S2 data revealed 22 highest-ranked between-group discriminant factors (all with a significance level of p<0.01). The conclusive discriminant analysis retained 13 of these factors as final canonical functions, and these showed a 97% grouping accuracy (33 of the 34 subjects examined); the same percentage of correct classifications was also achieved using only the 15 best indicators in the group of CNV/RT findings. Using both of these sets of highest-ranked discriminators, all of the normal subjects and all of the PAD patients were correctly classified; only 1 PICD patient was misclassified as normal when the first group of 13 factors was used, and another PICD patient was misclassified as PAD using the second group of 15 factors. Our findings suggest that, providing they are correctly performed and interpreted, these non-invasive techniques may be an important tool for identifying incipient stages of presenile Alzheimer-type dementia.This investigation was supported by National Research Council Grants (Aging Program Project: CNR No. 02854.04; 00394. PF40; 00467. PF40; Code No 954569).