Warding off a tyranny of reliability and validity

Researchers in the healthcare communication field come from many different educational backgrounds. Such diversity generally strengthens a field, but sometimes a set of beliefs or a particular orthodoxy may predominate in ways that are negative. We discuss one such example, noting how the research culture deriving from training in schools of education treats the concepts of reliability and validity. We note that some researchers working in fields such as medical education and healthcare communication use the terms “reliable” and “valid” loosely or even incorrectly, often referring to them as a single catch-phrase. More importantly, we caution healthcare communication researchers against a tyranny of reliability and validity in which researchers feel pressure to avoid creating unique instruments to study new questions, instead using instruments with previously demonstrated reliability and validity even when these may not directly capture the concept of interest. This practice is motivated by realistic fears that reviewers and editors will disapprove of their work because the instruments used are not known to be “reliable and valid.” We encourage the research community to take a more balanced approach wherein originality is not stifled, and in which creativity and rigor exist side by side.     

制备中国人物静态面部表情图片及信度效度的初探

目的初步制备开心、生气、吃惊、害怕、伤心、讨厌和中性7种中国人物静态面部表情图片库以提供情绪研究的取材,探讨采集真实人物面部表情的方法。方法选取学龄前期、学龄期、小学、中学、成年早期、成人中期和成人晚期5～80岁五官端正的健康中国人。告知受试者各种面部表情内在情绪涵义、等级强度及面部表情特征,继而以谈话法结合个体背景和经历诱发面部表情,所有面部表情均在统一拍摄条件下由同一人采用数码相机拍摄。初步制备7种基本面部表情图片,经研究小组成员初步筛选出的图片进行统一处理,制作成10cm×15cm不同面部表情图片;进而以健康大学生对面部表情图片进行第2次筛选,对筛选后识别一致率较高的面部表情图片进行重测信度评价,同时以日本女性面部表情图片为参照评价效度。结果第1次筛选出80张面部表情图片,59名大学生第2次筛选出21张面部表情图片（每种面部表情各3张）。28名大学生进行重测信度、表情强度和效度测试。21张面部表情图片的重测信度较日本女性面部表情图片略好,除生气2张、害怕3张、伤心1张及讨厌3张图片重测信度低于70％外,其余表情重测信度均较高,负性表情（生气、害怕、伤心和讨厌）图片的效度较低,开心、中性和吃惊表情图片的效度较高。结论中国人物静态面部表情是具有代表性和信度较高的表情图片,可作为情绪研究的材料。面部表情的表达及内在情绪的判断受异族效应和文化差异的影响。     

Real-life step and activity measurement: reliability and validity

Objectives: Investigation of the reliability and validity of activity monitoring using a range of methods, namely manual counting, self report and two commercially available activity monitors.

Study design: Reliability, accuracy and validity study.

Methods: Multiple convenience samples of healthy subjects were recruited to the study. Reliability of manual step count was determined using an intraclass correlation coefficient (ICC) (n = 10). Relationships between data from the Step Watch? monitor (SAM) and (a) manual step counts (n = 18); (b) a second (different) activity monitor (ActivPAL); and (c) self reported activity levels (n = 22) were assessed using correlations. A Pearson's correlation and paired t-test was used to assess relations between routinely used monitors.

Results: Intra-rater reliability for manual step counts was excellent (ICC 0.99), but inter-rater reliability was poor (ICC 0.26). Indoor accuracy of the SAM was 96.06% and outdoor accuracy was 99.58%. Moderate correlations (rho = 0.423 to 0.595, p < 0.05) were identified between the SAM monitor activity levels and self report diaries. The SAM and the ActivPAL were found to be internally reliable within themselves (ICC 0.96 and 0.95 respectively), significantly correlated (r = 0.93, p < 0.001) but also significantly different (t = 2.179, p < 0.05) when used simultaneously over the same circuit.

Conclusions: Activity monitors provide information that is related to actual activity and provide accurate and reliable data when tested on functional walking circuits. Activity monitors should not be used interchangeably due to the potential for systematic differences between the measurements obtained when applied simultaneously over the same repeated circuit.     

The Mood Disorder Questionnaire (MDQ) for detecting (hypo)manic episodes: Its validity and impact of recall bias

Background

Bipolar disorders often remain unrecognized in clinical practice, which may be a consequence of imprecise recall of manic symptoms earlier in life. This study will therefore examine the validity of the widely-used Mood Disorder Questionnaire (MDQ) in detecting a (hypo)manic episode and explore the impact of recall bias.

Methods

As an indication of impairments in recalling manic symptoms, we examined the long-term reliability of the MDQ after two years of follow-up in a sample of 2087 persons. Then, the validity of the MDQ was tested against the gold standard of a CIDI-based DSM-IV (hypo)manic episode. Its performance was compared for detecting a lifetime episode (at T1) versus a recent episode in the past two years (at T2).

Results

The long-term reliability of the MDQ was limited as the correct recall of individual items ranged from 44.6% to 68.8% after two years. The overall validity of the MDQ in detecting a lifetime (hypo)manic episode was limited and no adequate cut-off point with acceptable sensitivity and specificity could be identified. However, the MDQ accurately detected a recent episode with a sensitivity of 0.83 and a specificity of 0.82 for the standard and optimal cut-off point of ≥7. Taking into account two additional MDQ questions on clustering in time and severity of problems decreased its validity.

Limitations

Patients with a primary, clinical diagnosis of bipolar disorder were excluded.

Conclusions

The MDQ accurately detected recent (hypo)manic episodes, but imprecise recall may result in a limited performance for episodes earlier in life.     

中文版青少年癫痫患者生活质量问卷信度及效度检验

目的：将英文版青少年癫痫患者自陈式生活质量调查问卷（QOLIE-AD-48）翻译为中文,并检验其信度、效度及敏感度。方法：信度指标包括内部一致信度及重测信度．效度指标包括聚合及判别效度、结构效度。敏感度分析通过比较不同发作严重程度总分的差异是否显著来评价。结果：聚合／判效度结果显示,分量表内的何道题目与所在分量表的相关系数在0．4以上,每一题目与所在分量表的相关性高于其他分量表。总分内部一致信度为0．93,分量表在0．60～0．91之间,总分重测信度为0．76。总分随发作严重程度的增加而显著下降,敏感度良好。结论：QOLIE-AD-48中文版具有良好的效度、信度及敏感度,可以胜任中国青少年癫痫患者健康相关生活质量研究所需。     

Development,reliability and validity of a new motorized isometric dynamometer for measuring strength characteristics of elbow flexor muscles

The aim of this study was to develop a new apparatus capable of accurately recording and analysing torque – time curve during isometric muscle contractions of the elbow flexors. A total of 25 (age 29.3 ± 5.0 years) non-athlete male volunteers participated in this study. To establish the measurement precision of the new testing apparatus and to determine possible sources of error in the results, a series of tests was performed. Instrument validity was assessed by comparing each selected measure (lever arm angular position, torque and maximal isometric torque) to a criterion (external inclinometer, calibrated weights and isokinetic dynamometer, respectively), while trial-to-trial and day-to-day comparisons of measurements were made to estimate the reproducibility of the instrument. The statistical outcomes of validity tests demonstrated high agreement between measures and associated criteria (Pearson correlation coefficient's values >.97). Trial-to-trial correlations for maximal isometric torque in different elbow angles showed excellent reproducibility, as values of intraclass correlation coefficient (ICC_[2,1]) ranged from 0.889 to 0.971, with higher value in the middle angle (90°). The value of ICC_[2,1] (day-to-day) for maximal isometric torque in 90 degrees was 0.986 and the respective standard error of measurement (SEM) was 1.5 Nm. Our results indicate that the isometric evaluation of elbow flexor muscles with the new dynamometer is accurate and reproducible.     

Validation of the Seasonal Pattern Assessment Questionnaire (SPAQ)

The validity of the Seasonal Pattern Assessment Questionnaire (SPAQ) was examined by interviewing 81 individuals who had participated in an earlier community survey of seasonal affective disorder (SAD) in Iceland. When SAD and subsyndromal SAD (S-SAD) were combined into a ‘winter problem’ group, the questionnaire's sensitivity, specificity and positive predictive value for that group were 94%, 73% and 45%, respectively. The SPAQ discriminated poorly between SAD and S-SAD, and hence it had a poor case-finding ability for SAD. Clinical evaluation verified a diagnosis of SAD in individuals who had no previous information about this syndrome. The questionnaire furthermore identified a group of individuals who had generalized anxiety and marked seasonal variations. Clinical evaluation arrived at a similar prevalence rate of SAD as the questionnaire.     

The stability of the Seasonal Pattern Assessment Questionnaire score index over time and the validity compared to classification according to DSM-III-R

BACKGROUND: In 1984, Rosenthal et al. described a group of patients characterised by repeated winter depression with atypical symptoms eventually followed by summer mania or hypomania (seasonal affective disorder, SAD). The relationship between SAD and the classical manic-depressive disorder is uncertain. The aim of this study was to validate the Seasonal Pattern Assessment Questionnaire (SPAQ) classification in relation to the DSM-III-R criteria of seasonal patterns in a group of patients with bipolar affective disorder and to evaluate the stability of the SPAQ score index over time. METHODS: A group of bipolar patients were followed for 3 years with examinations every third month and at hospitalisation. At inclusion and at each following examination the patients were rated with the Hamilton Depression Rating Scale, the Newcastle Depression Rating Scale, and the Bech-Rafaelsen Mania Rating Scale. At inclusion and once a year the patients completed the SPAQ. RESULTS: Fifty-six patients agreed to participate and 46 patients completed 1 year or more. Eight patients, all women, had at least one SPAQ score index (SSI) of 11 or more. Women scored significantly higher than men on SPAQ. The test/re-test reliability of SSI between two consecutive years was good, but decreased as the time difference between tests increased. Moreover, we found no relation between seasonal variations of affective episodes according to SSI and DSM-III-R. LIMITATIONS: The group of patients with seasonality according to DSM-III-R or SSI was small. CONCLUSION: The validity of SSI between two consecutive years is good, but decreases as the time difference between tests increases. There is no relationship between seasonality defined by DSM-III-R and SSI. Female bipolar patients show, as the general population, seasonal variations in mood, energy, sociality, appetite, and sleep independently of their affective episodes. SSI cannot be used for the prediction of seasonal variation in a group of bipolar patients.     

大学生求职韧性问卷编制

目的:编制大学生求职韧性问卷,并检验其在大学生群体中的信度和效度。方法:通过前人文献和访谈获得初始项目,对两个样本进行集体施测,样本一用于项目分析和探索性因素分析,样本二的数据用于验证性因素分析,样本二的部分被试同时完成外部效标验证的问卷,包括动机控制、情绪控制、乐观等,同时还搜集了部分被试的班主任老师对其就业力的评价数据。结果:(1)探索性因素分析显示,求职韧性问卷分为4个维度,分别是接受自我、坚持行动、能力肯定、情绪管理;(2)验证性因素分析结果显示,求职韧性问卷的4个因子在样本二的拟合良好;(3)外部效度验证结果显示,求职韧性与动机控制、情绪控制、乐观、就业力评分呈中等程度相关;(4)求职韧性问卷Cronbachα系数为0.83,接受自己、坚持行动、能力认可和情绪管理的Cronbachα系数分别为0.73、0.69、0.64、0.65。结论:求职韧性问卷在大学生群体中的信效度良好,可以用于相关研究中的测量和评估。     

大学生自我心态图问卷的初步编制

目的 编制大学生自我心态图问卷（CESQ）。方法 在746名大学生中测试试问卷,检验信效度。结果 内在一致性系数为0．8129;根据数据结果。把原始分数转化为标准分数．分析出大学生的自我心态图的9种典型类型;开朗型、苦闷型、任劳任怨型、自我主张型、合理型、矛盾型、压抑型、幼稚型、平凡型。结论 CESQ具有较好的信效度,可作为大学生自我心态的测评工具使用。     

系统家庭动力学自评问卷的初步编制及信效度分析

目的：根据系统家庭治疗的家庭动力学理论,编制一个适合中国家庭的系统家庭动力学自评问卷。方法：按入组织标准抽昆明市五所医院精神疾病患者150人,随机整机抽取高中生63人,医学大专及本科学生87人,用编制的系统家庭动力学问卷对入组被试作测验,并作信、效度检验。结果：定型问卷为4个维度,30个条目：全问卷同质信度Cronbach's α系为0．6420,4个维度系数0．6142－0．8712;重测信度0．6178－0．9110,总分相关0．7112;因子分析证实问卷的结构效度较好,并具有一定的判别效度。结论：初编的系统家庭动力学问卷的信、效度已符合心理测量学要求,可用于家庭动力学的评估。     

A comparison of normal, bipolar and seasonal affective disorder subjects using the Seasonal Pattern Assessment Questionnaire

The Seasonal Pattern Assessment Questionnaire is an instrument for retrospective self-rating of change in mood and vegetative functions with the seasons. It has been used in studies to identify and characterise patients with Seasonal Affective Disorder (SAD) in three countries. In this paper, the test-retest reliability of all items and the sensitivity to differences between groups have been investigated. Clear differences were found between the SAD group and a normal group. A group of bipolar affective disorder patients had intermediate scores on reported seasonal changes between the normal and SAD groups. All groups reported significant changes in mood and vegetative functions during the winter in the direction symptomatic for SAD suggesting that SAD patients may suffer an exaggerated form of a normal seasonal variation.     

防御方式问卷(DSQ)信度和效度研究

目的 :探讨防御方式问卷 (DSQ)的信度和效度。方法 :采用防御方式问卷对 873例被试进行测试。结果 :防御方式问卷的大多数条目需要修改 ,部分条目应该淘汰。其重测信度在 0 .7以上 ,α系数为 0 .9。因素分析表明 ,DSQ测量的 2 4种防御方式可以归为五类 :释放型的不成熟防御方式、压抑型的不成熟防御方式、反向型的中间防御方式、释放型的中间防御方式和成熟的防御方式。结论 :本研究显示 ,DSQ的信度和效度均有待于进一步研究支持     

Validity and reliability of the Brazilian Portuguese version of the BACS (Brief Assessment of Cognition in Schizophrenia)

OBJECTIVE:To assess the validity and reliability of the Brazilian Portuguese version of the Brief Assessment of Cognition in Schizophrenia by examining its temporal stability, internal consistency, and discriminant and convergent validity.METHODS:The Brief Assessment of Cognition in Schizophrenia was administered to 116 stable patients with schizophrenia and 58 matched control subjects. To assess concurrent validity, a subset of patients underwent a traditional neuropsychological assessment.RESULTS:The patients with schizophrenia performed significantly worse than the controls (p<0.001) on all subtests of the Brief Assessment of Cognition in Schizophrenia and on the total score, which attests to the discriminant validity of the test. The global score of the Brief Assessment of Cognition in Schizophrenia was significantly correlated with all of the subtests and with the global score for the standard battery. The Brief Assessment of Cognition in Schizophrenia also had good test-retest reliability (rho>0.8). The internal consistency of the Brief Assessment of Cognition in Schizophrenia was high (Cronbach''s α  ϝ 0.874).CONCLUSION:The Brazilian Portuguese version of the Brief Assessment of Cognition in Schizophrenia exhibits good reliability and discriminant and concurrent validity and is a promising tool for easily assessing cognitive impairment in schizophrenia and for comparing the performance of Brazilian patients with that of patients from other countries.     

家庭教养方式问卷信效度再检验

目的:进一步检验程灶火等编制的《家庭教养方式问卷》(FUSQ)的信度和效度.方法:在芜湖市学校、社区和医院等抽取720名学生、社会工作人员、吸毒者和神经症患者,用《家庭教养方式问卷》进行测试,间隔1个月对其中200名被试进行重测.结果:问卷α系数、分半信度和重测信度分别是0.49～0.92;0.63～0.93和0.68～0.96;验证性因素分析显示各拟合指数符合标准;病例组多数维度得分低于对照组(P＜0.05).结论:《家庭教养方式问卷》具有较好的信度和效度,可用于临床和科研测量个体主观感受到的家庭教养方式.     

青春期性心理健康量表在中学生应用中的信度和效度

目的:检验青春期性心理健康量表在我国中学生中的信度和效度。方法:对937名在校中学生进行青春期性心理健康量表测查,分析量表的内部一致性、重测信度、条目间平均相关系数、总量表与分量表间的相关系数,进行验证性因素分析,比较一般中学生与无保护性行为中学生量表得分的差异。结果:青春期性心理健康量表全量表Cronbachα系数为0.81,重测信度为0.75,条目间的平均相关系数在0.32～0.41之间;各分量表得分与总量表得分的相关系数在0.54～0.86之间,分量表间的相关系数在0.14～0.46之间;验证性因子分析:一阶七因子结构模型的拟合指标最好;发生无保护性行为的学生分量表得分及量表总分均低于未发生危险性行为的学生。结论:青春期性心理健康量表适用于我国中学生。     

The validity and reliability of the Dutch Effort-Reward Imbalance Questionnaire

The reliability and validity of the Effort-Reward Imbalance Questionnaire were tested in 775 blue- and white-collar workers in the Netherlands. Cronbach's alpha revealed sufficient internal consistency of all subscales except Need for Control. With exploratory probabilistic scaling (Mokken) analysis, the psychometric qualities of the Need for Control scale were improved. With confirmatory factor analysis, the factorial validity of the Extrinsic Effort and Reward subscales was confirmed. A model with 3 separate dimensions for reward (status control, esteem reward, and monetary gratification) proved adequate, emphasizing the importance of distinguishing subscales. The congruent validity of the subscales and a hypothesized relationship with an external construct, health functioning, were confirmed.