Uncontrolled Web-Based Administration of Surveys on Factual Health-Related Knowledge: A Randomized Study of Untimed Versus Timed Quizzing

Background

Health knowledge and literacy are among the main determinants of health. Assessment of these issues via Web-based surveys is growing continuously. Research has suggested that approximately one-fifth of respondents submit cribbed answers, or cheat, on factual knowledge items, which may lead to measurement error. However, little is known about methods of discouraging cheating in Web-based surveys on health knowledge.

Objective

This study aimed at exploring the usefulness of imposing a survey time limit to prevent help-seeking and cheating.

Methods

On the basis of sample size estimation, 94 undergraduate students were randomly assigned in a 1:1 ratio to complete a Web-based survey on nutrition knowledge, with or without a time limit of 15 minutes (30 seconds per item); the topic of nutrition was chosen because of its particular relevance to public health. The questionnaire consisted of two parts. The first was the validated consumer-oriented nutrition knowledge scale (CoNKS) consisting of 20 true/false items; the second was an ad hoc questionnaire (AHQ) containing 10 questions that would be very difficult for people without health care qualifications to answer correctly. It therefore aimed at measuring cribbing and not nutrition knowledge. AHQ items were somewhat encyclopedic and amenable to Web searching, while CoNKS items had more complex wording, so that simple copying/pasting of a question in a search string would not produce an immediate correct answer.

Results

A total of 72 of the 94 subjects started the survey. Dropout rates were similar in both groups (11%, 4/35 and 14%, 5/37 in the untimed and timed groups, respectively). Most participants completed the survey from portable devices, such as mobile phones and tablets. To complete the survey, participants in the untimed group took a median 2.3 minutes longer than those in the timed group; the effect size was small (Cohen’s r=.29). Subjects in the untimed group scored significantly higher on CoNKS (mean difference of 1.2 points, P=.008) and the effect size was medium (Cohen’s d=0.67). By contrast, no significant between-group difference in AHQ scores was documented. Unexpectedly high AHQ scores were recorded in 23% (7/31) and 19% (6/32) untimed and timed respondents, respectively, very probably owing to “e-cheating”.

Conclusions

Cribbing answers to health knowledge items in researcher-uncontrolled conditions is likely to lead to overestimation of people’s knowledge; this should be considered during the design and implementation of Web-based surveys. Setting a time limit alone may not completely prevent cheating, as some cheats may be very fast in Web searching. More complex and contextualized wording of items and checking for the “findability” properties of items before implementing a Web-based health knowledge survey may discourage help-seeking, thus reducing measurement error. Studies with larger sample sizes and diverse populations are needed to confirm our results.     

Validity of Internet-Based Longitudinal Study Data: The Elephant in the Virtual Room

Background

Internet-based data collection relies on well-designed and validated questionnaires. The theory behind designing and validating questionnaires is well described, but few practical examples of how to approach validation are available in the literature.

Objective

We aimed to validate data collected in an ongoing Internet-based longitudinal health study through direct visits to participants and recall of their health records. We demonstrate that despite extensive pre-planning, social desirability can still affect data in unexpected ways and that anticipation of poor quality data may be confounded by positive validation.

Methods

Dogslife is a large-scale, Web-based longitudinal study of canine health, in which owners of Labrador Retrievers were recruited and questioned at regular intervals about the lifestyle and health of their dogs using an Internet-based questionnaire. The Dogslife questionnaire predominantly consists of closed-answer questions. In our work, two separate validation methodologies were used: (1) direct interviews with 43 participants during visits to their households and (2) comparison of owner-entered health reports with 139 historical health records.

Results

Our results indicate that user-derived measures should not be regarded as a single category; instead, each measurement should be considered separately as each presents its own challenge to participants. We recommend trying to ascertain the extent of recall decay within a study and, if necessary, using this to guide data collection timepoints and analyses. Finally, we recommend that multiple methods of communication facilitate validation studies and aid cohort engagement.

Conclusions

Our study highlighted how the theory underpinning online questionnaire design and validation translates into practical data issues when applied to Internet-based studies. Validation should be regarded as an extension of questionnaire design, and that validation work should commence as soon as sufficient data are available. We believe that validation is a crucial step and hope our suggested guidelines will help facilitate validation of other Internet-based cohort studies.     

The dimensionality of the Edinburgh Handedness Inventory: An analysis with models of the item response theory

Handedness is frequently measured with sum scores or quotients taken from laterality questionnaires like the Edinburgh Handedness Inventory (EHI). In classical test theory such data cannot be used to confirm either the unidimensionality (i.e., quantitative differentiation with the poles left-handed and right-handed) or multidimensionality (i.e., typological differentiation between left-, right-, and mixed-handers) of this personal characteristic. This study uses item response theory models to test the construct validity of the EHI on an item level in order to gather empirical support for the differentiation of handedness as well as the appropriateness of the items and the response format. The EHI was given to 540 participants (303 male and 237 female) aged 17–37 years. Results of mixed-Rasch analyses revealed that the best model was a two-class solution; that is, left- and right-handers (types) with quantitative differences between persons. Hence, unlike earlier model tests, this rejects both the unidimensionality of the handedness construct and the need to consider so-called mixed-handers. It is proposed that mixed-Rasch analyses should be applied more frequently to test the construct validity of other as well as more extensive handedness questionnaires.     

Dental treatment need among 20 to 25‐year‐old Swedes: discrepancy between subjective and objective need

Abstract

Objective. This methodological study aimed to determine whether the mandibular trabecular bone assessment from panoramic radiographs, using a visual index, corresponds to the evaluation obtained from periapical radiographs. Material and methods. A panoramic radiograph and corresponding periapical radiographs of the region of the lower premolars and molars were collected from each of 32 patients (mean age 18.5 ± 5.5 years). Two calibrated observers assessed randomly the interdental sites between the first molar and second premolar and between the two premolars on all the radiographs using a visual index. Evaluations were repeated with an interval of 60 days. The results of the repeated evaluations were used to assess intra- and inter-observer agreements, employing Kappa statistics. Spearman's correlation was used to determine the association between assessments of panoramic and periapical radiographs. Results. In total, 79 interdental sites were evaluated on the panoramic and periapical radiographs. The visual analysis of periapical radiographs revealed intra-observer agreements of 0.88 for observer 1 and 0.93 for observer 2, and an inter-observer agreement of 0.82. The intra-observer agreement for panoramic radiographs was 0.79 and 0.83 for observers 1 and 2, respectively, and the inter-observer agreement was 0.79. A substantial correlation was found between periapical and panoramic radiographs (rho = 0.737, p = 0.001). Conclusions. Although panoramic radiographs are less reliable than periapical radiographs, they can be used for assessment of the trabecular bone pattern with the aid of a visual index. Training on the method is recommended to obtain results with a high reproducibility.     

基于EQ-5D量表的温州地区原发性开角型青光眼患者的生活质量研究

目的观察温州地区原发性开角型青光眼(POAG)患者的生活质量及其关联特征。方法横断面研究。纳入2014年3月至2019年10月在温州医科大学附属眼视光医院开展的温州青光眼进展研究中确诊的POAG患者339例,采用欧洲五维度生活质量(EQ-5D)量表[含视觉模拟尺度(VAS)]评估POAG患者生活质量,并分析不同性别、年龄、视力及视野缺损程度、青光眼家族史、高血压、糖尿病、偏头痛以及日常睡眠、业余锻炼与生活质量的关系。健康效用值与VAS评分以M(P25,P75)表示,两组间比较采用Mann-Whitney U检验,多组间比较采用Kruskal-Wallis H检验。结果339例POAG患者中男性164例(48.4%),女性175例(51.6%),年龄(63±10)岁。34例(10.0%)患者接受过药物治疗(含1例手术治疗),305例(90.0%)患者入组前均未接受过抗青光眼治疗。POAG患者视野较好眼中未出现视野缺损者占10.5%(32/305),早期、中期、晚期视野缺损者分别占68.9%(210/305)、17.0%(52/305)、3.6%(11/305)。EQ-5D量表测得的POAG患者的健康效用值为1.000(1.000,1.000),平均值为0.964;VAS评分为80(75,90)分,平均值为81.58分。共94例POAG患者EQ-5D量表测得健康效用值下降,其中焦虑或抑郁和疼痛或不适比例分别为45.7%(43例)和34.1%(32例),行动能力存在困难13.8%(13例),日常活动受限6.4%(6例)。POAG患者视野较好眼中未出现视野缺损者及早期、中期视野缺损者的健康效用值均为1.000(1.000,1.000),晚期视野缺损者的健康效用值为1.000(0.862,1.000),不同视野缺损程度多组间健康效用值及VAS评分比较差异均无统计学意义(均P>0.05)。不同睡眠质量间健康效用值差异有统计学意义(H=17.465,P<0.01),睡眠质量很好患者健康效用值为1.000(1.000,1.000),睡眠质量一般患者健康效用值为1.000(0.866,1.000),差异有统计学意义(z=3.613;P<0.05)。合并偏头痛、合并高血压、合并糖尿病患者的健康效用值均为1.000(0.875,1.000),无偏头痛、无高血压、无糖尿病患者的健康效用值均为1.000(1.000,1.000),是否合并偏头痛、高血压、糖尿病之间健康效用值差异均有统计学意义(Z=-2.189,-3.864,-2.417;均P<0.05)。EQ-5D量表健康效用值与年龄、性别、青光眼家族史、业余锻炼程度、烟酒史以及抗青光眼药物治疗史之间未发现明显关联(均P>0.05)。结论温州地区POAG患者EQ-5D量表结果显示,早期视野缺损的患者生活质量较好,晚期视野缺损的患者生活质量降低;睡眠质量、合并症及心理生理不适也与POAG患者生活质量降低有关。(中华眼科杂志,2021,57:207-214)