Validating simulation and assessment devices in surgery: A review

There are standard criteria for scientific soundness that should be used to guide the development of all measures. The most important of these are validity and reliability. These criteria ensure that the data produced can be used with a reasonable level of confidence. This paper reviews the procedures required to demonstrate the validity and reliability of surgical simulators. It is concluded that those involved in developing surgical simulators should focus their efforts on demonstrating the construct validity and test-retest reliability of the devices.     

Test–Retest Reliability of the Preroad DriveABLE Competence Screen

ABSTRACT

Objective: To determine the test–retest reliability of a computerized preroad screen of prerequisite skills required for safe driving. Subjects & Methods: Driveable Screen administered to 39 individuals referred for driving evaluation due to cognitive decline or stroke. The Screen provides three recommendations—driving cessation, indeterminate, no evidence of reduced competence. Predicted probability of failing the road test is also generated. Results: Reliability according to the recommendation was substantial for the entire group (K = 0.654; CI = 0.459–0.848) and subgroup with stroke (K = 0.742; CI = 0.540–0.944). The numbers of participants with cognitive decline was too small to allow subgroup analyses. Of the nine participants with test–retest disagreement in recommendation, Test 2 was always better (McNemar's X² = 9.00, p = 0.01). Excellent reliability was found on probability of failing the Road Test (Intraclass Correlation Coefficient = 0.908; CI = 0.832–0.950). The test–retest reliability of the subtests ranged from fair to substantial. Conclusions: The Screen demonstrates sufficient test–retest reliability to encourage its use. However, the finding of significantly better scores on Test 2 for a portion of participants raises concern regarding its validity with repeat testing.     

Test–retest reliability and agreement of the Satisfaction with the Assistive Technology Services (SATS) instrument in two Nordic countries

Abstract

Purpose: The purpose of this study was to investigate test–retest reliability, agreement, internal consistency, and floor- and ceiling effects of the Danish and Finnish versions of the Satisfaction with the Assistive Technology Services (SATS) instrument among adult users of powered wheelchairs (PWCs) or powered scooters (scooters). Method: Test–retest design, two telephone interviews 7–18 days apart of 40 informants, with mean age of 67.5 (SD 13.09) years in the Danish; and 54 informants with mean age of 55.6 (SD 12.09) years in the Finnish sample. Results: The intra-class correlation coefficient varied between 0.57 and 0.93 for items in the Danish and between 0.41 and 0.93 in the Finnish sample. The percentage agreement varied between 54.2 and 79.5 for items in the Danish and between 69.2 and 81.1 in the Finnish sample, while the Cronbach’s alpha values varied between 0.87 and 0.96 in the two samples. A ceiling effect was found in all items of both samples. Conclusions: This study indicates that the SATS may be reliably administered for telephone interviews among adult PWC and scooter users, and give information about aspects of the service delivery process for quality development improvement purposes. Further psychometric testing of the SATS is required.     

The reliability of the Leeds Movement Performance Index (LMPI): a new tool for neurological physiotherapy

Abstract

Background: Measuring movement performance in people with neurological damage requires a tool that reflects physiotherapy assessment and clinical reasoning. The Leeds Movement Performance Index (LMPI) was previously developed by a group of neurological physiotherapists to fulfill these requirements. Objective: To assess the reliability of the LMPI for use in neurological physiotherapy practice. Methods: Twelve senior neurological physiotherapists were trained to use the LMPI and then asked to measure the movement performance of five patients whose movement had been previously video-recorded for this purpose. A retest session was completed after two weeks. Data were analysed to establish internal and external reliability. Results: Internal reliability was assessed using Cronbach's alpha coefficient, applied to the entire scale (0.862) and to each item (range 0.795–0.892). External (inter-rater) reliability was assessed by a calculation of the intraclass correlation coefficient for scores awarded by multiple raters (0.959), with individual item reliability ranging from 0.874 to 0.968. External (test–retest) reliability was assessed by calculating the Spearman's rank correlation coefficient between scores obtained on two testing occasions (0.792) with values of individual items ranging from 0.397 to 0.674. A variance components analysis partitioned variance into components arising from between-patient variability (55.2%) between-therapist variability (7.8%) and between-testing variability (2.8%). Conclusions: Results indicate that the LMPI is a reliable measurement tool when used by senior neurological physiotherapists.     

Clinical scales for the assessment of spasticity,associated phenomena,and function: a systematic review of the literature

Purpose: To characterise clinical assessment methods for spasticity and/or its functional consequences in clinical patient populations at risk to suffer from spasticity. Method: Systematic literature search and manual-based two-step review process of psychometric properties of clinical assessment scales for spasticity and associated phenomena, as well as of functional scales with an association with spasticity. Reviewed psychometric properties included internal consistency, interrater, intrarater as well as retest reliability, construct validity, ecological validity, and responsiveness. Results: Until May 2003 electronic database searches established a reference pool of 4151 references of which 90 references contributed to the review objectives. An additional 20 references were identified by an informal reference search. Twenty-four clinical scales that assess spasticity and/or related phenomena as well as 10 scales for 'active function' and three scales for 'passive function' with an association with spasticity could be identified. Some evidence signals that a high interrater reliability of the Ashworth and modified Ashworth scales can be achieved, however not in all circumstances. For many scales, reliability data is, however, missing. This is especially true for test retest reliability. Information about construct validity can promote our understanding of what individual scales are likely to assess. Many scales have been able to document changes after therapeutic intervention. Conclusions: The collated evidence can guide our clinical decision about when to use which scale and can promote evidence-based assessment of spasticity and related clinical phenomena.     

Alberta婴儿运动量表在正常婴儿中的信度研究

目的：了解Alberta婴儿运动量表（Alberta infant motor scale, AIMS）在中国北京城区正常足月婴儿中应用时的组间信度及组内信度。方法：在中国北京城区居住的45名正常足月婴儿加入本研究,平均月龄（6.89±2.97）个月（最小月龄4个月,最大月龄12.5个月）;其中男婴26例,女婴19例;小于6个月龄的婴儿23例,大于6个月龄的婴儿22例。3名评估者（评估者A、B、C）加入本研究。在组间信度的研究中,评估者A应用AIMS对婴儿进行现场运动评估并摄像,评估者B、C分别通过录像进行独立地AIMS评分,通过三者对同一婴儿的AIMS评分计算组间信度。在距第一次评估至少1个月以上,评估者B、C分别根据录像再次对每个婴儿进行AIMS评估,通过同一评估者对同一婴儿两次评估的AIMS评分计算组内信度。通过计算组内相关系数（intraclass correlation coefficient,ICC）检测信度。结果：在组间信度的研究中,总体ICC=0.995,小于6个月龄组ICC=0.903,大于6个月龄组ICC=0.974;在组内信度的研究中,总体ICCs=0.997—0.999,小于6个月龄组ICCs=0.892—0.972,大于6个月龄组ICCs=0.987—0.998。结论：AIMS在评估中国北京城区正常足月婴儿运动发育水平时具有较高的信度。