Validity and reliability of sincerity test for dynamic trunk motions

PURPOSE: Marras et al. developed a technique to evaluate sincerity of effort during dynamic trunk motion performance. The validity and reliability of the technique have not been evaluated. Therefore, the objective of this study was to first determine whether or not a sincerity of effort measure correctly identified those giving a sincere effort in a blinded randomized control trial and second to quantify inter-rater and test-retest reliability. METHODS: This article reports the findings of a two phase study. In phase one, the blinded evaluation, participants were randomly assigned to either a sincere or insincere performance condition. An examiner tested participants without knowledge of the participant's group membership. In the second phase, two examiners evaluated each participant twice to quantify inter-rater and test-retest reliability. RESULTS: In the blinded phase the specificity was 100% and sensitivity was 90% for identifying sincere and insincere effort, respectively. Phase two results showed no significant difference in probability of sincere effort between raters or between testing sessions. CONCLUSION: A performance criterion that accurately identifies sincere vs insincere group membership during functional evaluations was identified. There were no significant differences between raters or between testing sessions. These findings indicate that this test is reliable and possesses good predictive validity in assessing sincerity of effort.     

REPAS, a summary rating scale for resistance to passive movement: item selection, reliability and validity

Purpose. To establish: (i) item characteristics and item selection for the REPAS; (ii) internal consistency, inter-rater and test-retest reliability of the final REPAS version and its subtests; (iii) the association between the REPAS and selected other clinical scales of impairment and activity limitation.

Method. Thirty-three neurological patients with central paresis. Two REPAS assessments with a one-week interval by two independent raters. Concurrent assessment of the Motricity Index, Box-and-Block test, Functional Ambulation Category, Timed walking, Barthel Index, Disability Rating Scale, Carer Burden Scale, and Hygiene Score.

Results. Twenty-six of 52 REPAS items fulfilled the item selection criteria. The final test version showed a high internal consistency, inter-rater and test-retest reliability (correlation coefficients: 0.87 - 0.97, no significant difference between raters or with test repetition). Reliability of the arm and leg subtests was substantial (correlation coefficients: arm subtest 0.63 - 0.98, leg subtest 0.56 - 0.96). REPAS scores were moderately associated with basic ADL competence and a carer's burden with arm or leg adductor spasticity. The REPAS, arm subtest scores, degree of arm paresis and gross manual dexterity showed a moderately high association.

Conclusions. The Ashworth scale-based guidelines assured comparability of test administration and scoring. The REPAS is a reliable and valid summary rating scale for resistance to passive movement.     

Reliability of the Home Falls and Accidents Screening Tool (HOME FAST) for identifying older people at increased risk of falls

Purpose : The Home Falls and Accidents Screening Tool (HOME FAST) was designed as part of a comprehensive health assessment instrument to measure falls risk for older people within their home environment. This paper describes the evaluation of the inter-rater reliability of the HOME FAST.

Method : Forty home visits were undertaken, in urban and rural settings of the UK, by pairs of raters, one of whom was an expert rater. Occupational therapists, occupational therapy assistants and a social worker rated each home using the HOME FAST concurrently with an expert rater. The kappa statistic was used to determine the degree of agreement between pairs of raters.

Results : The overall kappa value for the checklist was 0.62, indicating a fair to good level of agreement between raters. 'Hazardous outside paths' was the only item that demonstrated poor agreement (kappa=0.30). The expert rater consistently identified more hazards than the other raters, and the level of agreement was stable between raters irrespective of the number of hazards present in the homes visited.

Conclusions : Definitions of non-applicable household features and improving the objectivity of the operational definitions for recognizing hazards that demand more qualitative judgements will enhance the reliability of the HOME FAST.     

Reliability and construct validity of the client-centred rehabilitation questionnaire

Purpose. A key component in assessing the performance of rehabilitation services is the client's perspective. The purpose of this paper is to report on the development of a publicly available measure of client-centred rehabilitation (CCRQ) that can be used for discriminative and evaluative purposes.

Method. Mixed qualitatative and quantitative methods were used. Phase 1: Identification of seven domains of client-centred rehabilitation based on a literature review, focus groups with clients, and review by content experts. Phase 2: Item generation for the seven conceptually derived subscales and cognitive interviews with inpatient rehabilitation patients. Phase 3: Psychometric testing for internal reliability, test-retest reliability and discriminative construct validity using data from a mailed, self-administered survey to 1568 patients discharged from two large inpatient rehabilitation facilities.

Results. The seven conceptually derived subscales all have strong internal (0.72 - 0.87) and test-retest reliability (0.74 - 0.85). Discriminative construct validity is demonstrated by the ability of subscales to identify significant differences between programs within two rehabilitation facilities.

Conclusions. The results for the reliability and validity of this measure support its value for use in clinical and quality improvement work as well as research.     

Inter-observer reliability of clinical outcome measures in a lower limb amputee population

Purpose : In an attempt to find a more clinically useful functional outcome measure specifically tailored for lower limb amputees undergoing inpatient prosthetic rehabilitation, a 6-month prospective assessment of inter-rater reliability for Harold Wood - Stanmore Mobility Scale Data, including two handicap scales, was undertaken. An analysis of the data is presented in this paper.

Methods : An inter-rater reliability study was undertaken using four observers to complete admission and discharge scores for the three disability/handicap scales on 14 consecutive patients over 6 months.

Results : The disability mobility scale demonstrated perfect observer agreement on admission and at discharge the inter-rater reliability for this measure was high (0.83). By contrast, reliability between observers for admission scores on the handicap mobility scale was poor at 0.49 but reasonably high on discharge (0.83). On admission, inter-rater reliability for handicap physical independence was very low (0.15). At discharge, reliability improved to 0.69 being more consistent with results achieved for the other axes.

Conclusions : This study confirms the good inter-rater reliability demonstrated previously in the literature but reveals poor inter-rater reliability for the two handicap scales. The latter will require modification before they can be used with confidence in conjunction with the disability scale.     

A new stroke activity scale-results of a reliability study

Purpose : To investigate the internal consistency, inter-rater and intra-rater reliability of a disability stroke activity scale (SAS) for stroke patients. Its intended use is as a measure of motor function at the level of disability in stroke patients.

Method : Twelve stroke in-patients were video-recorded performing the five activities from the SAS. Seven senior physiotherapists, experienced in stroke care, independently rated the recordings on two occasions, three weeks apart, using the SAS. Twelve hospital inpatients participated in the study. The subjects were aged between 48 and 86 and were between 6 and 87 days post stroke.

Results : Reliability for total scores was found to be excellent (generalizability correlation co-efficient (GCC) values ≥0.95) and reliability for individual item scores was good (kappa ≥0.7). Internal consistency reliability using Cronbach's alpha was also good (0.68 at time 1 and 0.68 at time 2).

Conclusion : The stroke activity scale is a reliable instrument for hospital stroke patients. It can be administered in less than 10 minutes and requires minimal equipment and training. Further work on the validity and responsiveness of the SAS is in progress.     

The QE Health Scale (QEHS): assessment of the clinical reliability and validity of a spiritually based holistic health measure

Purpose. To assess the clinical reliability and validity of a holistic health measure, the QE Health Scale (QEHS), for use with people with physical disabilities.

Method. A test-retest design saw the QEHS administered and compared with established measures of health at admission and discharge from three-week inpatient rehabilitation programmes. Data was analysed by factor and correlation analysis. Clinician-reported credibility and usefulness of the theoretical basis of the QEHS, the QEHS itself, and Patient Profiles derived from the QEHS were also used to evaluate clinical validity.

Results. The QEHS was judged to possess satisfactory reliability and validity.

Conclusion. The QEHS is a clinically reliable, valid, credible and useful holistic health instrument to facilitate client-centred therapeutic interventions, inform decision-making and evaluate outcomes for people with physical disabilities.     

The Orpington Prognostic Scale for patients with stroke: reliability and pilot predictive data for discharge destination and therapeutic services

PURPOSE: To determine the inter-rater and test-retest reliability of the Orpington Prognostic Scale (OPS) in patients with stroke. Pilot data were gathered to evaluate its predictive validity for discharge destination and therapeutic services required on discharge.METHOD: Ninety-four consecutive patients, admitted to hospital due to stroke participated. Pairs of physiotherapists (PT) and occupational therapists (OT) assessed patients using the OPS on days 7 and 14 post stroke. For inter-rater reliability, one rater performed the OPS while the other observed, each scoring the scale independently. For test-retest reliability, two different raters tested the subjects separately within the same day. Data were gathered on the discharge destination and the number of follow-up services prescribed.RESULTS: The inter-rater reliability as measured by the intraclass correlation coefficient (ICC) was 0.99 (95% CI 0.97 - 0.99). For test-retest reliability, the ICC was 0.95 (95% CI 0.90 - 0.98). The accuracy for predicting discharge to home using OPS 5.0 was 65% (95% CI 0.52 - 0.76). OPS scores were not related to number of follow-up services prescribed.CONCLUSIONS: Despite high inter-rater and test-retest reliability, the OPS has limited predictive accuracy for discharge destination and is a poor predictor of follow-up services.     

An evaluation of the hemiplegic patient based on the Bobath approach: a reliability study.

The intra- and inter-rater reliability of a motor function evaluation of stroke patients, based on the Bobath approach, was studied. The intraclass correlation coefficient (ICC) was used to determine the degree of agreement between repeated measurements on the same patient taken by the same rater and between measurements taken by three raters on the same patient. In the intra-rater study, each of 19 patients was evaluated in three different sessions by one of 19 raters. In the inter-rater study 18 patients were each evaluated by three different raters. The intra-rater data were highly reliable, with ICCs of 0.95 and 0.97 for the upper and lower limbs respectively. For the inter-rater study, the ICCs were 0.79 and 0.77 for the upper and lower limbs respectively. It can therefore be concluded that this instrument, previously demonstrated to quantify patient progress, is also reliable both in intra- and inter-rater dimensions.     

Assessment of neonatal resuscitation skills: a reliable and valid scoring system

OBJECTIVE: To study the reliability and validity of a scoring instrument for the assessment of neonatal resuscitation skills in a training setting. METHODS: Fourteen paediatric residents performed a neonatal resuscitation on a manikin, while being recorded with a video camera. The videotapes were analysed using an existing scoring instrument with an established face and content validity, adjusted for use in a training setting. Intra- and inter-rater reliability were assessed by comparing the ratings of the videotapes of three raters, one of who rated the videotapes twice. Intra-class coefficients (ICC) were calculated for the sum score, percentages of agreement and kappa coefficients for the individual items. To study construct validity, the performance of a second resuscitation of by residents was assessed after they had received feedback on their first performance. RESULTS: The ICC were 0.95 and 0.77 for intra- and inter-rater reliability, respectively. The median percentage of intra-rater agreement was 100%; inter-rater agreement 78.6-84.0%. The median kappa was 0.85 for intra-rater reliability, and 0.42-0.59 for inter-rater reliability. Residents showed a 10% improvement (95% confidence interval -4; 23%) on performance of a second resuscitation, which supports the instrument's construct validity. CONCLUSION: A useful and valid instrument with good intra-rater and reasonable inter-rater reliability is now available for the assessment of neonatal resuscitation skills in a training setting. Its reliability can be improved by using a more advanced manikin and by training of the raters.     

The new Swedish Post-Concussion Symptoms questionnaire: a measure of symptoms after mild traumatic brain injury and its concurrent validity and inter-rater reliability.

OBJECTIVE: To study the concurrent validity and the inter-rater reliability of the Post-Concussion Symptoms Questionnaire. DESIGN: The approach was to study the concurrent validity of the Post-Concussion Symptoms Questionnaire when used as an interview questionnaire compared with a self-report questionnaire administered by the patients. The inter-rater reliability was also studied when 2 different raters administered the Post-Concussion Symptoms Questionnaire interview. PATIENTS: Thirty-five patients with mild traumatic brain injury were consecutively contacted by telephone and asked whether they would be willing to participate in a follow-up intervention. METHODS: The Post-Concussion Symptoms Questionnaire was completed by the patients, who answered "Yes" or "No" to the standardized questions. The patients were then interviewed to check the certain "Yes" or "No" answers, 0-10 days after having completed the first Post-Concussion Symptoms Questionnaire. The raters filled in their ratings independently. RESULTS: The concurrent validity of answers in the questionnaire compared with those in the interview ranged from 82% to 100% agreement. The inter-rater reliability results ranged from 93% to 100% agreement between the raters. CONCLUSION: The Post-Concussion Symptoms Questionnaire with answers of "Yes" or "No" is a valid instrument. High reliability was found between the raters.     

Leisure Time Physical Activity Instrument and Physical Activity at Home and Work Instrument. Development, face validity, construct validity and test-retest reliability for subjects with fibromyalgia

Purpose. A new instrument measuring leisure time physical activity (LTPAI) in populations predominately engaging in low intensity activities and a new instrument measuring the Physical Activity at Home and Work (PAHWI) were designed.

Methods. Patients with long-lasting pain and expert physiotherapists participated in the development of the two instruments. Test-retest reliability was evaluated for the LTPAI and the PAHWI. Construct validity was evaluated for the LTPAI by comparing it with an instrument measuring physical activities for older people, six-minute walk test and aerobic capacity.

Population. 37 women with FM, with the mean age of 46 years (SD 8.4) and mean symptom duration of 11 years (SD 5.9) were recruited to the study.

Results. The mean time that the study population spent in physical activities during leisure time was 5.2 hours (SD 4.0) a week. Satisfactory test-retest reliability was found for the total score of LTPAI (ICC 0.86, CI 0.79 - 0.93) and for the PAHWI (ICC 0.91, CI 0.82 - 9.96). A significant association between the LTPAI and the six-minute walk test (r_s 0.40, p = 0.02) and another physical activity instrument (r_s 0.39, p = 0.02) was found. As expected, LTPAI did not have any association with aerobic capacity.

Conclusions. Face validity of the instruments was ensured during the development process. Satisfactory test-retest reliability was found for the LTPAI and the PAHWI. Significant but low associations were found between the LTPAI and the six-minute walk test and an instrument designed for older people, respectively, while no association was found between the LTPAI and aerobic capacity.     

The modified rivermead mobility index: validity and reliability

PURPOSE: This paper presents the evaluation of the following psychometric properties of the Modified Rivermead Mobility Index (MRMI): face/content validity, responsiveness, test-retest reliability, inter-rater reliability and internal consistency. This mobility scale represents a further development of the Rivermead Mobility Index (RMI). In its new form the scoring was adapted from a two-point to a six-point scale. The number of test items was reduced from fifteen to eight items in order to measure mobility-related items that physiotherapists consider being essential for demonstrating treatment effects in patients following a stroke. METHOD: A consensus exercise with forty-two physiotherapists attending a stroke care conference established face/content validity. Inter-rater and test-retest reliability were examined by assessing thirty patients by two independent raters selected from a pool of eight physiotherapists in two different settings, an elderly care unit and a stroke rehabilitation unit. All patients were hospitalised and had experienced a stroke within the past six weeks. Responsiveness was examined by calculating the effect size statistic on the admission and discharge score of sixteen acute patients following stroke. RESULTS: The results showed that the modified RMI was: responsive to change (effect size = 1.15), stable when tested on two occasions (paired t-test = 0.732; p = 0.47), highly reliable between raters (ICC = 0.98; p < 0.001) with high internal consistency (Cronbach's alpha = 0.93). CONCLUSIONS: These results suggest that when using the Modified RMI to assess patients in the early stages following stroke, similar results can be obtained by different raters, regardless of experience. However there needs to be a difference of more than 4.5 points (degree of measurement error at 95% confidence level) in the overall score to detect true changes in the patient's level of mobility.     

The Orpington Prognostic Scale for patients with stroke: Reliability and pilot predictive data for discharge destination and therapeutic services

Purpose.?To determine the inter-rater and test-retest reliability of the Orpington Prognostic Scale (OPS) in patients with stroke. Pilot data were gathered to evaluate its predictive validity for discharge destination and therapeutic services required on discharge.

Method.?Ninety-four consecutive patients, admitted to hospital due to stroke participated. Pairs of physiotherapists (PT) and occupational therapists (OT) assessed patients using the OPS on days 7 and 14 post stroke. For inter-rater reliability, one rater performed the OPS while the other observed, each scoring the scale independently. For test-retest reliability, two different raters tested the subjects separately within the same day. Data were gathered on the discharge destination and the number of follow-up services prescribed.

Results.?The inter-rater reliability as measured by the intraclass correlation coefficient (ICC) was 0.99 (95% CI 0.97 – 0.99). For test-retest reliability, the ICC was 0.95 (95% CI 0.90 – 0.98). The accuracy for predicting discharge to home using OPS ? 5.0 was 65% (95% CI 0.52 – 0.76). OPS scores were not related to number of follow-up services prescribed.

Conclusions.?Despite high inter-rater and test-retest reliability, the OPS has limited predictive accuracy for discharge destination and is a poor predictor of follow-up services.     

Reliability and validity of The Awareness of Social Inference Test (TASIT): a clinical test of social perception

Purpose. The Awareness of Social Inference Test (TASIT) is an audiovisual tool designed for the clinical assessment of social perception with alternate forms for re-testing. Part 1 assesses emotion recognition, Parts 2 and 3 assess the ability to interpret conversational remarks meant literally (i.e., sincere remarks and lies) or non-literally (i.e., sarcasm) as well as the ability to make judgments about the thoughts, intentions and feelings of speakers. This paper aims to examine TASIT's reliability and validity.

Method. Some 32 adults with severe, chronic brain injuries were administered Form A twice, one week apart. 38 adults with brain injuries were readministered alternate forms over a period of 5 - 26 weeks. Construct validity was examined in subsets of a sample of 116 adults with brain injuries by relating TASIT performance to standard tests of neuropsychological function and specific social perception measures.

Results. Test-retest reliability ranged from 0.74 - 0.88. Alternate forms reliability ranged from 0.62 - 0.83. TASIT performance was associated with face perception, information processing speed and working memory. Socially relevant new learning and executive tasks were significantly associated with TASIT performance whereas non-social tasks showed little association. Social perception tasks such as Ekman photos and theory of mind stories were also associated.

Conclusions. TASIT has adequate psychometric properties as a clinical test of social perception. It is not overly prone to practice effects and is reliable for repeat administrations. Performance on TASIT is affected by information processing speed, working memory, new learning and executive functioning, but the uniquely social material that comprises the stimuli for TASIT will provide useful insights into the particular difficulties people with clinical conditions experience when interpreting complex social phenomena.     

Reliability of four scales on balance disorders in persons with multiple sclerosis

PURPOSE: Balance rehabilitation is an important component of the retraining program in people with multiple sclerosis (MS). Measuring balance is fundamental for an accurate assessment and therapy selection. The aim of this study was to compare interrater and test-retest reliability of the Berg Balance Scale, the Dynamic Gait Index, the Dizziness Handicap Inventory and the Activities-specific Balance Confidence. METHOD: A group of 25 persons were enrolled in the study. The group consisted of 8 males and 17 females, mean age 41.7 years (12.5 years, SD). The onset of pathology was 8.7 years (8.8 years SD) before the beginning of the study. To assess the test-retest reliability two consecutive assessments were collected by the same rater. To assess the inter-rater reliability persons were concurrently assessed by two raters. RESULTS: The Intraclass Correlation Coefficients (ICCs) for interrater reliability ranged between 0.94 and 0.96. The ICCs for test-retest reliability ranged between 0.85 and 0.96. CONCLUSION: The inter-rater reliability of the instruments proved to be satisfactory. Lower but acceptable results were obtained for the test-retest paradigm. The data obtained in this study suggest that these scales are reliable tools for assessing balance function in persons suffering from MS.     

Reliability of the Body Awareness Scale-Health

In physical therapy the clinical assessment Body Awareness Scale-Health (BAS-H) focusing on the quality of movements and movement behaviour has previously been studied for validity. The aim of this study was to address the inter-rater reliability and test-retest reliability in three groups. The groups assessed were: patients in psychiatric care with eating disorders (n = 26), patients in rehabilitation of prolonged musculoskeletal pain (n = 22) and healthy individuals (n = 22). Results revealed inter-rater reliability (n = 70) of the BAS-H total to be 79.9 % with acceptable agreement (accepting one scale-step of difference) and 48.7% with perfect agreement. Weighted Kappa ranged between 0.34 and 0.92. Test-retest reliability (n = 54) as a mean for both raters were found to be 90.5% for the BAS-H total with acceptable agreement and 60.4% with perfect agreement. Weighted Kappa ranged between 0.65 and 0.92. The BAS-H seems to be a reliable assessment in the rehabilitation of patient with prolonged pain, psychiatric disorders and healthy controls when used according to the manual. The authors, however, suggest some revisions.     

The modified Rivermead Mobility Index: validity and reliability

Purpose : This paper presents the evaluation of the following psychometric properties of the Modified Rivermead Mobility Index (MRMI): face/content validity, responsiveness, testretest reliability, inter-rater reliability and internal consistency. This mobility scale represents a further development of the Rivermead Mobility Index (RMI). In its new form the scoring was adapted from a two-point to a six-point scale. The number of test items was reduced from fifteen to eight items in order to measure mobility-related items that physiotherapists consider being essential for demonstrating treatment effects in patients following a stroke. Method : A consensus exercise with forty-two physiotherapists attending a stroke care conference established face/content validity. Inter-rater and test-retest reliability were examined by assessing thirty patients by two independent raters selected from a pool of eight physiotherapists in two different settings, an elderly care unit and a stroke rehabilitation unit. All patients were hospitalised and had experienced a stroke within the past six weeks. Responsiveness was examined by calculating the effect size statistic on the admission and discharge score of sixteen acute patients following stroke. Results : The results showed that the modified RMI was: responsive to change (effect size = 1.15), stable when tested on two occasions (paired t-test = 0.732; p 0.47), highly reliable between raters (ICC = 0.98; p<0.001) with high internal consistency (Cronbach's alpha = 0.93). Conclusions : These results suggest that when using the Modified RMI to assess patients in the early stages following stroke, similar results can be obtained by different raters, regardless of experience. However there needs to be a difference of more than 4.5 points (degree of measurement error at 95% confidence level) in the overall score to detect true changes in the patient's level of mobility.