Inter-observer reliability of clinical outcome measures in a lower limb amputee population

Purpose : In an attempt to find a more clinically useful functional outcome measure specifically tailored for lower limb amputees undergoing inpatient prosthetic rehabilitation, a 6-month prospective assessment of inter-rater reliability for Harold Wood - Stanmore Mobility Scale Data, including two handicap scales, was undertaken. An analysis of the data is presented in this paper.

Methods : An inter-rater reliability study was undertaken using four observers to complete admission and discharge scores for the three disability/handicap scales on 14 consecutive patients over 6 months.

Results : The disability mobility scale demonstrated perfect observer agreement on admission and at discharge the inter-rater reliability for this measure was high (0.83). By contrast, reliability between observers for admission scores on the handicap mobility scale was poor at 0.49 but reasonably high on discharge (0.83). On admission, inter-rater reliability for handicap physical independence was very low (0.15). At discharge, reliability improved to 0.69 being more consistent with results achieved for the other axes.

Conclusions : This study confirms the good inter-rater reliability demonstrated previously in the literature but reveals poor inter-rater reliability for the two handicap scales. The latter will require modification before they can be used with confidence in conjunction with the disability scale.     

康复护理日常生活活动评定量表的信度和效度研究

背景在康复医学中有许多种日常生活活动(ADL)能力评定量表,但符合康复护理程序的评定量表甚少.为此,浙江省康复中心从2000年开始研究设计了康复护理ADL(rehabilitative nursing ADL,RNADL)评定量表,并制定了评定内容和评定标准.目的探讨康复护理ADL评定量表在患者日常生活活动功能评测中的重测信度、内部一致性及量表的有效信度.设计重复测量设计.单位浙江省望江山疗养院,浙江省康复中心.对象2001-04/2003-04浙江省望江山疗养院康复医学科进行康复治疗的ADL功能障碍的40例患者.方法对40例住院康复患者用RNADL量表进行评测者内部、评测者之间的信度分析;内部一致性用Cronbach α计算;并分别用RNADL和改良的Barthel指数(MBI)进行出、入院各项目评定,用Pearson分析入院、出院及出入院之间的相关性.主要观察指标RNADL评定量表的信度和效度检测结果.结果RNADL评测者内部和评测者之间的信度良好,各项组内相关系数(ICC)范围为0.966～0.998,r=0.996～0.997;内部一致性信度系数α=0.822,各项之间α=0.925,RNADL内部一致性较好.RNADL和MBI的比较研究得出出、入院RNADL与MBI改变值间相关性良好(r=0.89,P＜0.001).结论RNADL具有很高的重测信度,内部一致性较好,与MBI有良好的关联效度,是康复护理日常生活活动能力可信、有效的评测量表.     

Is a day hospital rehabilitation programme associated with reduction of handicap in stroke patients?

OBJECTIVE: (1) To assess whether a rehabilitation day hospital programme is associated with a reduced handicap level of stroke patients. (2) To estimate the relationship between the London Handicap Scale (LHS) and other outcome measures. (3) To examine the effect of demographic parameters (age, gender, family status, education) on LHS scores. DESIGN: A prospective longitudinal survey. SETTING: An urban geriatric rehabilitation day hospital. SUBJECTS: Two hundred and seven elderly stroke patients admitted between December 1999 and February 2001. Main outcome measures: London Handicap Scale (LHS), Functional Independent Measure (FIM), Nottingham Extended ADL Index, timed get up and go test. RESULTS: LHS scores at discharge changed significantly (p < 0.008) for mobility, physical independence and occupation. The overall change in LHS score was 2.3 points (20%); effect size 0.43. A significant relationship was found between discharge score of LHS and admission score of FIM, Nottingham Index, timed get up and go and age. Multiple linear regressions did not identify a good predictor for the discharge score of LHS. Higher education was associated with higher LHS scores on admission (p = 0.016) but with less success in correcting handicap (p = 0.046). CONCLUSIONS: A day hospital programme is associated with reduced level of handicap in stroke patients. The LHS is a useful and simple scale for measuring change in these patients. LHS in stroke patients correlates with other outcome measures, yet they cannot be used interchangeably. A significant relationship between education and level of handicap exists.     

The Orpington Prognostic Scale for patients with stroke: reliability and pilot predictive data for discharge destination and therapeutic services

PURPOSE: To determine the inter-rater and test-retest reliability of the Orpington Prognostic Scale (OPS) in patients with stroke. Pilot data were gathered to evaluate its predictive validity for discharge destination and therapeutic services required on discharge.METHOD: Ninety-four consecutive patients, admitted to hospital due to stroke participated. Pairs of physiotherapists (PT) and occupational therapists (OT) assessed patients using the OPS on days 7 and 14 post stroke. For inter-rater reliability, one rater performed the OPS while the other observed, each scoring the scale independently. For test-retest reliability, two different raters tested the subjects separately within the same day. Data were gathered on the discharge destination and the number of follow-up services prescribed.RESULTS: The inter-rater reliability as measured by the intraclass correlation coefficient (ICC) was 0.99 (95% CI 0.97 - 0.99). For test-retest reliability, the ICC was 0.95 (95% CI 0.90 - 0.98). The accuracy for predicting discharge to home using OPS 5.0 was 65% (95% CI 0.52 - 0.76). OPS scores were not related to number of follow-up services prescribed.CONCLUSIONS: Despite high inter-rater and test-retest reliability, the OPS has limited predictive accuracy for discharge destination and is a poor predictor of follow-up services.     

The validity,reliability, and responsiveness of the modified Iowa Level of Assistance scale in hospitalized older adults in subacute care

Abstract

Objective: The objective of this study was to investigate the psychometric properties of the Modified Iowa Level of Assistance scale in hospitalized older adults in subacute care.

Design: A cohort, measurement-focused study.

Participants and setting: Fifty-eight older adults, aged 65?years and older, were recruited from a subacute rehabilitation hospital.

Methods: Inter-rater reliability was established by having two physiotherapists independently assess each participant within 24-h of each other. Construct validity was established using “known-groups” validity, while concurrent validity was also examined by correlating modified Iowa Level of Assistance scores with the Elderly Mobility Scale. Responsiveness was assessed by examining the difference in modified Iowa Level of Assistance scores from admission to discharge.

Results: The mean age of participants was 82.8?years (SD 7.5; range 68–97). The modified Iowa Level of Assistance scale was found to be reliable, valid, and responsive in this sample of hospitalized older adults. It had excellent inter-rater reliability (intraclass correlation coefficient [2,1] 0.96; 95% confidence intervals (CI) 0.93, 0.98) and no systematic differences across the range of scores. The scale displayed a mean difference between two known groups of 11.4 points and correlated significantly and negatively with the Elderly Mobility Scale (Spearman’s rho???0.90). The modified Iowa Level of Assistance score also changed significantly over the course of the hospital admission with an effect size of 1.2.

Conclusions: The modified Iowa Level of Assistance scale is a valid measure with excellent inter-rater reliability in hospitalized older adults. It is responsive to functional change during hospital admission and may be useful for routine outcome assessment for hospitalized older adults in subacute care.

• Implications for Rehabilitation

• The mILOA scale is a valid, reliable, and responsive outcome measure that can be used to quantify the gait and mobility impairments in hospitalized older adults in subacute care.

• For optimal reliability and responsiveness, consistent administration of the mILOA scale will be required particularly for higher level mobility tasks such as negotiating a step.

     

Disability scales: assessment of reliability

Modified or newly developed disability scales have to be assessed for their validity in terms of an appropriate standard and for reproducibility--inter- and intraobserver variability and intrasubject variability. For ordinal scales with more than two points or categories, correlation or regression coefficients are appropriate estimates of validity. An assessment of the sensitivity and specificity of such a scale is not feasible. Indices of proportion agreement or correlation analysis are frequently used to assess the reproducibility of disability scales. These procedures do not, however, correct for chance-expected agreement between two or more sets of observations. In a study of a 31-point ADL (activities of daily living) index used to measure the level of disability in patients with chronic diseases, scores independently rated by two observers were strongly correlated (r = 0.962), yet the two sets of observations were significantly different. An estimate of kappa statistic, that corrects for chance agreement, showed that there was in fact a poor (36.3%) overall agreement between the observers. It is concluded that the correlation coefficient often overestimates the degree of true agreement, may conceal significant disagreements, and may give misleading information about reproducibility. The kappa statistic should always be used in the assessments of reproducibility of disability scales. Agreement between more than two sets of observations can also be assessed by estimating kappa.     

Rehabilitation status in multiple handicap.

The Edinburgh Rehabilitation Status Scale (ERSS) was applied to 129 attenders at a day center for physical disability. All of the attenders had significant neurological impairment dating from birth or from infancy. Fifty-nine of the subjects (46%) had been diagnosed on conventional grounds as having mental handicap as well as physical disability. ERSS scores were compared with Barthel Index scores in all subjects and with PULSES profile in 50 subjects (27 physical disability alone, 23 combined with mental handicap). The ERSS scores clearly demonstrated significant differences in the level of disablement between the two groups in all four subscales as well as in total scores; the Barthel scores showed differences in self-care and total scores, and, to a lesser extent, in mobility. PULSES failed to differentiate the groups except in one of its subscales. The ERSS is a sensitive index of overall function and is useful in highlighting the additive effect of mental handicap and physical disability, a difference which may not be apparent when other assessment scales are used.     

Comparison of three preverbal scales for postoperative pain assessment in a diverse pediatric sample

The purposes of this study were to compare the reliability and validity of three pain measurement scales for assessing pain in preverbal and nonverbal children and to determine which of the scales was mos tappropriate in a clinical setting to evaluate pain for infants and young children regardless of developmental stage or cognitive or physical disability. Pain scales tested were revised versions of the Riley Infant Pain Scale (RIPS), the Nursing Assessment of Pain Intensity (NAPI), and the Postoperative Pain Score (POPS). Purposive sampling of 391 postoperative infants and children was used for evaluation of pain in a midwestern children's hospital. Four assessments with each scale were done 1 hr apart by trained observers blinded to pain medications. Data analyses supported high inter-rater reliability, satisfactory discrimination between pain and no-pain observations, and suggested acceptability for all three scales with lower caregiver burden for RIPS and NAPI.     

The measurement of psychotic acuity by nursing staff

Accurate evaluation of patients' psychotic state is essential to decrease psychotic symptoms and protect the patient and others. The aim of this paper is to conduct a literature review in order to access the utility, reliability and validity of current rating scales that are purported to measure psychotic acuity of inpatient population. A search of a number of electronic databases was undertaken to retrieve potential articles that focus on the measurement of acute psychosis. We identified some conceptual and theoretical problems when using a scale that is monitoring the progress of discharge and assesses the outcome of treatments. The findings revealed a difficulty in finding a commonly agreed definition of acute psychosis and a problem of obtaining frequent measures, and the frequency of measurement and fluctuation in psychosis. The most dominant scales in assessing psychosis were reviewed: the Global Assessment of Functioning Scale, the Brief Psychiatric Rating Scale and the Positive and Negative Symptom Scale. Several issues related to the scales' inter-rater reliability and construct validity remain unexplored. None of these scales addressed the conceptual and theoretical problems that we identified. A new scale that will measure acuity of symptoms in inpatient settings needs to be created.     

The Orpington Prognostic Scale for patients with stroke: Reliability and pilot predictive data for discharge destination and therapeutic services

Purpose.?To determine the inter-rater and test-retest reliability of the Orpington Prognostic Scale (OPS) in patients with stroke. Pilot data were gathered to evaluate its predictive validity for discharge destination and therapeutic services required on discharge.

Method.?Ninety-four consecutive patients, admitted to hospital due to stroke participated. Pairs of physiotherapists (PT) and occupational therapists (OT) assessed patients using the OPS on days 7 and 14 post stroke. For inter-rater reliability, one rater performed the OPS while the other observed, each scoring the scale independently. For test-retest reliability, two different raters tested the subjects separately within the same day. Data were gathered on the discharge destination and the number of follow-up services prescribed.

Results.?The inter-rater reliability as measured by the intraclass correlation coefficient (ICC) was 0.99 (95% CI 0.97 – 0.99). For test-retest reliability, the ICC was 0.95 (95% CI 0.90 – 0.98). The accuracy for predicting discharge to home using OPS ? 5.0 was 65% (95% CI 0.52 – 0.76). OPS scores were not related to number of follow-up services prescribed.

Conclusions.?Despite high inter-rater and test-retest reliability, the OPS has limited predictive accuracy for discharge destination and is a poor predictor of follow-up services.     

A retrospective analysis of outcome in severe Guillain-Barre syndrome following combined neurological and rehabilitation management

Purpose: To determine the outcome in severe Guillain-Barre syndrome following combined neurological and rehabilitation management using standardized disability and handicap measures and to identify the factors which affect this outcome. Methods: A retrospective study was performed of 24 patients with Guillain-Barre Syndrome admitted over a 3 year period to the neurological rehabilitation unit of the National Hospital for Neurology and Neurosurgery, Queen Square, London. Disability and handicap on admission and discharge were measured using the modified Barthel Index (BI), Functional Independence Measure (FIM), Environmental Status Scale (ESS) and Handicap Assessment Scale (HAS). Data was collected to identify any factors affecting outcome; age at onset, time to nadir, duration of ventilation, total in-patient stay, duration of stay at the neurological rehabilitation unit, symptoms, signs and electrophysiological findings. Results: The majority of patients had multiple problems. Outcome was related to the duration of rehabilitation. The mean modified BI score and the FIM score increased whereas ESS scores and HAS scores decreased, these changes were compatible with a reduction in disability and handicap. Conclusions: Significant improvement in function occurred during rehabilitation. These changes can be demonstrated using standardized outcome measures.     

Functional outcomes of cancer patients in an acute inpatient setting at King Fahad Medical City

[Purpose] Cancer survivors have functional impairments that adversely affect patients’ quality of life (QoL). Acute rehabilitation helps to reduce disability and improves QoL in cancer survivors. This study investigated the potential improvement in mobility levels and QoL of cancer patients during acute inpatient physical therapy (PT) from admission to discharge. [Participants and Methods] This was a cross-sectional study conducted at King Fahad Medical City, Riyadh. Acute inpatient cancer survivors (n=99) were assessed at their admission and discharge. The primary outcome measure was the AM-PAC “6-Clicks” Basic Mobility, Functional Assessment of Cancer Therapy-General (FACT-G7) and the Karnofsky Performance Scale (KPS). [Results] Overall, 82.8% of cancer patients were discharged home. There were significant improvements in all the three outcome measures for all the patients from admission to discharge. Patients who were discharged home exhibited significantly better improvement in all the scales. Factors that predicted discharge mobility and quality of life were discharge destination, number of PT sessions, and baseline admission scores. [Conclusion] The study found that acute inpatient cancer rehabilitation helps to improve mobility and QoL. Rehabilitation programs available in Saudi Arabia are limited, and it is important to integrate the cancer rehabilitation model into the oncology services.     

A new stroke activity scale-results of a reliability study

Purpose : To investigate the internal consistency, inter-rater and intra-rater reliability of a disability stroke activity scale (SAS) for stroke patients. Its intended use is as a measure of motor function at the level of disability in stroke patients.

Method : Twelve stroke in-patients were video-recorded performing the five activities from the SAS. Seven senior physiotherapists, experienced in stroke care, independently rated the recordings on two occasions, three weeks apart, using the SAS. Twelve hospital inpatients participated in the study. The subjects were aged between 48 and 86 and were between 6 and 87 days post stroke.

Results : Reliability for total scores was found to be excellent (generalizability correlation co-efficient (GCC) values ≥0.95) and reliability for individual item scores was good (kappa ≥0.7). Internal consistency reliability using Cronbach's alpha was also good (0.68 at time 1 and 0.68 at time 2).

Conclusion : The stroke activity scale is a reliable instrument for hospital stroke patients. It can be administered in less than 10 minutes and requires minimal equipment and training. Further work on the validity and responsiveness of the SAS is in progress.     

The Subjective Index of Physical and Social Outcome: its usefulness in a younger stroke population

Studies on stroke subjects that aim to improve their well-being or community support have not been shown to be effective when measures of disability and handicap have been employed. This paper illustrates the usefulness of the Subjective Index of Physical and Social Outcome (SIPSO) with young adults following a stroke. The study comprised a cross-sectional survey of people (57% male; 39% female; median age 55.7) with a stroke (1-10 years ago), aged 18-65, recruited via NHS stroke registers and young stroke groups (response rate 53%). The SIPSO was incorporated into a postal needs-assessment questionnaire. Median SIPSO score: 24.5 (IQR 17-32). The SIPSO internal reliability: ICC 0.91 (95% CI, 0.90-0.92), item to total SIPSO correlations ranged from 0.52-0.83. The SIPSO construct validity was good: those with poorer employment, mobility and sex-life outcomes had lower SIPSO scores than those with better outcomes; there was no association between age and SIPSO scores, SIPSO scores were not significantly different for men and women. Test-retest reliability was good. The SIPSO had excellent reliability and validity properties in his population. Further work on its responsiveness needs to be carried out. The measurement of personal experience of integration can be a vital basis for effective clinical care.     

Relationship between self-reported function and disability and balance performance measures in the elderly

This study evaluated the reliability and validity of the Hebrew version of the Late-Life Function and Disability Instrument (LLFDI). Fifty-five older adults (mean age 79.7 +/- 5.2) participated. We calculated test-retest reliability with intraclass correlation coefficients (ICCs). Partial correlations determined the construct validity with a balance measure (Berg Balance Scale [BBS]) and a mobility measure (Timed Up and Go [TUG] test). We examined known-group validity by comparing the scores of cane and noncane users. Test-retest ICCs ranged from good to excellent (0.77-0.90) for the function component and fair to good for the disability component (0.63-0.83), except for the disability management role subscale (0.46). BBS and TUG were associated with LLFDI overall function (r = 0.48, p < 0.001 and r = -0.52, p < 0.001, respectively). TUG and BBS were weakly associated with disability limitations (r = -0.26 and 0.32, respectively) and disability frequency (r = -0.16 and 0.24, respectively). Cane users showed significantly lower function scores than noncane users. We demonstrated that the Hebrew version of the LLFDI reliably and validly assesses older adults' function and disability. The LLFDI is recommended as an outcome instrument in studies in which older adults' function and disability are outcomes of interest.     

The reliability of balance, mobility and self-care measures in a population of adults with a learning disability known to a physiotherapy service

OBJECTIVE: To estimate the reliability of three measures of balance, mobility and activity for use in clinical and research physiotherapy, with adults with a learning disability. DESIGN: Prospective study to investigate test-retest and inter-rater reliability. SETTING: Participants' homes and day centres. MEASURES: The Berg Balance Scale (BBS), the Rivermead Mobility Index (RMI) and the Barthel Activities of Daily Living Index (BI). PARTICIPANTS: Of the 181 adults known to the Nottingham Community Physiotherapy Service for Adults with Learning Disabilities, 64 with a known Rivermead Mobility score of less than three were excluded. Of 117 randomized, a further 21 were found to fail this criteria, 27 had acute medical, social or behavioural problems, 22 were unable to participate or refused: therefore 47 entered the study. METHODS: Participants were visited in their own homes by two researchers on two occasions, one week apart and rated independently by each rater. Agreement was assessed with the kappa statistic (kappa) and percentage agreement for each item in each scale, and described using standard classification. Intraclass correlation coefficients for inter-rater and test-retest total scores and average differences of total scores, their standard deviations and limits of agreement, were calculated. RESULTS: For inter-rater observations, the Barthel Index and the Rivermead Mobility Index had almost perfect agreement (kappa = 0.86-1.00 and 0.89-1.00 respectively), with the Berg Balance Scale having substantial to almost perfect agreement (kappa = 0.74-1.00). For test-retest comparisons, both the Barthel Index and the Rivermead Mobility Index demonstrated moderate to almost perfect agreement (kappa = 0.57-1.00 and 0.45-1.00 respectively). Kappa scores for the Berg Balance Scale varied from low to almost perfect agreement (kappa = 0.37-1.00). CONCLUSIONS: The Berg Balance Scale, Rivermead Mobility Index and Barthel Activities of Daily Living Index are all reliable clinical and research tools for physiotherapists working with adults with learning disabilities.     

The modified rivermead mobility index: validity and reliability

PURPOSE: This paper presents the evaluation of the following psychometric properties of the Modified Rivermead Mobility Index (MRMI): face/content validity, responsiveness, test-retest reliability, inter-rater reliability and internal consistency. This mobility scale represents a further development of the Rivermead Mobility Index (RMI). In its new form the scoring was adapted from a two-point to a six-point scale. The number of test items was reduced from fifteen to eight items in order to measure mobility-related items that physiotherapists consider being essential for demonstrating treatment effects in patients following a stroke. METHOD: A consensus exercise with forty-two physiotherapists attending a stroke care conference established face/content validity. Inter-rater and test-retest reliability were examined by assessing thirty patients by two independent raters selected from a pool of eight physiotherapists in two different settings, an elderly care unit and a stroke rehabilitation unit. All patients were hospitalised and had experienced a stroke within the past six weeks. Responsiveness was examined by calculating the effect size statistic on the admission and discharge score of sixteen acute patients following stroke. RESULTS: The results showed that the modified RMI was: responsive to change (effect size = 1.15), stable when tested on two occasions (paired t-test = 0.732; p = 0.47), highly reliable between raters (ICC = 0.98; p < 0.001) with high internal consistency (Cronbach's alpha = 0.93). CONCLUSIONS: These results suggest that when using the Modified RMI to assess patients in the early stages following stroke, similar results can be obtained by different raters, regardless of experience. However there needs to be a difference of more than 4.5 points (degree of measurement error at 95% confidence level) in the overall score to detect true changes in the patient's level of mobility.     

The modified Rivermead Mobility Index: validity and reliability

Purpose : This paper presents the evaluation of the following psychometric properties of the Modified Rivermead Mobility Index (MRMI): face/content validity, responsiveness, testretest reliability, inter-rater reliability and internal consistency. This mobility scale represents a further development of the Rivermead Mobility Index (RMI). In its new form the scoring was adapted from a two-point to a six-point scale. The number of test items was reduced from fifteen to eight items in order to measure mobility-related items that physiotherapists consider being essential for demonstrating treatment effects in patients following a stroke. Method : A consensus exercise with forty-two physiotherapists attending a stroke care conference established face/content validity. Inter-rater and test-retest reliability were examined by assessing thirty patients by two independent raters selected from a pool of eight physiotherapists in two different settings, an elderly care unit and a stroke rehabilitation unit. All patients were hospitalised and had experienced a stroke within the past six weeks. Responsiveness was examined by calculating the effect size statistic on the admission and discharge score of sixteen acute patients following stroke. Results : The results showed that the modified RMI was: responsive to change (effect size = 1.15), stable when tested on two occasions (paired t-test = 0.732; p 0.47), highly reliable between raters (ICC = 0.98; p<0.001) with high internal consistency (Cronbach's alpha = 0.93). Conclusions : These results suggest that when using the Modified RMI to assess patients in the early stages following stroke, similar results can be obtained by different raters, regardless of experience. However there needs to be a difference of more than 4.5 points (degree of measurement error at 95% confidence level) in the overall score to detect true changes in the patient's level of mobility.     

Robinson Bashall Functional Assessment for arthritis patients: reliability and validity

The Robinson Bashall Functional Assessment (RBFA) for arthritis patients has four scales: self-care, ambulation, transportation, and activity tolerance. Criticism has been directed at the Assessment because it is relatively time consuming and requires a trained professional to conduct it. Tests of reliability revealed that the internal consistency of the total Assessment was moderate, interrater reliability was high, and test-retest reliability ranged from moderate to high. Analysis of scale validity (content, clinical, concurrent, and discriminant validity) showed content validity to be low to moderate. Therapists found the scales more useful in identifying patient problems than in assessing improvement in function over time. There were low positive correlations between dressing, undressing, and ambulation scores and ROM scores on admission. Correlations between the American Rheumatism Association Functional Classification and scales of the RBFA were significant, while the Assessment as a whole and all four scales discriminated significantly between the functional abilities of patients on admission and discharge.