Measures of short-term test-retest reliability of computerized neurobehavioral tests

Neurobehavioral test batteries are often administered repeatedly to evaluate changes over time or effects of clinical interventions or neurotoxic exposures. Time intervals between test sessions range from very short (hours) to very long (decades). The aim of this study was to compare the impact of two brief time intervals on the test-retest reliability of frequently used neurobehavioral tests. Fifty healthy adults were recruited through newspaper advertisements in Portland, Oregon. Participants were divided into either a 6h (same-day) or 1-week retest group. All participants completed a battery of tests from the computerized Behavioral Assessment and Research System (BARS). Reliability was assessed by Pearson product-moment correlation and by intraclass correlation coefficient (ICC). The test battery generally showed adequate reliability in the short-term (week) and very short-term (day) and stability in performance over repeated administration when examined by multiple measures. Intraclass correlation coefficient ranged from 0.35 to 0.85. The magnitude of variation of performance in the administered tests was equally distributed around zero (i.e. no difference). The findings suggest that neurobehavioral tests such as BARS may be a useful tool for the assessment of acute exposures and clinical status where short-term evaluation is required (e.g. in the same-day or within 1 week).     

Adaptation of the Behavioral Assessment and Research System (BARS) for evaluating neurobehavioral performance in Filipino children

Neurobehavioral tests have long been used to assess health effects in exposed working adult populations. The heightened concern over the potential impact of environmental exposures on neurological functioning in children has led to the development of test batteries for use with children. There is a need for reliable, easy-to-administer batteries to assess neurotoxic exposure in children. One such test battery previously validated with Spanish- and English-speaking children ages 4 and older, combines computerized tests from the Behavioral Assessment and Research System (BARS) with non-computerized tests. The goal of the present study was to determine the feasibility of using standardized neurobehavioral tests in preschool and school-aged Filipino children. Test instructions were translated into the vernacular, Tagalog or Tagalog-English ("Taglish") and some instructions and materials were modified to be appropriate for the target populations. The battery was administered to 4-6-year-old Filipino children (N=50). The performance of the Filipino children was compared to data previously collected from Spanish- and English-speaking children tested in the US. The majority of children had no difficulty completing the tests in the battery with the exception of the Symbol-Digit test and Digit Span-reverse. The three groups showed similar patterns of performance on the tests and the older children performed better than the younger children on all of the tests. The findings from this study demonstrate the utility of using this test battery to assess cognitive and motor performance in Filipino children. Tests in the battery assess a range of functions and the measures are sensitive to age differences. The current battery has been utilized in several cultures and socio-economic status classes, with only minor modifications needed. This study demonstrates the importance of pilot testing the methods before use in a new population, to ensure that the test is valid for that culture.     

Further standardization of some NES3 tests

NES3 is a computer-based neurobehavioral testing system designed for use in investigating potential cognitive impairment. Data from NES3 tests employed in a study of epilepsy self-management were analyzed to estimate test-retest reliability for NES3 tests on a large sample and to estimate the effect of some common covariates of test performance. A total of 319 participants in an epilepsy self-management study were examined on three occasions (baseline, 3-month follow-up, and 6-month follow-up) with a set of psychological questionnaires and seven neuropsychological tests from NES3. Test-retest correlations were calculated between measures obtained at baseline and at 3 months. Principal components analysis was performed on the baseline data. The potential effects of covariates (age, education, reading test scores, depression status, and examiner) were investigated in regression models. Three-month test-retest correlations were excellent for Adult Reading Test (ART) (r=0.95), strong for Digit-Symbol (r=0.82), Sequence B (r=0.79), and Sequence A (r=0.76); and modest (r's between 0.56 and 0.67) for Digit Span Forward and Backward, Visual Span Forward and Backward, and Pattern Memory. Alternate-forms correlations were strong for HVLT (r's between 0.71 and 0.82). Principal components analysis yielded four interpretable components. Age and reading score were significant covariates of virtually all of the test summary measures, while education, gender, race, and depression were not generally significant covariates. Changes to the method of calculation of some summary measures, changes to the initial instructions to the subjects, and addition of correctional feedback to subjects during the tests appeared to improve the reliability of some NES3 tests. Implementation of the HVLT and ART in computer-assisted format added breadth of coverage to the battery. NES3 tests may provide reliable, efficient data for use in epidemiologic studies of potential cognitive effects of occupational and environmental exposures.     

Reliability and validity of the Norwegian version of the Brief Agitation Rating Scale (BARS) in dementia

Objective: To examine the test–retest reliability and validity of the Norwegian Brief Agitation Rating Scale (BARS), a short form of the Cohen-Mansfield Agitation Inventory (CMAI) assessing the frequency of agitation in dementia.

Methods: We investigated the internal consistency, test–retest reliability and the validity of BARS. In the validity study, we compared the BARS scores with the Neuropsychiatric Inventory – Nursing Home Version subscale Agitation/Aggression (NPI-NH/AA) and the Cornell Scale for Depression in Dementia subscale Agitation (CSDD/A).

Results: In the reliability study, Cronbach's alpha was 0.76; the test–retest reliability of the BARS showed a Spearman's rho of 0.64, but this increased to 0.86 when we deleted the item ‘complaining’. In the validation study, the BARS score correlated with the NPI-NH/AA and the CSDD/A scores, Spearman's rho 0.55 and 0.52, respectively. These correlations changed when controlling for the Clinical Dementia Rating (CDR) Scale stages. The highest correlations between the BARS and the NPI-NH/AA and the BARS and the CSDD/A were found among patients with CDR score 2.

Conclusions: The study indicates that the Norwegian version of BARS is a reliable and valid instrument to test agitation in dementia, but a version without the item ‘complaining’ would be better.     

Development of a brief ataxia rating scale (BARS) based on a modified form of the ICARS

To develop a brief ataxia rating scale (BARS) for use by movement disorder specialists and general neurologists. Current ataxia rating scales are cumbersome and not designed for clinical practice. We first modified the International Cooperative Ataxia Rating Scale (ICARS) by adding seven ataxia tests (modified ICARS, or MICARS), and observed only minimally increased scores. We then used the statistics package R to find a five‐test subset in MICARS that would correlate best with the total MICARS score. This was accomplished first without constraints and then with the clinical constraint requiring one test each of Gait, Kinetic Function‐Arm, Kinetic Function‐Leg, Speech, and Eye Movements. We validated these clinical constraints by factor analysis. We then validated the results in a second cohort of patients; evaluated inter‐rater reliability in a third cohort; and used the same data set to compare BARS with the Scale for the Assessment and Rating of Ataxia (SARA). Correlation of ICARS with the seven additional tests that when added to ICARS form MICARS was 0.88. There were 31,481 five‐test subtests (48% of possible combinations) that had a correlation with total MICARS score of ≥0.90. The strongest correlation of an unconstrained five‐test subset was 0.963. The clinically constrained subtest validated by factor analysis, BARS, had a correlation with MICARS‐minus‐BARS of 0.952. Cronbach alpha for BARS and SARA was 0.90 and 0.92 respectively; and inter‐rater reliability (intraclass correlation coefficient) was 0.91 and 0.93 respectively. BARS is valid, reliable, and sufficiently fast and accurate for clinical purposes. © 2009 Movement Disorder Society     

Reliability and validity of the Youth Self-Report, Bangladesh version

The objectives of this study were to develop the Bangladesh version of the Youth Self-Report (YSR), and assess its reliability and validity in an adolescent population in Dhaka city, Bangladesh. One-hundred-and-eighty-seven boys (mean age: 14.6 years, SD: 2.1) and 137 girls (mean age: 15.2 years, SD: 2.0) from residential areas, and 27 boys (mean age: 17.0, SD: 1.4) and 14 girls (mean age: 15.4, SD: 2.8) from a psychiatric hospital, all within the range of 11-18 years, were interviewed using a questionnaire that consisted of the Bangla translation of the YSR and other questions. Thirty-eight randomly selected adolescents were administered the same questionnaire one week after the first administration. All the core scores and most of the subscales showed high internal consistency other than small item-number subscales, and satisfactory test-retest reliability. Good discriminant validity was shown for most of the scale scores. This study showed that the Bangla translation of the YSR had sufficient reliability and validity for use in Bangladesh. The cutoff scores of the scales were higher for the Bangla version than for the original English version, and further studies exploring this point would be an asset.     

Feasibility, test-retest reliability, and interrater reliability of the Modified Ashworth Scale and Modified Tardieu Scale in persons with profound intellectual and multiple disabilities

Caregivers of persons with profound intellectual and multiple disabilities (PIMD) often describe the quality of the daily movements of these persons in terms of flexibility or stiffness. Objective outcome measures for flexibility and stiffness are muscle tone or level of spasticity. Two instruments used to grade muscle tone and spasticity are the Modified Ashworth Scale (MAS) and the Modified Tardieu Scale (MTS). To date, however, no research has been performed to determine the psychometric properties of the MAS and MTS in persons with PIMD. Therefore, the purpose of this study was to determine the feasibility, test-retest reliability, and interrater reliability of the MAS and MTS in persons with PIMD. We assessed 35 participants on the MAS and MTS twice, first for the test and second a week later for the retest. Two observers performed the measurements. Feasibility was assessed based on the percentage of successful measurements. Test-retest and interrater reliability were determined by using the Wilcoxon signed rank test, intraclass correlation coefficients (ICC), Spearman's correlation, and either limits of agreement (LOA) or quadratically weighted kappa. The feasibility of the measurements was good, because an acceptable percentage of successful measurements were performed. MAS measurements had substantial to almost perfect quadratically weighted kappa (>0.8) and an acceptable ICC (>0.8) for both inter- and intrarater reliability. However, MTS measurements had insufficient ICCs, Spearman's correlations, and LOAs for both inter- and interrater reliability. Our data indicated that the feasibility of the MAS and MTS for measuring muscle tone in persons with PIMD was good. The MAS had sufficient test-retest and interrater reliability; however, the MTS had an insufficient test-retest and interrater reliability in persons with PIMD. Thus, the MAS may be a good method for evaluating the quality of daily movements in persons with PIMD. Providing test administrators with training and clear instructions will improve test reliability.     

Smiling faces and other rewards: using the Behavioral Assessment and Research System (BARS) with unique populations

Concern about the exposure of children and adolescents to occupational and environmental hazards has increased, and so has the need to develop testing methods that can adequately assess the effects of exposure in children. A computerized testing system, the Behavioral Assessment and Research System (BARS), was successfully modified to test both younger populations and populations which do not speak English, the original language of the battery. These adaptations were modifications of the existing features of the BARS system which was designed to assess the broadest possible audience: simple language instructions broken down into basic concepts (step-by-step training with competency testing at each instruction step); a token dispenser along with a "smiling face" stimulus that reinforced appropriate performance; and adjustable parameter settings (e.g., number of trials, difficulty). Data from four groups demonstrate the feasibility of using BARS with children as young as preschool age and for non-English speaking children.     

Psychometric properties of the Spence Children's Anxiety Scale (SCAS) in Mainland Chinese children and adolescents

This study examined the psychometric properties of the Spence Children's Anxiety Scale (SCAS) in a Mainland Chinese community sample. The 38-item Chinese version of SCAS was administered to 1878 children and adolescents. Multiple group confirmatory factor analyses supported a common 6-factor model of SCAS for children and adolescents, and for boys and girls. The internal consistency and test-retest reliability of SCAS were satisfactory. Convergent and divergent validity of SCAS were supported by significant correlations with a measure of anxiety to a greater extent than with a measure of depression. Adolescents reported higher anxiety levels than children, and girls reported higher anxiety levels than boys. Compared to other studies, anxiety symptoms of Chinese children were found at a moderate level, but anxiety symptoms of Chinese adolescents were found at a high level. Our findings suggest that the SCAS is suitable for assessing anxiety symptoms in Mainland Chinese children and adolescents.     

Development of the Behavioral Assessment and Research System (BARS) to detect and characterize neurotoxicity in humans

The Behavioral Assessment and Research System (BARS) is a computer-based testing system designed to assess neurobehavioral function in humans. It was developed to provide a series or battery of neurobehavioral tests optimized for the detection of neurotoxicity in non-mainstream human populations, specifically people with limited education or literacy. Key to meeting this goal were simply-stated instructions divided into an elemental series of steps, a 9BUTTON response unit to replace the computer keyboard for responding, and spoken instructions. Modifications all underwent serial testing in target populations to successively hone the changes to be more effective. A similar process was followed when developing adjustable parameters, test reliability assessments, and when implementing these tests with populations from different cultural groups and children. The principles and experiences that guided the development of BARS should inform the development of future testing systems to ensure that the new tests can be used with non-mainstream populations, which may be increasingly subject to neurotoxic exposures.     

Test-retest reliability and practice effects of a rapid screen of mild traumatic brain injury

Test-retest reliabilities and practice effects of measures from the Rapid Screen of Concussion (RSC), in addition to the Digit Symbol Substitution Test (Digit Symbol), were examined. Twenty five male participants were tested three times; each testing session scheduled a week apart. The test-retest reliability estimates for most measures were reasonably good, ranging from .79 to .97. An exception was the delayed word recall test, which has had a reliability estimate of .66 for the first retest, and .59 for the second retest. Practice effects were evident from Times 1 to 2 on the sentence comprehension and delayed recall subtests of the RSC, Digit Symbol and a composite score. There was also a practice effect of the same magnitude found from Time 2 to Time 3 on Digit Symbol, delayed recall and the composite score. Statistics on measures for both the first and second retest intervals, with associated practice effects, are presented to enable the calculation of reliable change indices (RCI). The RCI may be used to assess any improvement in cognitive functioning after mild Traumatic Brain Injury.     

A clinical tool to measure trunk control in children with cerebral palsy: the Trunk Control Measurement Scale

In this study the psychometric properties of the Trunk Control Measurement Scale (TCMS) in children with cerebral palsy (CP) were examined. Twenty-six children with spastic CP (mean age 11 years 3 months, range 8-15 years; Gross Motor Function Classification System level I n = 11, level II n = 5, level III n = 10) were included in this study. To determine the discriminant ability of the TCMS, 30 typically developing (TD) children (mean age 10 years 6 months, range 8-15 years) were also included. For inter-rater reliability, two testers scored all children simultaneously. To determine test-retest reliability, participants were reassessed on a second test occasion. For construct validity, the Gross Motor Function Measure (GMFM) was administered. Intraclass correlation coefficients (ICC) ranged from 0.91 to 0.99 for inter-rater and test-retest reliability. Kappa and weighted kappa values ranged for all but one item from 0.45 to 1. The standard error of measurement was 2.9% and 3.4%, and the smallest detectable difference for repeated measurements was 8% and 9.43% between raters and test-retest, respectively. Cronbach's alpha coefficients ranged from 0.82 to 0.94. Spearman rank correlation with the GMFM was 0.88 and increasing coefficients were found from dimension B to E. Subscale and total TCMS scores showed significant differences between children with CP and TD children (p < 0.0001). The results support the reliability and validity of the TCMS in children with spastic CP. The scale gives insight into the strengths and weaknesses of the child's trunk performance and therefore can have valuable clinical use.     

Evaluation of three aggression/agitation behaviour rating scales for use on an acute admission and assessment psychogeriatric ward

Background. In recent years many instruments measuring aggressive and agitated behaviours among the elderly in a variety of settings have emerged. Individual instruments have only occasionally been compared with each other. Method. Some psychometric properties of three aggression/agitation scales on an acute assessment and admission psychogeriatric ward were examined. The correlation between the Rating Scale for Aggressive Behaviour in the Elderly (RAGE), the Cohen-Mansfield Agitation Inventory (CMAI) and the Brief Agitation Rating Scale (BARS) and their internal consistencies and test–retest and interrater reliabilites were measured. Results. The RAGE was strongly correlated with the CMAI (rho=+0.73) and the BARS (rho=+0.72). The CMAI was strongly correlated with the CMAI (rho=+0·84). The internal consistency, as measured by Cronbach's alpha, was greater than 0·8 on all three scales. The test–retest and interrater reliability correlations were 0·75 or greater for all three scales (except the BARS interrater reliability correlation of 0·6). Conclusions. All three scales have good psychometric properties for use in acute admission and assessment psychogeriatric wards. © 1998 John Wiley & Sons, Ltd.     

Reliability of the Child Behavior Checklist for the assessment of behavioral problems of children and youth with mild mental retardation

The assessment of psychopathology in persons with mental retardation requires reliable and valid instruments. In the present study, the reliability of the Child Behavior Checklist was determined, using data of 42 children and youth with mild mental retardation, with ages from 10 to 18 years. Kappa coefficients and intra-class correlations were computed to determine the reliability at item level and syndrome level. At item level, mean kappa's for inter-rater and test-retest reliability were 0.267 and 0.52, respectively. At syndrome level, mean intra-class correlations for inter-rater and test-retest reliability were 0.493 and 0.775, respectively. These results suggest that the Child Behavior Checklist may not always represent a reliable checklist for the assessment of psychopathology among children and youth with mild mental retardation.     

Reliability and validity of the screen for child anxiety related emotional disorders (SCARED) in Chinese children

The present study examined the psychometric properties of the screen for child anxiety related emotional disorders (SCARED) in a large community sample of Chinese children. The 41-item version of the SCARED was administered to 1559 primary and junior high school students (774 boys and 785 girls, mean age 11.8+/-2.11) in 12 Chinese cities. The SCARED demonstrated moderate to high internal consistency (alpha=0.43-0.89) and test-retest reliability (intraclass correlation coefficients=0.46-0.77 over 2 weeks and 0.24-0.67 over 12 weeks), moderate parent-child correlation (r=0.49-0.59) and good discriminant validity (between anxiety and non-anxiety disorders). The SCARED total score was significantly correlated with the internalizing factor of the child behavior checklist (0.41). Factor analyses revealed the same five-factor structure as the original SCARED. These findings support that the SCARED is a reliable and valid anxiety screening instrument in Chinese children.     

Construct validity and responsiveness of Movakic: An instrument for the evaluation of motor abilities in children with severe multiple disabilities

Movakic is a newly developed instrument for measurement of motor abilities in children with severe multiple disabilities, with a satisfactory feasibility and content validity and good inter-observer and test-retest reliability.The objective of this study was to investigate its construct validity and responsiveness to change.Sixty children with severe multiple disabilities (mean age 7.7 years, range 2–16) were measured using Movakic six times during 18 months. Construct validity was assessed by correlating Movakic scores with expert judgment. In order to assess responsiveness, scores during 3-months intervals were compared (mean score-changes and intraclass correlations) during which some children experienced meaningful events influencing motor abilities and during which others experienced no such event.Forty-five percent of children had a lower cognitive development level than 6-month, 52% had Gross Motor Function Classification System level V and 37% had level IV. For 27 children all measurements were completed, six children dropped out. Construct validity was good (r = 0.50–0.71). Responsiveness was demonstrated by significantly larger score changes after events than when such events did not occur.Movakic is a valid instrument for measuring motor abilities in children with severe multiple disabilities. Results suggest responsiveness to change in motor abilities after meaningful events.     

Reliability and feasibility of the six minute walk test in subjects with myotonic dystrophy

The objective was to describe test-retest reliability and feasibility of the six minute walk test in adult subjects with myotonic dystrophy type 1. Twelve subjects (28-68 years, mean 44) performed three six minute walk tests on two occasions, one week apart. Relative reliability was high (ICC(2.1)=0.99) and absolute reliability values were low (standard error of measurement 12 m, repeatability 33 m). Feasibility was investigated in a sample of 64 subjects (19-70 years, mean 43). Fifty-two subjects were able to perform two tests on the same day. Subjects with severe proximal weakness had difficulties performing repeated tests. A practice trial followed by a second test on the same day can be recommended for most subjects, and the best test should be used for evaluations. In conclusion, even though the study sample was small, the present study indicates that the six minute walk test is reliable and feasible in subjects with myotonic dystrophy type 1.     

Reliability of the Melbourne Assessment of Unilateral Upper Limb Function

This study examines the reliability of the Melbourne Assessment of Unilateral Upper Limb Function: a quantitative test of quality of movement in children with neurological impairment. The assessment was administered to 20 children aged from 5 to 16 years (mean age 9 years 10 months, SD 2 years 10 months) who had various types and degrees of cerebral palsy (CP). The performances of the 20 children during assessment were videotaped for subsequent scoring by 15 occupational therapists. Scores were analyzed for internal consistency of test items, inter- and intrarater reliability of scorings of the same videotapes, and test-retest reliability using repeat videotaping. Results revealed very high internal consistency of test items (alpha=0.96), moderate to high agreement both within and between raters for all test items (intraclass correlations of at least 0.7) apart from item 16 (hand to mouth and down), and high interrater reliability (0.95) and intrarater reliability (0.97) for total test scores. Test-retest results revealed moderate to high intrarater reliability for item totals (mean of 0.83 and 0.79) for each rater and high reliability for test totals (0.98 and 0.97). These findings indicate that the Melbourne Assessment of Unilateral Upper Limb Function is a reliable tool for measuring the quality of unilateral upper-limb movement in children with CP.     

Validation of the behavioural activity rating scale (BARS): a novel measure of activity in agitated patients.

We report psychometric results of the Behavioural Activity Rating Scale (BARS) using data from three Phase III clinical trials of intramuscular ziprasidone in acutely agitated patients with psychosis (Studies 1 and 2) or in stable psychotic patients (Study 3). Convergent validity and divergent validity were assessed with baseline data from Studies 1 and 2 in subjects with acute agitation. To investigate convergent validity, we sought Pearson and Spearman correlation of BARS scores with scores on the Clinical Global Impression of Severity (CGI-S) Scale and a predefined cluster of agitation-related items from the Positive and Negative Syndrome Scale (PANSS). For divergent validity, we sought Pearson and Spearman correlation between BARS scores and a predefined cluster of PANSS items measuring negative symptoms. Discriminant validity was investigated with the help of subjects with moderate psychopathology (Study 3). Wilcoxon rank-sum and two-sample t tests determined whether mean (or median) BARS scores differed between subjects with acute agitation (Studies 1 and 2) and moderate psychopathology (Study 3). Responsiveness to treatment effect and rater reliability were also evaluated. In Study 2, Pearson correlation coefficients of BARS scores with PANSS agitation items and CGI-S were moderate (convergent validity) and statistically significant (P<0.005). The correlation between BARS scores and PANSS negative component scores was low (divergent validity). Treatment effect size was larger for BARS than for PANSS agitation items and CGI-S (responsive to treatment differences). Virtually perfect inter- and intra-rater reliability was achieved. Study 1 produced similar results. BARS showed psychometrically valid properties for measurement of behavioral activity in acutely agitated patients with psychosis.     

Validity and reproducibility of a new diagnostic motor performance test in children with suspected myopathy

To spare more children from painful muscle biopsy, a new non-invasive diagnostic motor performance test is undergoing development. Fifteen functional items were used to measure muscle strength and muscle endurance in 68 patients (47 males, 21 females; mean age 7y 8mo, SD 2y 2mo; range 4 to 11y), who had been referred to our specialist centre in the past 3 years on suspicion of myopathy. All the patients had undergone muscle biopsy. To correct the patients' outcomes for age, sex, and body size, regression prediction equations were obtained from a stratified random sample of 64 normally developing primary-school children aged 4 to 11 years (32 males, 32 females; mean age 8y 1mo, SD 2y 4mo). Feasibility was evaluated on the basis of five criteria. Validity was assessed using logistic regression analysis, receiver operating characteristic analysis, and sensitivity and specificity at a specifically chosen cut-off point. Reproducibility was evaluated by test-retest reliability in a stratified random sample of 40 patients who returned for re-measurements using the intraclass correlation coefficient. Seven items satisfied all five feasibility criteria, had high diagnostic power, and high test-retest reliability. The motor performance test can improve diagnostic procedure in children suspected of having myopathy.