Cross-National Reliability of Clinician-Rated Outcome Measures in Child and Adolescent Mental Health Services

Clinician-rated measures are in extensive use as routine outcome measures in child and adolescent mental health services. We investigated cross-national differences and inter-rater reliability of the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA), the Children’s Global Assessment Scale (CGAS) and the Global Assessment of Psychosocial Disability (GAPD). Thirty clinicians from 5 nations independently rated 20 written vignettes. The national groups afterwards established national consensus ratings. There were no cross-national differences in independent scores, but there were differences in national consensus scores, which were also more severe than independent scores. The ICC for the HoNOSCA total score was 0.84, for the CGAS 0.61 and for the GAPD 0.54. These measures may usefully contribute to cross-national comparison studies.     

Inter-rater Reliability of Clinician-rated Outcome Measures in Child and Adolescent Mental Health Services

This study investigated the inter-rater reliability when 169 out of 171 clinicians working in 10 Norwegian child and adolescent mental health services rated 20 written vignettes using the following outcome measures: Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA), Children's Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD). Three clinicians rated both patients and vignettes. On vignettes the intraclass correlation coefficient (ICC) for the HoNOSCA total score was 0.81 (single scales 0.47-0.96), for the CGAS 0.61 and for the GAPD 0.60. The reliability was not lower on patients. The rater's profession, experience or clinic did not have effect on the scores.     

Routine outcome measurement in a child and adolescent mental health service: an evaluation of HoNOSCA. The Health of the Nation Outcome Scales for Children and Adolescents.

OBJECTIVE: This paper evaluates a range of properties for a clinician-based instrument designed for routine use in a child and adolescent mental health service (CAMHS). METHOD: The Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) is a new outcome measure with great promise. Case vignettes were used to examine interrater reliability. HoNOSCA was implemented for routine outpatient use by multidisciplinary staff with a return rate of 84%. The 305 ratings obtained at assessment were analysed by age, gender and diagnosis. A sample of 145 paired ratings with a 3-month interval were examined for the measurement of change over time. RESULTS: Interrater reliability of the total score indicates moderate reliability if absolute scores are used and good reliability if the total score is used for relative comparisons. Most scales have good to very good reliability. The scales discriminated between age and gender in the expected way. HoNOSCA correlated with clinicians' views of change and was sensitive to change over a 3-month period. The total score seemed a proxy for severity. CONCLUSION: Routine outcome instruments must be explored in settings where they will be used and with realistic training. HoNOSCA appears to be of value in routine outcome measurement and although questions remain about reliability and validity, the results strongly support further investigation.     

Randomised controlled trial of school-based humanistic counselling for emotional distress in young people: Feasibility study and preliminary indications of efficacy

Background

Clinician-rated measures are used extensively in child and adolescent mental health services (CAMHS). The Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) is a short clinician-rated measure developed for ordinary clinical practice, with increasing use internationally. Several studies have investigated its psychometric properties, but there are few data on its correspondence with other methods, rated by other informants. We compared the HoNOSCA with the well-established Achenbach System of Empirically Based Assessment (ASEBA) questionnaires: the Child Behavior Checklist (CBCL), the Teacher's Report Form (TRF), and the Youth Self-Report (YSR).

Methods

Data on 153 patients aged 6-17 years at seven outpatient CAMHS clinics in Norway were analysed. Clinicians completed the HoNOSCA, whereas parents, teachers, and adolescents filled in the ASEBA forms. HoNOSCA total score and nine of its scales were compared with similar ASEBA scales. With a multiple regression model, we investigated how the ASEBA ratings predicted the clinician-rated HoNOSCA and whether the different informants' scores made any unique contribution to the prediction of the HoNOSCA scales.

Results

We found moderate correlations between the total problems rated by the clinicians (HoNOSCA) and by the other informants (ASEBA) and good correspondence between eight of the nine HoNOSCA scales and the similar ASEBA scales. The exception was HoNOSCA scale 8 psychosomatic symptoms compared with the ASEBA somatic problems scale. In the regression analyses, the CBCL and TRF total problems scores together explained 27% of the variance in the HoNOSCA total scores (23% for the age group 11-17 years, also including the YSR). The CBCL provided unique information for the prediction of the HoNOSCA total score, HoNOSCA scale 1 aggressive behaviour, HoNOSCA scale 2 overactivity or attention problems, HoNOSCA scale 9 emotional symptoms, and HoNOSCA scale 10 peer problems; the TRF for all these except HoNOSCA scale 9 emotional symptoms; and the YSR for HoNOSCA scale 9 emotional symptoms only.

Conclusion

This study supports the concurrent validity of the HoNOSCA. It also demonstrates that parents, teachers and adolescents all contribute unique information in relation to the clinician-rated HoNOSCA, indicating that the HoNOSCA ratings reflect unique perspectives from multiple informants.     

The Health of the Nation Outcome Scales for Children and Adolescents in an adolescent in-patient sample

OBJECtive: The primary aims of the study were to examine the reliability and validity of the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) in a sample of adolescents requiring medium to long-term in-patient psychiatric treatment and to examine the association between HoNOSCA scores and age, gender and length of treatment. METHODS: A multidisciplinary team completed the HoNOSCA for 51 adolescent patients at intake and at 3- and 6-months following admission to the unit. RESULTS: The study provided support for the test-retest reliability, concurrent and convergent validity, but not the internal reliability, of the HoNOSCA. Total HoNOSCA scores at intake were similar to those found in adolescent outpatient samples, although there were some differences at the level of individual items. Similarly, while the total HoNOSCA score showed some sensitivity to change, using the total HoNOSCA score obscured important changes in specific domains of functioning over the course of admission. CONCLUSION: The HoNOSCA was found to be a valid measure of global functioning at intake, thereby supporting its use in an adolescent psychiatric unit. However, focusing on individual items, rather than total score, appears more useful in evaluating the impact of inpatient psychiatric treatment on adolescents.     

IQ as a moderator of outcome in severity of children's mental health status after treatment in outpatient clinics

ABSTRACT: BACKGROUND: Psychotherapy is an effective treatment for mental health disorders, but even with the most efficacious treatment, many patients do not experience improvement. Moderator analysis can identify the conditions under which treatment is effective or whether there are factors that can attenuate the effects of treatment. METHODS: In this study, linear mixed model analysis was used to examine whether the Full Scale IQ (FSIQ), Performance IQ (PIQ) and Verbal IQ (VIQ) on the Wechsler Intelligence Scale for Children - Third Edition, moderated outcomes in general functioning and symptom load. A total of 132 patients treated at three outpatient child and adolescent mental health services (CAMHS) were assessed at three different time points. The Children's Global Assessment Scale (CGAS) and the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) were used to measure the severity of impairments in general functioning and symptom load. IQ was assessed at the start of treatment. RESULTS: Moderator analysis revealed that the FSIQ x time interaction predicted changes in CGAS scores (p < .01), and that the PIQ x time interaction predicted changes in HoNOSCA scores(p < .05). The slopes and intercepts in HoNOSCA scores covaried negatively and significantly (p < .05). The same pattern was not detected for the CGAS scores (p = .08). CONCLUSIONS: FISQ and PIQ moderated change in general functioning and symptom load, respectively. This implies that patients with higher IQ scores had a steeper improvement slope than those with lower scores. The patients with the highest initial symptom loads showed the greatest improvement, this pattern was not found in the improvement of general functioning.     

Translation and standardization of the HoNOSCA (Health of the Nation Outcome Scales for Children and Adolescents) scale in a Greek sample

The Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) is a brief measure of outcomes. Evidence for the effectiveness of patient treatments conducted in mental health services is limited in Greece. Thus, in an attempt to employ an easily administered and valid outcome measure, to assess outcomes in clinical practice for children and adolescents, but also to be able to compare the results across countries, the HoNOSCA was the measure of choice. In this study we have translated and validated it in a Greek sample. We have investigated the inter-rater reliability, intraclass correlation, concurrent validity, and clinical change across time, of the HoNOSCA Greek translation. The results show that the Greek translation of HoNOSCA is a reliable and valid instrument. It can be used for clinical, managerial, research and audit purposes, but, most importantly, to facilitate multinational clinical research and comparison of data with other countries.     

Evaluation of a family support service: short-term outcome

The aim of this study was to evaluate the short-term impact of two types of family support services (FSS) for children with behavioural problems (N = 51 and 49 respectively). These were compared with children (N = 40) matched for presenting problems and referred to specialist CAMHS in a locality without FSS. Pre and postintervention measures were the Health of the Nation Outcome Scales (HoNOSCA), the Strengths and Difficulties Questionnaire (SDQ), and a service utilization checklist. Although all three groups were associated with decrease of child behavioural scores, the two family support services provided an earlier response and resulted in significantly higher reduction of HoNOSCA and SDQ scores. The two FSS types implemented different 'levels' of interventions, as FSS-B resulted in a more favourable short-term outcome and appeared to operate more closely to specialist services, while FSS-A appeared to adopt a more preventive role. Family support services have an important role within a comprehensive child mental health service, but the specificity of their interventions for different target client groups requires further investigation.     

Applicability of the Clinical Global Impressions-Aggression (CGI-A) scale for use in file audit trials

The aim of this study was to evaluate the applicability and validity of a clinical global impressions-aggression (CGI-A) scale in retrospective trials. CGI-A scores were derived from clinical records, BPRS scores were assessed prospectively in 101 patients with first-episode psychosis. CGI-A showed highly significant correlations to the brief psychiatric rating scale (BPRS) excited component (BPRS-EC; r=0.32; p=0.001) and BPRS resistance (r=0.25; p=0.011), to aggression at admission (r=0.70; p<0.001) and to involuntary treatment (r=0.55; p<0.001). The CGI-A retains high construct and face validity in retrospective trials.     

Portuguese version of the Delirium Rating Scale-Revised-98: reliability and validity

OBJECTIVE: To assess the validity and the reliability of the Portuguese version of the Delirium Rating Scale-Revised-98 (DRS-R-98). METHODS: The scale was translated into Portuguese and back-translated into English. After assessing its face validity, five diagnostic groups (n=64; delirium, depression, dementia, schizophrenia and others) were evaluated by two independent researchers blinded to the diagnosis. Diagnosis and severity of delirium as measured by the DRS-R-98 were compared to clinical diagnosis, Mini-Mental State Exam, Confusion Assessment Method, and Clinical Global Impressions scale (CGI). RESULTS: Mean and median DRS-R-98 total scores significantly distinguished delirium from the other groups (p<0.001). Inter-rater reliability (ICC between 0.9 and 1) and internal consistency (alpha=0.91) were very high. DRS-R-98 severity scores correlated highly with the CGI. Mean DRS-R-98 severity scores during delirium differed significantly (p<0.01) from the post-treatment values. The area under the curve established by ROC analysis was 0.99 and using the cut-off value of 20 the scale showed sensitivity and specificity of 92.6% and 94.6%, respectively. CONCLUSION: The Portuguese version of the DRS-R-98 is a valid and reliable measure of delirium that distinguishes delirium from other disorders and is sensitive to change in delirium severity, which may be of great value for longitudinal studies.     

Performance of a shortened Scale for Assessment of Positive Symptoms for Parkinson's disease psychosis

BackgroundParkinson's disease psychosis is a frequent and serious complication of advanced disease, but few disease-specific outcome measures exist.MethodsUsing baseline scores from 4 clinical trials, we identified relevant items that assessed Parkinson's disease psychosis to create a shortened version of the Scale for Assessment of Positive Symptoms. We then analyzed the validity and treatment sensitivity of the shortened scale. Principal component analyses evaluated the underlying structure. Scores were compared across age, gender, trial, cognition, and country of origin. Sensitivity to change was assessed by comparing change in psychosis scores to the clinical global impression of improvement score, and effect sizes were calculated to evaluate treatment response.ResultsNine items were selected based on face-validity and symptom frequency. Principal component analysis yielded a 4-factor structure and identified delusions and visual, auditory, and somatic hallucinations as distinct constructs. Baseline total scores were similar across study, gender, region, and age group. The clinically meaningful change in the shortened scale, defined as a 1-unit change in clinical global impression, was 2.33 points, and the effect size was ?0.722. The change in scores did not significantly differ between those with cognitive impairment and those without.ConclusionsThe shortened Scale for Assessment of Positive Symptoms for Parkinson's disease retains the reliability, sensitivity to change, and effect size of the larger scale while reducing administration time and, more importantly, score variability. The scale is an effective outcome measure for use in clinical trials.     

Does the HoNOS 65+ meet the criteria for a clinical outcome indicator for mental health services for older people?

BACKGROUND: A clinical indicator should demonstrate clinically meaningful change, be relevant, allow comparisons between services, be acceptable to clinicians, and have acceptable validity, reliability and sensitivity to change. The HoNOS 65+ has been suggested as a clinical outcome indicator. The sensitivity to change of the HoNOS 65+ is not known. METHODS: This is a prospective study using routine clinical data. A pilot cohort (n = 42) was used to measure the concurrent validity of the HoNOS 65+ with the Mini-Mental State Examination (MMSE), Geriatric Depression Scale (GDS-15) and Brief Agitation Rating Scale (BARS). The main cohort of 245 consecutive referrals to a community mental health service for older adults was used to assess sensitivity to change against the CIBIC+. RESULTS: The HoNOS 65+ was acceptable to case managers, most HoNOS 65+ items had excellent interrater reliability and the HoNOS 65+ had good concurrent validity. Changes in the HoNOS 65+ scores between assessment and discharge had a moderate, but significant correlation with CIBIC+ scores. CONCLUSION: The HoNOS 65+ meets the criteria for a clinical outcome indicator for community mental health services for older people. The HoNOS 65+ is sensitive to change.     

Using the GAF as a national mental health outcome measure in the Department of Veterans Affairs

OBJECTIVE: Data from the Veterans Health Administration (VHA) were used to evaluate the strengths and weaknesses of the Global Assessment of Functioning (GAF), a single-item mental health status measure, as an outcome measure for large mental health care systems. METHODS: The sample consisted of VHA mental health patients who had at least two GAF scores 45 days apart in 2002 (N=283,754). First, to evaluate the discriminant validity of the GAF change measures, the authors examined the association of these measures with sociodemographic and clinical characteristics. Facility-level risk-adjusted measures of GAF change were then created in three different clinical samples at more than 130 VHA medical centers, adjusting for patients' sociodemographic characteristics and diagnoses. The internal consistency of the scale created by using these items and their consistency across medical centers over time was evaluated. RESULTS: The analysis supported the discriminant validity of the GAF-derived measures. As expected, veterans who had a diagnosis of schizophrenia or Alzheimer's disease or who had service-connected disability ratings above 50 percent had lower baseline GAF scores and showed less improvement. The overall GAF performance measure had a high level of internal consistency (a standardized alpha of .85) and was highly consistent across facilities over time. CONCLUSIONS: The results of this study provide preliminary empirical support for cautious use of a GAF-derived scale in monitoring changes in average facility-level outcomes over time. However, because of the potential for gaming of the measures and uncontrolled variation in the scale's administration across facilities, the scale should not be used to compare outcomes across facilities.     

The neurobehavioural rating scale-revised: sensitivity and validity in closed head injury assessment

OBJECTIVES: To investigate the factor structure and psychometric properties of the neurobehavioural rating scale-revised (NRS-R) and to determine its usefulness in clinical trials. METHODS: A consecutive series of patients sustaining severe closed head injury were evacuated to one of 11 large regional North American trauma centres and entered into a randomised, phase III, multicentre clinical trial investigating the therapeutic use of moderate hypothermia. Acute care personnel were blinded to outcome and outcome personnel were blinded to treatment condition. The Glasgow outcome scale (GOS) was the primary outcome measure. Secondary outcome measures included the disability rating scale (DRS) and the NRS-R. RESULTS: Exploratory factor analysis of NRS-R data collected at 6 months after injury (n=210) resulted in a five factor model including: (1) executive/cognition, (2) positive symptoms, (3) negative symptoms, (4) mood/affect, and (5) oral/motor. These factors showed acceptable internal consistency (0.62 to 0.88), low to moderate interfactor correlations (0.19 to 0.61), and discriminated well between GOS defined groups. Factor validity was demonstrated by significant correlations with specific neuropsychological domains. Significant change was measured from 3 to 6 months after injury for the total score (sum of all 29 item ratings) and all factor scores except mood/affect and positive symptoms. The total score and all factor scores correlated significantly with concurrent GOS and DRS scores. CONCLUSIONS: The NRS-R is well suited as a secondary outcome measure for clinical trials as its completion rate exceeds that of neuropsychological assessment and it provides important neurobehavioural information complementary to that provided by global outcome and neuropsychological measures.     

Diskussion: Kritiska synpunkter på psykiatrisk variabelforskning: Replik till Karl Erik Törnqvist (Nordisk psykiat. Medlems-blad1958/hefte 1, s. 48)

Abstract

Background: Consumer satisfaction studies with the Child and Adolescent Mental Health Service (CAMHS) have mainly assessed evaluations in a short-term follow-up perspective. Adolescent reports with CAMHS have not been included nationally. Aims: The purposes of this study were to explore adolescent and parental satisfaction with the CAMHS in a 3–4-year follow-up perspective, and to examine the relationships between reported consumer satisfaction and clinical parameters such as reason for adolescent referral, emotional/behavioral symptoms and treatment outcome. Methods: Of 190 adolescent–parent pairs in a sample of CAMHS outpatients, 120 completed a Consumer Satisfaction Questionnaire. Parents assessed adolescent emotional/behavior problems both at baseline and at follow-up by completing the Child Behavior Checklist (CBCL). Correlations were examined between adolescent and parental evaluations. The relationships between service satisfaction and symptom load at baseline and follow-up and treatment outcome at follow-up were explored. Results: Overall, adolescents and parents were satisfied with the services received from the CAMHS. The correlations between adolescent and parent consumer satisfaction ratings were low to moderate. Consumer satisfaction was significantly and negatively correlated with symptom load on the CBCL Total Problems scores at baseline, but not at follow-up. There was no difference in satisfaction levels between those who improved after treatment and those who did not. Conclusions: Given the differences in informant ratings of consumer satisfaction, it is important to include both adolescent and parental perceptions in evaluations of CAMHS services and treatment outcomes. Consumer satisfaction should serve as a supplement to established standardized outcome measures.     

Feasibility and validity of computerized ecological momentary assessment in schizophrenia

BACKGROUND: Computerized Ecological Momentary Assessment (EMAc) techniques permit the assessment of daily life behaviors and experiences. The present investigation examined the feasibility and validity of this assessment methodology in outpatients with schizophrenia. METHODS: Outpatients with schizophrenia or schizoaffective disorder (n = 54) received a battery of standard laboratory clinical and functional outcome measures and then completed electronic questionnaires on a personal digital assistant (PDA) microcomputer 4 times per day for 1 week. RESULTS: Generally good compliance (87%) with EMAc was found, and participants rated their experience with the study positively. The data collected in daily life demonstrated expected patterns across the assessment week and were significantly associated with scores from standard laboratory instruments measuring similar constructs. CONCLUSIONS: EMAc is a feasible and valid approach to data collection in community-dwelling people with schizophrenia, and it may provide important information that is inaccessible via standard clinical and functional outcome measures administered in the laboratory.     

Quality of life: eating disorders

OBJECTIVE: There is a lack of measurements with predictive validity that are specific for quality of life (QOL) in patients with eating disorders. METHOD: A total of 306 eating disorder patients treated as inpatients completed the Quality of Life for Eating Disorders (QOL ED): 109 at both admission and discharge from hospital, 65 at both admission and after 12 months. Patients also completed well-validated measures of eating disorders, psychological dysfunction and general physical and mental QOL. QOL ED consists of 20 self-report questions that provide scores for the domains of behaviour, eating disorder feelings, psychological feelings, effects on daily life, effects on acute medical status and body weight, and a global score. RESULTS: QOL ED domain scores correlated appropriately with previously validated well-known measures of eating disorders, psychological dysfunction, general QOL and behaviour and body weight (p < 0.001). The QOL ED shows high reliability (Cronbach's alpha = 0.93). All scores changed significantly and appropriately during inpatient hospital treatment and between admission and 12 months after discharge from hospital (p < 0.001). The scores differed for anorexia nervosa, bulimia, eating disorder not specified (EDNOS) and no diagnosis. All no diagnosis (recovered) domain and global scores were significantly different from all diagnoses scores (p < 0.001). CONCLUSION: This quick, simple instrument fulfils all potential uses for QOL assessments in the clinical and research settings associated with eating disorders, including outcome.     

The Health of the Nation Outcome Scales for Child & Adolescent Mental Health (HoNOSCA)

The child and adolescent version of the Health of the Nation Outcome Scales (HoNOSCA) represents the first attempt at a routine outcome measure for Child and Adolescent Mental Health Services in the U.K. Extensive field trials suggested that the scales were both acceptable to clinicians from the various disciplines working in this area and also valid and reliable. A growing number of services are now using the scales in audit and research, supported by the national HoNOSCA base that provides training and co-ordinates further developments.