The future of outcomes measurement: item banking,tailored short-forms,and computerized adaptive assessment

The use of item banks and computerized adaptive testing (CAT) begins with clear definitions of important outcomes, and references those definitions to specific questions gathered into large and well-studied pools, or “banks” of items. Items can be selected from the bank to form customized short scales, or can be administered in a sequence and length determined by a computer programmed for precision and clinical relevance. Although far from perfect, such item banks can form a common definition and understanding of human symptoms and functional problems such as fatigue, pain, depression, mobility, social function, sensory function, and many other health concepts that we can only measure by asking people directly. The support of the National Institutes of Health (NIH), as witnessed by its cooperative agreement with measurement experts through the NIH Roadmap Initiative known as PROMIS (www.nihpromis.org), is a big step in that direction. Our approach to item banking and CAT is practical; as focused on application as it is on science or theory. From a practical perspective, we frequently must decide whether to re-write and retest an item, add more items to fill gaps (often at the ceiling of the measure), re-test a bank after some modifications, or split up a bank into units that are more unidimensional, yet less clinically relevant or complete. These decisions are not easy, and yet they are rarely unforgiving. We encourage people to build practical tools that are capable of producing multiple short form measures and CAT administrations from common banks, and to further our understanding of these banks with various clinical populations and ages, so that with time the scores that emerge from these many activities begin to have not only a common metric and range, but a shared meaning and understanding across users. In this paper, we provide an overview of item banking and CAT, discuss our approach to item banking and its byproducts, describe testing options, discuss an example of CAT for fatigue, and discuss models for long term sustainability of an entity such as PROMIS. Some barriers to success include limitations in the methods themselves, controversies and disagreements across approaches, and end-user reluctance to move away from the familiar.     

Setting standards for severity of common symptoms in oncology using the PROMIS item banks and expert judgment

Background

Although the use of patient-reported outcome measures (PROs) has increased markedly, clinical interpretation of scores remains lacking. We developed a method to identify clinical severity thresholds for pain, fatigue, depression, and anxiety in people with cancer.

Methods

Using available Patient-Reported Outcomes Measurement Information System (PROMIS) item bank response data collected on 840 cancer patients, symptom vignettes across a range of symptom severity were developed and placed on index cards. Cards represented symptom severity at five-point intervals differences on the T score metric [mean = 50; standard deviation (SD) = 10]. Symptom vignettes for each symptom were anchored on these standardized scores at 0.5 SD increments across the full range of severity. Clinical experts, blind to the PROMIS score associated with each vignette, rank-ordered the vignettes by severity, then arrived at consensus regarding which two vignettes were at the upper and lower boundaries of normal and mildly symptomatic for each symptom. The procedure was repeated to identify cut scores separating mildly from moderately symptomatic, and moderately from severely symptomatic scores. Clinician severity rankings were then compared to the T scores upon which the vignettes were based.

Results

For each of the targeted PROs, the severity rankings reached by clinician consensus perfectly matched the numerical rankings of their associated T scores. Across all symptoms, the thresholds (cut scores) identified to differentiate normal from mildly symptomatic were near a T score of 50. Cut scores differentiating mildly from moderately symptomatic were at or near 60, and those separating moderately from severely symptomatic were at or near 70.

Conclusions

The study results provide empirically generated PROMIS T score thresholds that differentiate levels of symptom severity for pain interference, fatigue, anxiety, and depression. The convergence of clinical judgment with self-reported patient severity scores supports the validity of this methodology to derive clinically relevant symptom severity levels for PROMIS symptom measures in other settings.     

Psychometric properties of a brief,clinically relevant measure of pain in patients with hepatocellular carcinoma

Purpose

Due to diagnosis at advanced stages, comorbidities, and the impact of treatment, patients with hepatocellular carcinoma (HCC) may experience pain. The purpose of this study was to evaluate the psychometric properties of a brief, clinically relevant measure of pain in HCC.

Methods

We conducted a secondary data analysis from four longitudinal studies of patients with HCC (total n = 304). All patients completed the FACT-Hepatobiliary (FACT-Hep) questionnaire, and 49 patients completed the Brief Pain Inventory (BPI) Interference scale. We conducted confirmatory factor analysis (CFA), Rasch modeling, and correlational analysis to assess the psychometrics of the three items on the FACT-Hep that assess HCC-relevant pain scale.

Results

Patients had an average age of 63.5 (±12.2) and were mostly male (76 %). The mean three-item pain subscale score was 8.5 ± 3.0. Seventy-four (24.3 %) patients reported no pain (score = 12). Results of a one-factor CFA supported unidimensionality of the items, and all items fit the Rasch model. An item-person map demonstrated that the three items covered all patients with non-extreme scores. Pain scores were significantly associated with baseline general health-related quality of life (FACT-General, r = 0.60, p < 0.001) and pain interference (BPI, r = ?0.63, p < 0.001).

Conclusions

The three FACT-Hep pain items are unidimensional, cover the range of pain experienced by most patients with HCC, and demonstrate convergent validity. This pain subscale is, if future research demonstrates its sensitivity to change, potentially useful for HCC clinical trials.     

Building a Measure of Fatigue: The Functional Assessment of Chronic Illness Therapy Fatigue Scale

The shared goal of all clinical disciplines is to optimize the well-being of people who become patients and find themselves diminished by illness and recovery. This goal relies on sound tools to evaluate both real and perceived deficits in a way that can be used for a particular patient over time and also across medical disciplines and patient populations. Fatigue is a critical and notoriously subjective aspect of many illnesses. Although the soundness of research is often correlated with the objectivity of data, certain clinical measures must, by definition, be patient centered, with all the complexities and challenges of patient-reported evaluations. Measurement of fatigue has been an important and evolving component of symptom management in the field of oncology. The Functional Assessment of Chronic Illness Therapy Fatigue Scale is a self-administered fatigue-assessment tool that has found wide application across diverse medical fields and that has demonstrated validity and utility across a broad range of populations. The Functional Assessment of Chronic Illness Therapy Fatigue Scale has become one in a repository of tools in the item banks that are accumulating under the auspices of The Patient-Reported Outcomes Measurement Information System, a National Institutes of Health initiative to deploy the most clinically relevant and technologically agile tools that we have to advance research in medicine and patient care. As much as with any other discipline, physical medicine and rehabilitation stands to gain from the collective knowledge and creative horizons in the assessment and treatment of fatigue.     

Self-reported fatigue: one dimension or more? Lessons from the Functional Assessment of Chronic Illness Therapy��Fatigue (FACIT-F) questionnaire

Across two general population (total n?=?1,878) and two cancer (total n?=?3,140) samples, we evaluated the dimensionality of self-reported fatigue as measured by the Functional Assessment of Chronic Illness Therapy??Fatigue (FACIT-F) instrument. After evaluating dimensionality of the FACIT-F, we compared the conceptually distinct fatigue experience versus fatigue impact scores in each sample. Confirmatory factor analysis of the 13-item scale showed very good fit to a single dimension (??unidimensional??) model for each sample (comparative fit index range?=?0.92?C0.97). Using a bifactor model to compare the loading of each item with the general fatigue factor versus the identified sub-domain (experience or impact), we found the item-general loading to be higher than that of the item-sub-domain factor in 52 of 52 comparisons (13 items; four samples). When scored separately, experience and impact scores were correlated highly (range?=?0.80?C0.88), yet their difference relative to one another was significant (p?<?0.001). Consistently across samples, experience scores were systematically higher (more endorsement) than impact scores, by a margin of 0.21?C0.46?SD units. This suggests that the fatigue experience and the impact of fatigue upon function are reported along a single dimensional continuum, but that experience is more likely than impact upon function to be endorsed at lower levels of fatigue. Fatigue as an outcome or trial endpoint can be expressed as a single number, and the experience of the symptom is more likely to be endorsed at mild levels of fatigue, presumably before the symptom exerts an adverse impact upon function.     

Procedures for the analysis of differential item functioning (DIF) for small sample sizes

An item with differential item functioning (DIF) displays different statistical properties, conditional on a matching variable. The presence of DIF in measures can invalidate the conclusions of medical outcome studies. Numerous approaches have been developed to examine DIF in many areas, including education and health-related quality of life. There is little consensus in the research community regarding selection of one best method, and most methods require large sample sizes. This article describes some approaches to examine DIF with small samples (e.g., less than 200).     

Development of the Childhood Atopic Dermatitis Impact Scale: initial validation of a quality-of-life measure for young children with atopic dermatitis and their families

To measure the effects of atopic dermatitis (AD) on the quality of life of affected young children and their families, we developed a prototype 62-item instrument, the Childhood Atopic Dermatitis Impact Scale (CADIS). The prototype CADIS was developed from a comprehensive conceptual framework based on data from parents and clinicians. The instrument had eight subscales (four each for child and parent): physical health, emotional health, physical functioning, and social functioning. The goal of this work was to test the validity of and to refine the prototype of CADIS. Two hundred seventy parents of children under the age of 6 y with AD responded to the instrument. Content validity was demonstrated by expert and parent reviews of the drafted and refined instrument, and by analyzing parents' responses to open-ended questions about their children's skin disease. Construct validity was assessed in exploratory factor analyses which supported a refinement in the conceptual framework to consist of two dimensions with five domains: child dimensions (symptoms and activity limitation/behavior), and parent dimensions (family/social function, sleep, and emotions). Seventeen items were eliminated, yielding a 45-item refined version of CADIS (score 0-180) with evidence of content and construct validity and suggested use in clinical research.     

A cross-sectional study of carnitine deficiency and fatigue in pediatric cancer patients

Purpose

Carnitine deficiency has been found in cancer patients and has been associated with fatigue. This study aimed to explore the prevalence of carnitine deficiency in pediatric cancer patients and its relationship with fatigue and other potential contributing factors.

Methods

Children with cancer or Langerhans cell histiocytosis who were receiving treatment or had completed therapy were eligible. Patients completed the Pediatric Functional Assessment of Chronic Illness-Fatigue, the Pediatric Quality of Life Inventory Multidimensional Fatigue Scale, a numeric fatigue rating, and had carnitine levels obtained. Carnitine deficiency was defined as a total and/or free carnitine level less than normal for age or an acylcarnitine value higher than normal for age.

Results

Data from 142 children aged 8–17 were analyzed. Twenty-eight of 142 (19.7 %) had decreased total and 42.8 % (12/28) had decreased free carnitine levels. No patients had elevated acylcarnitine levels or elevated ratios. Patients with versus without carnitine deficiency differed by age (p = 0.043), treatment (p = 0.037), duration since last chemotherapy (p = 0.020), and body mass index (p = 0.010), but not fatigue, when all data were analyzed together. Yet, a negative relationship between fatigue and carnitine levels was found on a subgroup (off-therapy; fatigue worse than the norm).

Conclusion

No significant association between fatigue and carnitine level was demonstrated when data from all patients were analyzed together; however, a significant yet unexpected relationship was found for patients who completed therapy and reported elevated fatigue. Given the small sample size, these results should be interpreted with caution. Future studies to explore impact upon excessive carnitine levels are warranted.

     

The impact of assistive technology and environmental interventions on function and living situation status with people who are ageing with developmental disabilities

Purpose : A longitudinal study of 109 people with developmental disabilities, age 35 and older, was done to study the additive impact of mid to later life assistive technology and environmental interventions (AT-EI) on function and living situation status. All subjects were trying to transition out of institutional settings to community settings. Method : Functional status were measured at two times (Time 1 baseline and Time 2 an average of three years post intervention) on 32 functional activities under two conditions: without AT (person only) and with AT (environment adjusted). Rasch analysis was performed to convert ordinal functional scores to equal interval measures, with 95% confidence intervals computed to compare differences in function, with and without AT, across time. Results : Results indicated that over 70% of subjects had better function with AT versus without AT at both time points. Over time, function did not change when rated without AT; however, when rated with AT, 13.6% had better function at Time 2. Subjects living in the community at Time 2 had significantly higher functional scores as compared to subjects in institutions, regardless of AT condition. Additional quantitative and qualitative data on AT-EI use, needs, and barriers and supports to its integration into everyday activities are reported. Conclusions : Results suggest a beneficial impact of later life AT-EI assessment and programming for people who are ageing with developmental disabilities, and qualitatively point to the influence of the social and physical living context upon AT-EI use and relationship to community living decisions long term.