Feasibility and Construct Validity of PROMIS and “Legacy” Instruments in an Academic Scleroderma Clinic

ObjectiveThe National Institutes of Health Patient-Reported Outcomes Measurement Information System (PROMIS) roadmap initiative is a cooperative group program of research designed to develop, evaluate, and standardize item banks to measure patient-reported outcomes relevant across medical conditions. The objective of the current study was to assess feasibility and evaluation of the construct validity of PROMIS item banks versus legacy measures in an observational study in systemic sclerosis (SSc). We hypothesized that the PROMIS item banks can be administered in a clinical setting if there is adequate staff support without disrupting the flow of clinic.MethodsPatients with SSc in a single academic center completed computerized adaptive test (CAT) administered PROMIS item banks during the clinic visit and legacy measures (using paper and pencil). The construct validity of PROMIS items was evaluated by examining correlations with corresponding legacy measures using multitrait-multimethod analysis.ResultsParticipants consisted of 143 SSc patients with an average age of 51.5 years; 71% were female and 68% were white. The average number of items completed for each CAT-administered item bank ranged from 5 to 8 (69 CAT items per patient), and the average time to complete each CAT-administered item bank ranged from 48 seconds to 1.9 minutes per patient (average time = 11.9 minutes/per patient for 11 banks). All correlations between PROMIS domains and respective legacy measures were large and in the hypothesized direction (ranged from 0.61 to 0.82).ConclusionOur study supports the construct validity of the CAT-administered PROMIS item banks and shows that they can be administered successfully in a clinic with support staff. Future studies should assess the feasibility of PROMIS item banks in a busy clinical practice.     

The PROMIS Fatigue<Subscript>FM</Subscript> Profile: a self-report measure of fatigue for use in fibromyalgia

Purpose

Fibromyalgia (FM) is characterized by myriad symptoms and problems. Fatigue is one of the most common, distressing, and disabling symptoms in FM. The purpose of this study was to use fatigue item banks that were developed as part of the Patient-Reported Outcomes Measurement Information System (PROMIS) to devise a self-report measure of fatigue for use in individuals with FM.

Methods

A sample of 532 adults with FM (age range = 18–77, 96.1 % female) completed the PROMIS fatigue item bank. Factor analyses and item response theory analyses were used to identify dimensionality and optimally performing items. These data were used in combination with clinical input to select items for a fatigue self-report measure for use in FM.

Results

Factor analyses revealed four distinct factors in the PROMIS fatigue item bank; items for each univariate subscale were identified by selecting four items with high item information values. A 16-item measure, the PROMIS Fatigue_FM Profile, consisting of four 4-item short forms reflecting fatigue experience (“intensity”) and fatigue impact in three subdomains—social, cognitive, and motivation—was created. The new PROMIS Fatigue_FM Profile short forms showed excellent internal reliability, low ceiling and floor effects, and equivalent or higher test information compared to the standard 4- and 7-item PROMIS fatigue short forms.

Conclusions

The newly developed PROMIS Fatigue_FM Profile, a 16-item measure consisting of four 4-item short forms of self-reported fatigue severity, shows early evidence of good psychometric characteristics, provides the ability to use short forms that assess distinct aspects of fatigue experience and fatigue impact, and demonstrates equivalent or higher levels of test information compared to standard PROMIS fatigue short forms with similar number of items. The PROMIS Fatigue_FM Profile indicated fatigue experience and impact levels approximately 1.5 standard deviations above the normative sample mean across all short forms. Future work to evaluate the validity and reliability of this new measure in individuals with FM is needed.

     

Item usage in a multidimensional computerized adaptive test (MCAT) measuring health-related quality of life

Purpose

Examining item usage is an important step in evaluating the performance of a computerized adaptive test (CAT). We study item usage for a newly developed multidimensional CAT which draws items from three PROMIS domains, as well as a disease-specific one.

Methods

The multidimensional item bank used in the current study contained 194 items from four domains: the PROMIS domains fatigue, physical function, and ability to participate in social roles and activities, and a disease-specific domain (the COPD-SIB). The item bank was calibrated using the multidimensional graded response model and data of 795 patients with chronic obstructive pulmonary disease. To evaluate the item usage rates of all individual items in our item bank, CAT simulations were performed on responses generated based on a multivariate uniform distribution. The outcome variables included active bank size and item overuse (usage rate larger than the expected item usage rate).

Results

For average θ-values, the overall active bank size was 9–10%; this number quickly increased as θ-values became more extreme. For values of ?2 and +2, the overall active bank size equaled 39–40%. There was 78% overlap between overused items and active bank size for average θ-values. For more extreme θ-values, the overused items made up a much smaller part of the active bank size: here the overlap was only 35%.

Conclusions

Our results strengthen the claim that relatively short item banks may suffice when using polytomous items (and no content constraints/exposure control mechanisms), especially when using MCAT.

     

Item banking to improve, shorten and computerize self-reported fatigue: An illustration of steps to create a core item bank from the FACIT-Fatigue Scale

Fatigue is a common symptom among cancer patients and the general population. Due to its subjective nature, fatigue has been difficult to effectively and efficiently assess. Modern computerized adaptive testing (CAT) can enable precise assessment of fatigue using a small number of items from a fatigue item bank. CAT enables brief assessment by selecting questions from an item bank that provide the maximum amount of information given a person's previous responses. This article illustrates steps to prepare such an item bank, using 13 items from the Functional Assessment of Chronic Illness Therapy Fatigue Subscale (FACIT-F) as the basis. Samples included 1022 cancer patients and 1010 people from the general population. An Item Response Theory (IRT)-based rating scale model, a polytomous extension of the Rasch dichotomous model was utilized. Nine items demonstrating acceptable psychometric properties were selected and positioned on the fatigue continuum. The fatigue levels measured by these nine items along with their response categories covered 66.8% of the general population and 82.6% of the cancer patients. Although the operational CAT algorithms to handle polytomously scored items are still in progress, we illustrated how CAT may work by using nine core items to measure level of fatigue. Using this illustration, a fatigue measure comparable to its full-length 13-item scale administration was obtained using four items. The resulting item bank can serve as a core to which will be added a psychometrically sound and operational item bank covering the entire fatigue continuum.     

Developing tailored instruments: item banking and computerized adaptive assessment

Item banks and Computerized Adaptive Testing (CAT) have the potential to greatly improve the assessment of health outcomes. This review describes the unique features of item banks and CAT and discusses how to develop item banks. In CAT, a computer selects the items from an item bank that are most relevant for and informative about the particular respondent; thus optimizing test relevance and precision. Item response theory (IRT) provides the foundation for selecting the items that are most informative for the particular respondent and for scoring responses on a common metric. The development of an item bank is a multi-stage process that requires a clear definition of the construct to be measured, good items, a careful psychometric analysis of the items, and a clear specification of the final CAT. The psychometric analysis needs to evaluate the assumptions of the IRT model such as unidimensionality and local independence; that the items function the same way in different subgroups of the population; and that there is an adequate fit between the data and the chosen item response models. Also, interpretation guidelines need to be established to help the clinical application of the assessment. Although medical research can draw upon expertise from educational testing in the development of item banks and CAT, the medical field also encounters unique opportunities and challenges.     

Predicting EuroQol (EQ-5D) scores from the patient-reported outcomes measurement information system (PROMIS) global items and domain item banks in a United States sample

Background  Preference-based health index scores provide a single summary score assessing overall health-related quality of life and are useful as an outcome measure in clinical studies, for estimating quality-adjusted life years for economic evaluations, and for monitoring the health of populations. We predicted EuroQoL (EQ-5D) index scores from patient-reported outcomes measurement information system (PROMIS) global items and domain item banks. Methods  This was a secondary analysis of health outcome data collected in an internet survey as part of the PROMIS Wave 1 field testing. For this study, we included the 10 global items and the physical function, fatigue, pain impact, anxiety, and depression item banks. Linear regression analyses were used to predict EQ-5D index scores based on the global items and selected domain banks. Results  The regression models using eight of the PROMIS global items (quality of life, physical activities, mental health, emotional problems, social activities, pain, and fatigue and either general health or physical health items) explained 65% of the variance in the EQ-5D. When the PROMIS domain scores were included in a regression model, 57% of the variance was explained in EQ-5D scores. Comparisons of predicted to actual EQ-5D scores by age and gender groups showed that they were similar. Conclusions  EQ-5D preference scores can be predicted accurately from either the PROMIS global items or selected domain banks. Application of the derived regression model allows the estimation of health preference scores from the PROMIS health measures for use in economic evaluations.     

Patient-reported outcomes measurement information system (PROMIS) domain names and definitions revisions: further evaluation of content validity in IRT-derived item banks

Purpose  

Content validity of patient-reported outcomes (PROs) is evaluated primarily during item development, but subsequent psychometric analyses, particularly for item response theory (IRT)-derived scales, often result in considerable item pruning and potential loss of content. After selecting items for the PROMIS banks based on psychometric and content considerations, we invited external content expert reviews of the degree to which the initial domain names and definitions represented the calibrated item bank content.     

The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008

ObjectivesPatient-reported outcomes (PROs) are essential when evaluating many new treatments in health care; yet, current measures have been limited by a lack of precision, standardization, and comparability of scores across studies and diseases. The Patient-Reported Outcomes Measurement Information System (PROMIS) provides item banks that offer the potential for efficient (minimizes item number without compromising reliability), flexible (enables optional use of interchangeable items), and precise (has minimal error in estimate) measurement of commonly studied PROs. We report results from the first large-scale testing of PROMIS items.Study Design and SettingFourteen item pools were tested in the U.S. general population and clinical groups using an online panel and clinic recruitment. A scale-setting subsample was created reflecting demographics proportional to the 2000 U.S. census.ResultsUsing item-response theory (graded response model), 11 item banks were calibrated on a sample of 21,133, measuring components of self-reported physical, mental, and social health, along with a 10-item Global Health Scale. Short forms from each bank were developed and compared with the overall bank and with other well-validated and widely accepted (“legacy”) measures. All item banks demonstrated good reliability across most of the score distributions. Construct validity was supported by moderate to strong correlations with legacy measures.ConclusionPROMIS item banks and their short forms provide evidence that they are reliable and precise measures of generic symptoms and functional reports comparable to legacy instruments. Further testing will continue to validate and test PROMIS items and banks in diverse clinical populations.     

An item bank was created to improve the measurement of cancer-related fatigue

OBJECTIVE: Cancer-related fatigue (CRF) is one of the most common unrelieved symptoms experienced by patients. CRF is underrecognized and undertreated due to a lack of clinically sensitive instruments that integrate easily into clinics. Modern computerized adaptive testing (CAT) can overcome these obstacles by enabling precise assessment of fatigue without requiring the administration of a large number of questions. A working item bank is essential for development of a CAT platform. The present report describes the building of an operational item bank for use in clinical settings with the ultimate goal of improving CRF identification and treatment. STUDY DESIGN AND SETTING: The sample included 301 cancer patients. Psychometric properties of items were examined by using Rasch analysis, an Item Response Theory (IRT) model. RESULTS AND CONCLUSION: The final bank includes 72 items. These 72 unidimensional items explained 57.5% of the variance, based on factor analysis results. Excellent internal consistency (alpha=0.99) and acceptable item-total correlation were found (range: 0.51-0.85). The 72 items covered a reasonable range of the fatigue continuum. No significant ceiling effects, floor effects, or gaps were found. A sample short form was created for demonstration purposes. The resulting bank is amenable to the development of a CAT platform.     

Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS)

OBJECTIVE: The Patient-Reported Outcomes Measurement Information System (PROMIS) was initiated to improve precision, reduce respondent burden, and enhance the comparability of health outcomes measures. We used item response theory (IRT) to construct and evaluate a preliminary item bank for physical function assuming four subdomains. STUDY DESIGN AND SETTING: Data from seven samples (N=17,726) using 136 items from nine questionnaires were evaluated. A generalized partial credit model was used to estimate item parameters, which were normed to a mean of 50 (SD=10) in the US population. Item bank properties were evaluated through Computerized Adaptive Test (CAT) simulations. RESULTS: IRT requirements were fulfilled by 70 items covering activities of daily living, lower extremity, and central body functions. The original item context partly affected parameter stability. Items on upper body function, and need for aid or devices did not fit the IRT model. In simulations, a 10-item CAT eliminated floor and decreased ceiling effects, achieving a small standard error (< 2.2) across scores from 20 to 50 (reliability >0.95 for a representative US sample). This precision was not achieved over a similar range by any comparable fixed length item sets. CONCLUSION: The methods of the PROMIS project are likely to substantially improve measures of physical function and to increase the efficiency of their administration using CAT.     

The PROMIS Physical Function item bank was calibrated to a standardized metric and shown to improve measurement efficiency

ObjectiveTo document the development and psychometric evaluation of the Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function (PF) item bank and static instruments.Study Design and SettingThe items were evaluated using qualitative and quantitative methods. A total of 16,065 adults answered item subsets (n > 2,200/item) on the Internet, with oversampling of the chronically ill. Classical test and item response theory methods were used to evaluate 149 PROMIS PF items plus 10 Short Form-36 and 20 Health Assessment Questionnaire-Disability Index items. A graded response model was used to estimate item parameters, which were normed to a mean of 50 (standard deviation [SD] = 10) in a US general population sample.ResultsThe final bank consists of 124 PROMIS items covering upper, central, and lower extremity functions and instrumental activities of daily living. In simulations, a 10-item computerized adaptive test (CAT) eliminated floor and decreased ceiling effects, achieving higher measurement precision than any comparable length static tool across four SDs of the measurement range. Improved psychometric properties were transferred to the CAT's superior ability to identify differences between age and disease groups.ConclusionThe item bank provides a common metric and can improve the measurement of PF by facilitating the standardization of patient-reported outcome measures and implementation of CATs for more efficient PF assessments over a larger range.     

Development and psychometric properties of the PROMIS® pediatric fatigue item banks

Purpose

This paper reports on the development and psychometric properties of self-reported pediatric fatigue item banks as part of the Patient-Reported Outcomes Measurement Information System (PROMIS).

Methods

Candidate items were developed by using PROMIS qualitative methodology. The resulting 39 items (25 tiredness related and 14 energy related) were field tested in a sample that included 3,048 participants aged 8–17 years. We used confirmatory factor analysis (CFA) to evaluate dimensionality and differential item functioning (DIF) analysis to evaluate parameter stability between genders and by age; we examined residual correlations to evaluate local dependence (LD) among items and estimated the parameters of item response theory (IRT) models.

Results

Of 3,048 participants, 48 % were males, 60 % were white, and 23 % had at least one chronic condition. CFA results suggest two moderately correlated factors. Two items were removed due to high LD, and three due to gender-based DIF. Two item banks were calibrated separately using IRT: Tired and (Lack of) Energy, which consisted of 23 and 11 items, respectively; 10- and 8-item short-forms were created.

Conclusion

The PROMIS assessment of self-reported fatigue in pediatrics includes two item banks: Tired and (Lack of) Energy. Both demonstrated satisfactory psychometric properties and can be used for research settings.     

Construction of the eight-item patient-reported outcomes measurement information system pediatric physical function scales: built using item response theory

Objective

To create self-report physical function (PF) measures for children using modern psychometric methods for item analysis as part of patient-reported outcomes measurement information system (PROMIS).

Study Design and Setting

PROMIS qualitative methodology was applied to develop two PF item pools that comprised 32 mobility and 38 upper extremity items. Items were computer administered to subjects aged 8-17 years. Scale dimensionality and sources of local dependence (LD) were evaluated with factor analysis. Items were analyzed for differential item functioning (DIF) between genders. Items with LD, DIF, or low discrimination were considered for removal. Computerized adaptive testing performance was simulated, and short forms were constructed.

Results

Three thousand forty-eight children (51.8% female, 40% nonwhite, and 22.7% chronically ill) participated. At least 754 respondents answered each item. Factor analytical results confirmed two dimensions of PF. Fifty-two of 70 items tested were retained. A 23-item mobility bank and a 29-item upper extremity bank resulted, and an eight-item short forms were created. The item banks have high information from the population mean to three standard deviations below.

Conclusions

PROMIS pediatric PF item banks and eight-item short forms assess two dimensions, mobility, and upper extremity function and show good psychometric characteristics after large-scale testing.     

Having a fit: impact of number of items and distribution of data on traditional criteria for assessing IRT’s unidimensionality assumption

Purpose  Confirmatory factor analysis fit criteria typically are used to evaluate the unidimensionality of item banks. This study explored the degree to which the values of these statistics are affected by two characteristics of item banks developed to measure health outcomes: large numbers of items and nonnormal data. Methods  Analyses were conducted on simulated and observed data. Observed data were responses to the Patient-Reported Outcome Measurement Information System (PROMIS) Pain Impact Item Bank. Simulated data fit the graded response model and conformed to a normal distribution or mirrored the distribution of the observed data. Confirmatory factor analyses (CFA), parallel analysis, and bifactor analysis were conducted. Results  CFA fit values were found to be sensitive to data distribution and number of items. In some instances impact of distribution and item number was quite large. Conclusions  We concluded that using traditional cutoffs and standards for CFA fit statistics is not recommended for establishing unidimensionality of item banks. An investigative approach is favored over reliance on published criteria. We found bifactor analysis to be appealing in this regard because it allows evaluation of the relative impact of secondary dimensions. In addition to these methodological conclusions, we judged the items of the PROMIS Pain Impact bank to be sufficiently unidimensional for item response theory (IRT) modeling.     

Cognitive interviewing in the evaluation of fatigue items: Results from the patient-reported outcomes measurement information system (PROMIS)

OBJECTIVES: Cognitive Interviewing (CI) is a technique increasingly used to obtain respondent feedback on potential items during questionnaire development. No standard guidelines exist by which to incorporate CI feedback in deciding to retain, revise, or eliminate potential items. We used CI in developing fatigue items for the National Institutes of Health (NIH) Patient-Reported Outcomes Measurement Information System (PROMIS) Roadmap initiative. Our aims were to describe the CI process, formally evaluate the utility of decisions made on the basis of CI, and offer suggestions for future research. METHODS: Participants were 22 patients with a diverse range of chronic health conditions. During CI, each participant provided feedback on a series of items. We then reviewed the CI data and decided whether to retain, revise, or eliminate each potential item. Following this, we developed or adopted three quantitative methods to compare retained versus eliminated items. RESULTS: Retained items raised fewer serious concerns, were less likely to be viewed as non-applicable, and were less likely to display problems with clarity or to make incorrect assumptions about respondents. CONCLUSIONS: CI was useful in developing the PROMIS fatigue items and the methods used to judge CI for the present item set may be useful for future investigations.     

Selection of items for a computer-adaptive test to measure fatigue in patients with rheumatoid arthritis: a Delphi approach

Purpose

Computer-adaptive tests (CATs) can measure precisely at individual level with few items selected from an item bank. Our aim was to select fatigue items to develop a CAT for rheumatoid arthritis (RA) and include expert opinions that are important for content validity of measurement instruments.

Methods

Items were included from existing fatigue questionnaires and generated from interview material. In a Delphi procedure, rheumatologists, nurses, and patients evaluated the initial pool of 294 items. Items were selected for the CAT development if rated as adequate by at least 80% of the participants (when 50% or less agreed, they were excluded). Remaining items were adjusted based on participants?? comments and re-evaluated in the next round. The procedure stopped when all items were selected or rejected.

Results

A total of 10 rheumatologists, 20 nurses, and 15 rheumatoid arthritis patients participated. After the first round, 96 of 294 items were directly selected. Nine items were directly excluded, and remaining items were adjusted. In the second round, 124 items were presented for re-evaluation. Ultimately, 245 items were selected.

Conclusion

This study revealed a qualitatively evaluated item pool to be used for the item bank/CAT development. The Delphi procedure is a beneficial approach to select adequate items for measuring fatigue in RA.     

Difference in method of administration did not significantly impact item response: an IRT-based analysis from the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative

Purpose

To test the impact of method of administration (MOA) on the measurement characteristics of items developed in the Patient-Reported Outcomes Measurement Information System (PROMIS).

Methods

Two non-overlapping parallel 8-item forms from each of three PROMIS domains (physical function, fatigue, and depression) were completed by 923 adults (age 18–89) with chronic obstructive pulmonary disease, depression, or rheumatoid arthritis. In a randomized cross-over design, subjects answered one form by interactive voice response (IVR) technology, paper questionnaire (PQ), personal digital assistant (PDA), or personal computer (PC) on the Internet, and a second form by PC, in the same administration. Structural invariance, equivalence of item responses, and measurement precision were evaluated using confirmatory factor analysis and item response theory methods.

Results

Multigroup confirmatory factor analysis supported equivalence of factor structure across MOA. Analyses by item response theory found no differences in item location parameters and strongly supported the equivalence of scores across MOA.

Conclusions

We found no statistically or clinically significant differences in score levels in IVR, PQ, or PDA administration as compared to PC. Availability of large item response theory-calibrated PROMIS item banks allowed for innovations in study design and analysis.