Replenishing a computerized adaptive test of patient-reported daily activity functioning

Purpose  Computerized adaptive testing (CAT) item banks may need to be updated, but before new items can be added, they must be linked to the previous CAT. The purpose of this study was to evaluate 41 pretest items prior to including them into an operational CAT. Methods  We recruited 6,882 patients with spine, lower extremity, upper extremity, and nonorthopedic impairments who received outpatient rehabilitation in one of 147 clinics across 13 states of the USA. Forty-one new Daily Activity (DA) items were administered along with the Activity Measure for Post-Acute Care Daily Activity CAT (DA-CAT-1) in five separate waves. We compared the scoring consistency with the full item bank, test information function (TIF), person standard errors (SEs), and content range of the DA-CAT-1 to the new CAT (DA-CAT-2) with the pretest items by real data simulations. Results  We retained 29 of the 41 pretest items. Scores from the DA-CAT-2 were more consistent (ICC = 0.90 versus 0.96) than DA-CAT-1 when compared with the full item bank. TIF and person SEs were improved for persons with higher levels of DA functioning, and ceiling effects were reduced from 16.1% to 6.1%. Conclusions  Item response theory and online calibration methods were valuable in improving the DA-CAT.     

Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes

Purpose

Multidimensional item response theory and computerized adaptive testing (CAT) are increasingly used in mental health, quality of life (QoL), and patient-reported outcome measurement. Although multidimensional assessment techniques hold promises, they are more challenging in their application than unidimensional ones. The authors comment on minimal standards when developing multidimensional CATs.

Methods

Prompted by pioneering papers published in QLR, the authors reflect on existing guidance and discussions from different psychometric communities, including guidelines developed for unidimensional CATs in the PROMIS project.

Results

The commentary focuses on two key topics: (1) the design, evaluation, and calibration of multidimensional item banks and (2) how to study the efficiency and precision of a multidimensional item bank. The authors suggest that the development of a carefully designed and calibrated item bank encompasses a construction phase and a psychometric phase. With respect to efficiency and precision, item banks should be large enough to provide adequate precision over the full range of the latent constructs. Therefore CAT performance should be studied as a function of the latent constructs and with reference to relevant benchmarks. Solutions are also suggested for simulation studies using real data, which often result in too optimistic evaluations of an item bank’s efficiency and precision.

Discussion

Multidimensional CAT applications are promising but complex statistical assessment tools which necessitate detailed theoretical frameworks and methodological scrutiny when testing their appropriateness for practical applications. The authors advise researchers to evaluate item banks with a broad set of methods, describe their choices in detail, and substantiate their approach for validation.

     

An evaluation of patient-reported outcomes found computerized adaptive testing was efficient in assessing stress perception

ObjectivesThis study aimed to develop and evaluate a first computerized adaptive test (CAT) for the measurement of stress perception (Stress-CAT), in terms of the two dimensions: exposure to stress and stress reaction.Study Design and SettingItem response theory modeling was performed using a two-parameter model (Generalized Partial Credit Model). The evaluation of the Stress-CAT comprised a simulation study and real clinical application. A total of 1,092 psychosomatic patients (N1) were studied. Two hundred simulees (N2) were generated for a simulated response data set. Then the Stress-CAT was given to n = 116 inpatients, (N3) together with established stress questionnaires as validity criteria.ResultsThe final banks included n = 38 stress exposure items and n = 31 stress reaction items. In the first simulation study, CAT scores could be estimated with a high measurement precision (SE < 0.32; ρ > 0.90) using 7.0 ± 2.3 (M ± SD) stress reaction items and 11.6 ± 1.7 stress exposure items. The second simulation study reanalyzed real patients data (N1) and showed an average use of items of 5.6 ± 2.1 for the dimension stress reaction and 10.0 ± 4.9 for the dimension stress exposure. Convergent validity showed significantly high correlations.ConclusionsThe Stress-CAT is short and precise, potentially lowering the response burden of patients in clinical decision making.     

An evaluation of a patient-reported outcomes found computerized adaptive testing was efficient in assessing osteoarthritis impact

BACKGROUND AND OBJECTIVES: Evaluate a patient-reported outcomes questionnaire that uses computerized adaptive testing (CAT) to measure the impact of osteoarthritis (OA) on functioning and well-being. MATERIALS AND METHODS: OA patients completed 37 questions about the impact of OA on physical, social and role functioning, emotional well-being, and vitality. Questionnaire responses were calibrated and scored using item response theory, and two scores were estimated: a Total-OA score based on patients' responses to all 37 questions, and a simulated CAT-OA score where the computer selected and scored the five most informative questions for each patient. Agreement between Total-OA and CAT-OA scores was assessed using correlations. Discriminant validity of Total-OA and CAT-OA scores was assessed with analysis of variance. Criterion measures included OA pain and severity, patient global assessment, and missed work days. RESULTS: Simulated CAT-OA and Total-OA scores correlated highly (r = 0.96). Both Total-OA and simulated CAT-OA scores discriminated significantly between patients differing on the criterion measures. F-statistics across criterion measures ranged from 39.0 (P < .001) to 225.1 (P < .001) for the Total-OA score, and from 40.5 (P < .001) to 221.5 (P < .001) for the simulated CAT-OA score. CONCLUSIONS: CAT methods produce valid and precise estimates of the impact of OA on functioning and well-being with significant reduction in response burden.     

Ensuring content validity of patient-reported outcomes: a shadow-test approach to their adaptive measurement

Purpose

Most computerized adaptive testing (CAT) applications in patient-reported outcomes (PRO) measurement to date are reliability-centric, with a primary objective of maximizing measurement efficiency. A key concern and a potential threat to validity is that, when left unconstrained, individual CAT administrations could have items with systematically different attributes, e.g., sub-domain coverage. This paper aims to provide a solution to the problem from an optimal test design framework using the shadow-test approach to CAT.

Methods

Following the approach, a case study was conducted using the PROMIS^® (Patient-Reported Outcomes Measurement Information System) fatigue item bank both with empirical and simulated response data. Comparisons between CAT administrations without and with the enforcement of content and item pool usage constraints were examined.

Results

The unconstrained CAT exhibited a high degree of variation in items selected from different substrata of the item bank. Contrastingly, the shadow-test approach delivered CAT administrations conforming to all specifications with a minimal loss in measurement efficiency.

Conclusions

The optimal test design and shadow-test approach to CAT provide a flexible framework for solving complex test-assembly problems with better control of their domain coverage than for the conventional use of CAT in PRO measurement. Applications in a wide array of PRO domains are expected to lead to more controlled and balanced use of CAT in the field.

     

A systematic review of measurement properties of patient-reported outcome measures for use in patients with foot or ankle diseases

Purpose

To identify currently available patient-reported outcome measures (PROMs) used in patients with foot or ankle diseases; and to critically appraise, compare and synthesize the psychometric evidence for the identified PROMs.

Methods

Literature searches were performed in Medline and EMBASE from their inception to January 25th, 2016. Methodological quality was evaluated using the COSMIN checklist. The final rating of the methodological quality of each study for each property was the lowest rating among the items within that property. The psychometric evidence of the properties investigated in the included articles was assessed using the quality criteria established by Terwee et al. The methodological quality ratings and psychometric evidence assessments were synthesized using the method first proposed by Schellingerhout et al.

Results

In total, 3077 articles were identified by the literature search, from which 115 studies investigating 50 PRO instruments were included in the review process. The Foot Function Index (FFI) was the most explored instrument, while the Manchester-Oxford Foot Questionnaire (MOXFQ) demonstrated the best properties.

Conclusion

Most PROMs on foot and ankle diseases have limited evidence for their psychometric properties. The MOXFQ, with the highest overall ratings, could be a useful PROM for evaluating patients with foot or ankle diseases, based on current available evidence. More research is needed to improve the quality of the standards used to assess PROMs and the studies making these assessments.

     

Methods for interpreting change over time in patient-reported outcome measures

Purpose

Interpretation guidelines are needed for patient-reported outcome (PRO) measures’ change scores to evaluate efficacy of an intervention and to communicate PRO results to regulators, patients, physicians, and providers. The 2009 Food and Drug Administration (FDA) Guidance for Industry Patient-Reported Outcomes (PRO) Measures: Use in Medical Product Development to Support Labeling Claims (hereafter referred to as the final FDA PRO Guidance) provides some recommendations for the interpretation of change in PRO scores as evidence of treatment efficacy.

Methods

This article reviews the evolution of the methods and the terminology used to describe and aid in the communication of meaningful PRO change score thresholds.

Results

Anchor- and distribution-based methods have played important roles, and the FDA has recently stressed the importance of cross-sectional patient global assessments of concept as anchor-based methods for estimation of the responder definition, which describes an individual-level treatment benefit. The final FDA PRO Guidance proposes the cumulative distribution function (CDF) of responses as a useful method to depict the effect of treatments across the study population.

Conclusions

While CDFs serve an important role, they should not be a replacement for the careful investigation of a PRO’s relevant responder definition using anchor-based methods and providing stakeholders with a relevant threshold for the interpretation of change over time.     

The future of outcomes measurement: item banking,tailored short-forms,and computerized adaptive assessment

The use of item banks and computerized adaptive testing (CAT) begins with clear definitions of important outcomes, and references those definitions to specific questions gathered into large and well-studied pools, or “banks” of items. Items can be selected from the bank to form customized short scales, or can be administered in a sequence and length determined by a computer programmed for precision and clinical relevance. Although far from perfect, such item banks can form a common definition and understanding of human symptoms and functional problems such as fatigue, pain, depression, mobility, social function, sensory function, and many other health concepts that we can only measure by asking people directly. The support of the National Institutes of Health (NIH), as witnessed by its cooperative agreement with measurement experts through the NIH Roadmap Initiative known as PROMIS (www.nihpromis.org), is a big step in that direction. Our approach to item banking and CAT is practical; as focused on application as it is on science or theory. From a practical perspective, we frequently must decide whether to re-write and retest an item, add more items to fill gaps (often at the ceiling of the measure), re-test a bank after some modifications, or split up a bank into units that are more unidimensional, yet less clinically relevant or complete. These decisions are not easy, and yet they are rarely unforgiving. We encourage people to build practical tools that are capable of producing multiple short form measures and CAT administrations from common banks, and to further our understanding of these banks with various clinical populations and ages, so that with time the scores that emerge from these many activities begin to have not only a common metric and range, but a shared meaning and understanding across users. In this paper, we provide an overview of item banking and CAT, discuss our approach to item banking and its byproducts, describe testing options, discuss an example of CAT for fatigue, and discuss models for long term sustainability of an entity such as PROMIS. Some barriers to success include limitations in the methods themselves, controversies and disagreements across approaches, and end-user reluctance to move away from the familiar.     

Strategies to improve patient-reported outcome completion rates in longitudinal studies

Quality of Life Research - The quality of patient-reported outcome (PRO) data can be compromised by non-response (NR) to scheduled questionnaires, particularly if reasons for NR are related to...     

COSMIN guideline for systematic reviews of patient-reported outcome measures

Purpose

Systematic reviews of patient-reported outcome measures (PROMs) differ from reviews of interventions and diagnostic test accuracy studies and are complex. In fact, conducting a review of one or more PROMs comprises of multiple reviews (i.e., one review for each measurement property of each PROM). In the absence of guidance specifically designed for reviews on measurement properties, our aim was to develop a guideline for conducting systematic reviews of PROMs.

Methods

Based on literature reviews and expert opinions, and in concordance with existing guidelines, the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) steering committee developed a guideline for systematic reviews of PROMs.

Results

A consecutive ten-step procedure for conducting a systematic review of PROMs is proposed. Steps 1–4 concern preparing and performing the literature search, and selecting relevant studies. Steps 5–8 concern the evaluation of the quality of the eligible studies, the measurement properties, and the interpretability and feasibility aspects. Steps 9 and 10 concern formulating recommendations and reporting the systematic review.

Conclusions

The COSMIN guideline for systematic reviews of PROMs includes methodology to combine the methodological quality of studies on measurement properties with the quality of the PROM itself (i.e., its measurement properties). This enables reviewers to draw transparent conclusions and making evidence-based recommendations on the quality of PROMs, and supports the evidence-based selection of PROMs for use in research and in clinical practice.

     

Late response to patient-reported outcome questionnaires after surgery was associated with worse outcome

ObjectivesNonresponse to patient-reported outcome (PRO) questionnaires after surgery might bias the results. Our aim was to gauge the potential impact of nonresponse bias by comparing the outcomes of early and late responders.Study Design and SettingThis study compares 59,565 early and 20,735 late responders who underwent a hip or knee replacement, hernia repair, or varicose vein (VV) surgery. The association between timeliness of response and three outcomes (the mean postoperative disease-specific PRO and generic PRO scores and the proportion reporting a fair or poor result) was examined by regression analysis.ResultsLate responders were slightly more likely to be young, nonwhite, deprived, and have a more severe preoperative condition with poorer quality of life. Late response was associated with a slightly poorer outcome in all four procedures although not statistically significant (P < 0.05) for VV surgery. Adjusting for preoperative characteristics reduced the strength of the associations, but they remained statistically significant.ConclusionAs nonresponse to PRO questionnaires introduces slight bias, differences in response rates between hospitals should be taken into account when making comparisons so as to avoid overestimating the performance of those with lower response rates and failing to detect poor performing hospitals.     

An evaluation of patient-reported outcome measures in lower limb reconstruction surgery

Purpose

To assess the measurement properties (acceptability, validity, reliability and responsiveness), of the MOS 36-Item Short-Form Health Survey (SF-36), the EQ-5D, the Short-Form McGill Pain Questionnaire (SF-MPQ) and the Musculoskeletal Functional Assessment Instrument (MFA), in patients who have undergone limb reconstruction surgery (LRS).

Methods

Four instruments measuring patient-reported outcome were completed at baseline and 12?months from surgery.

Results

101 LRS patients were recruited with 95 responding at baseline and 71 at a 12-month follow-up. Response rates at baseline were over 94%. In three instruments, there was evidence of floor or ceiling effect, the exception being the EQ-5D. Cronbach??s ?? statistics of internal consistency reliability were acceptable at ??0.80 for all dimensions of the MFA, the SF-MPQ PRI(S) and seven of the SF-36 dimensions. When comparing mean changes in scores between baseline and 12?months, the most responsive measure was the SF-36 with an average Standardised Response Mean of 0.48 for those who reported their health as better. Statistically significant differences were observed between the health change groups (??worse??, ??better?? and ??same??) for four dimensions of the SF-36, the two summary scores and the SF-6D.

Conclusions

Variation and poor performance of some of the instruments resulted in a recommendation of using the SF-36 and the SF-6D for LRS patients.     

Viewing assessments of patient-reported heath status as conversations: Implications for developing and evaluating patient-reported outcome measures

Quality of Life Research - Patient-reported outcome measures (PROMs) are frequently used in research to reflect the patient’s perspective. In this commentary, I argue that further...     

Linking measurement error to minimal important change of patient-reported outcomes

ObjectiveTo provide a practical example showing how to relate measurement error to minimal important change (MIC) to estimate sample sizes required for detecting important changes in research, and to determine the usefulness of patient-reported outcomes (PROs) in daily clinical practice.Study Design and SettingThe standard error of measurement (SEM) can be calculated as a parameter of measurement error. MIC can be determined by taking the mean change in score on the questionnaire in the subgroup of patients that reported to be “a little better” or “better” on a global rating scale minus the mean change in score in the subgroup of patients that reported to be “not changed.” Based on SEM and MIC, formulas are presented that can be used to estimate sample sizes and to determine whether the questionnaire can be used to monitor changes in individual patients in daily clinical practice.ResultsA fully worked-out example is presented, using Western Ontario and McMaster University Osteoarthritis Index data from a cohort of patients undergoing hip or knee replacement.ConclusionWe strongly recommend investigators to present the SEM and MIC of PROs and use these to estimate sample sizes for research and to determine the usefulness of these outcomes in daily clinical practice.     

Using patient-reported measurement to pave the path towards personalized medicine

Objective

Given the potential and importance of personalized or individualized medicine for health care delivery and its effects on patients’ quality of life, a plenary session was devoted to personalized medicine during the 19th Annual Conference of the International Society for Quality of Life Research held in October 2012 in Budapest, Hungary. This paper summarizes the three presentations and discusses their implications for quality-of-life research.

Methods

Reviews of the literature and presentation of empirical studies.

Results

Personalized screening for breast cancer. To individualize screening and only target those women with an increased risk for breast cancer, researchers at the Karolinska Institutet in Stockholm perform a large population-based study to identify high-risk women based on lifestyle, genetics, mammographic morphology, and other markers as well as quality of life. Personalized support for treatment adherence. Inclusion of a simple, brief adherence measure into the clinical visit has demonstrated significant improvement in medication-taking behaviour and resultant improvement in health status. Personalized diagnosis of mental disorders. The DSM-5, the current manual for mental disorders, contains patient-based symptom and diagnosis severity measures that allow more individualized diagnosis than was hitherto possible.

Conclusions

Personalized medicine will continue to be increasingly applied and holds the potential to improve health outcomes including quality of life. At the same time, it will invite a host of new ethical, practical, and psychosocial questions. Further reflection and discussion of how our field can embrace and address these emerging challenges is needed.     

Item response theory and its applications to patient-reported outcomes measurement

This article provides an overview of item response theory (IRT) models and how they can be appropriately applied to patient-reported outcomes (PROs) measurement. Specifically, the following topics are discussed: (a) basics of IRT, (b) types of IRT models, (c) how IRT models have been applied to date, and (d) new directions in applying IRT to PRO measurements.     

Reflection paper on copyright,patient-reported outcome instruments and their translations

With the growth of patient-reported outcome (PRO) measurement, questions arise regarding how copyright protection applies to PRO instruments in general and to their translations in particular. The main objectives of this reflection paper are: 1) to help authors of PRO instruments understand basic rules of intellectual property and copyright that protect the integrity of their instruments and derivatives; and 2) to provide recommendations to authors and users of PRO instruments to prevent misuse or abuse.National laws on intellectual property (IP) and the international Berne Convention fully apply to PRO instruments since they are creations of the mind. Therefore, the copyright holder / owner / claimant of a PRO instrument, i.e., the person or legal entity who owns the copyright of the instrument, is granted exclusive rights that are divided into two main categories: moral and economic rights. Moral rights are: 1) the right of attribution (or right of paternity), i.e., the right to claim authorship of the work, 2) the right against false attribution, and 3) the right of integrity, i.e., the right to object to any mutilation, deformation or modification of the work. Economic rights represent the exclusive rights of the author to make or authorize reproduction, development of derivative works, distribution and communication to the public. In other words, the PRO instrument’s copyright holder controls access (distribution, reproduction), and authorizes all derivative works, i.e., adaptations (e.g., electronic formats), modifications (e.g., shorter versions), and translations. Hence, the access to and use of an original PRO instrument and its derivatives in any kind of research should always be associated with the identification of its copyright holder. However, in some cases, this identification may be challenging, in particular when copyright ownership is not clearly defined. To prevent ownership conflicts as well as misuse or abuse of PRO instruments, the ISOQOL Translation and Cultural Adaptation Special Interest Group (TCA-SIG) provides recommendations to authors of PRO instruments and their users. In particular, the TCA-SIG recommends that the ownership of PRO instruments and their derivatives should be defined from the beginning (i.e., from the development of the instrument) and along the life cycle of the instrument between all parties involved. These recommendations apply not only to PRO instruments but also to all the other clinical outcome assessments (COAs), since they are also creations of the mind.