Evaluation of point-of-care PRO assessment in clinic settings: integration,parallel-forms reliability,and patient acceptability of electronic QOL measures during clinic visits

Purpose

Assessment of patient-reported outcomes (PROs), such as health-related quality of life, has become an important component of healthcare that measures the impact of disease and medical treatment on patient health. Collecting PROs during point-of-care assessments and integrating them into the clinical setting, however, remains challenging. The objective of this pilot study was to evaluate the reliability, usability, and acceptability of point-of-care electronic PRO assessments implemented in a prostate cancer clinic.

Methods

Fifty subjects completed paper–pencil and computerized formats of the Expanded Prostate Cancer Index Composite (EPIC), a validated, condition-specific QOL instrument, at separate times before treatment. Parallel-forms reliability was evaluated by comparing mean scores, variations in response distribution, and correlations between administration formats. Correlation coefficients of at least 0.70 were used for reliability testing. Differences between administration forms, indicating potential bias, were compared using the signed-rank test. A 6-item acceptability scale was also used to evaluate patient acceptability and satisfaction with the electronic format.

Results

Mean scores and standard deviations were similar between the paper–pencil and electronic forms across all EPIC instrument domains, and no assessment bias was found. Each EPIC domain demonstrated a high reliability between administration formats (correlation coefficients: 0.70–0.98). The majority (>90 %) of respondents found that the computerized QOL format was user friendly and simple to use.

Conclusions

Point-of-care computerized QOL assessments were reliable and acceptable to patients in this study, supporting the feasibility of PRO integration at the point-of-care in clinical settings.

     

International Development of the Unidimensional Fatigue Impact Scale (U-FIS)

ObjectiveThe 22-item Unidimensional Fatigue Impact Scale (U-FIS) provides an index of the impact of fatigue on patients with multiple sclerosis (MS). The objective is to produce eight new language versions of the U-FIS: Canadian-English, Canadian-French, French, German, Italian, Spanish, Swedish, and US-English.MethodsThe U-FIS was translated via two translation panels. Cognitive debriefing interviews conducted with patients in each country assessed face and content validity. Scaling and psychometric properties were assessed via survey data with patients in each country completing the U-FIS, Nottingham Health Profile (NHP), and demographic questions.ResultsCognitive debriefing interviews demonstrated U-FIS acceptability. Analysis of postal survey data showed all new language versions to be unidimensional. Reliability was high, with test-retest correlations and internal-consistency coefficients exceeding 0.85. Initial evidence of validity was provided by moderate to high correlations with NHP scales. The U-FIS was able to discriminate between groups based on employment status, perceived MS severity, and general health.ConclusionThe U-FIS is a practical new measure of the impact of fatigue. It was successfully adapted into eight new languages to broaden availability for researchers. Psychometric analyses indicated that the new language versions were unidimensional and reproducible with promising construct validity.     

A study of a multi-source feedback system for international medical graduates holding defined licences

OBJECTIVE: To develop and assess the feasibility and psychometric properties of multi-source feedback questionnaires to monitor international medical graduates practising in Canada under 'defined' licences. METHOD: Four questionnaires (patient, co-worker, colleague and self) were developed and administered in 2 phases through paper-based and telephone or Internet formats. Reliability was assessed with Cronbach's alpha and generalisability coefficient analyses. Validity was established through mean ratings, 'unable to respond' rates and factor analyses. RESULTS: A total of 37 doctors participated in the 2 phases. Overall response rates were 70% for patients, 86% for co-workers, 72% for medical colleagues and 92% for self, with response rates higher for the paper-based format than the Internet and phone formats. The instruments had high internal consistency reliability, with Cronbach's alphas of 0.83 for self-assessment and > 0.90 for the other instruments. The generalisability coefficients were Ep(2) = 0.71 for 25 patients on a 13-item survey, Ep(2) = 0.59 for 8 co-workers on a 13-item survey, and Ep(2) = 0.67 for 8 colleagues on a 21-item questionnaire. The range of mean scores was narrow (between 4 and 5) for all items and all surveys. The factor analyses identified that 2 factors accounted for 70% or more of the variance for the patient and colleague surveys and 60% of the variance for the co-worker survey. CONCLUSION: These data suggest that the instruments have reasonable psychometric properties. Traditional survey methods (i.e. paper-based) yielded better results than Internet or phone methods for this group of doctors.     

A comparison of visual analogue and numerical rating scale formats for the Lung Cancer Symptom Scale (LCSS): Does format affect patient ratings of symptoms and quality of life?

Problem and Purpose: The Lung Cancer Symptom Scale (LCSS), a site-specific health-related quality of life measure for patients with lung cancer, was originally developed using a Visual Analogue Scale (VAS) format. However, the VAS format is not readily compatible with data management and software programs using scanning. The primary aim of this study was to evaluate the convergence of ratings obtained with a Numerical Rating Scale (NRS), with an 11-pt response category format, to those obtained with a VAS format. The intent was to determine the degree of agreement between two formats to generalize the existing psychometric properties for the original measure to the new presentation. Design/setting: This methodological study evaluated the feasibility, reliability, and validity of a NRS format for the LCSS. The study was conducted at two cancer centers in New York City. Patients/procedures: Sixty-eight patients with non-small cell lung cancer (NSCLC) completed both versions of the LCSS along with demographic and feasibility questions on a single occasion. The VAS form was administered first, followed by the NRS form to prevent bias. The intraclass correlation coefficient (ICC), Lins concordance correlation coefficient (CCC), and Bland–Altman plots were used to evaluate agreement and to characterize bias. Results: Cronbachs alpha for the NRS format total score was 0.89 for the 68 patients with NSCLC. Agreement was excellent, with both the ICC and CCC 0.90 for the two summary scores (total score and average symptom burden index) for the LCSS. Only five of the nine individual items showed this level of strict agreement. An agreement criterion of 0.80 (representing excellent) was observed for seven of the nine individual items (all but appetite loss and hemoptysis). Mean differences tended to be slightly lower for the VAS format compared to the NRS format (more so for the appetite and hemoptysis items), with evidence of scale shift for the same two items. The summary measures showed good concordance as measured by the ICC and CCC, but did display mean differences (VAS – NRS) of –2.7 and –3.1, respectively. Conclusions: Overall, the NRS format for the LCSS suitable for scanning has good feasibility, reliability (internal consistency), and convergent validity. The complete set of concordance evaluation measures supports the reproducibility of VAS scores by NRS scores, particularly for the two summary scores.     

"Perfect health" versus "disease free": the impact of anchor point choice on the measurement of preferences and the calculation of disease-specific disutilities.

BACKGROUND: During preference testing, some investigators use "perfect health" as the upper anchor point of their measurement scale ("Q scale"), whereas others use "disease free" ("q scale"), which can confound the interpretation and comparison of study results. METHODS: We measured current health preferences among 74 patients with cervical spondylotic myelopathy (CSM) on both the Q and q scales using the visual analogue scale (VAS), standard gamble (SG), time tradeoff (TTO), and willingness to pay (WTP). RESULTS: There were significant differences in mean Q and q scale values for the VAS, SG, and WTP (for all, P < 0.011); there were no significant differences for mean TTO values (P = 0.592). CSM accounted for 63% to 82% of total disutility, whereas other comorbidities accounted for 28% to 37%. CONCLUSIONS: Preferences for CSM differ when measured on the Q and q scales. Caution should be used when comparing and interpreting health values measured on scales with different upper anchors.     

Changing Hospital Newborn Nursery Practice: Results from a Statewide “Back to Sleep” Nurses Training Program

OBJECTIVE: In response to findings from a statewide survey of hospital nurses, the authors designed, conducted, and evaluated a "Back to Sleep" nursing curriculum and training program in Missouri hospitals using two distinct training formats. This article evaluates the initial and follow-up outcomes for training participants and assesses the impact of training format on participant outcomes. METHODS: Participants selected training format by hospital site. In each training format, participants responded to a pre and post test questionnaire measuring knowledge, beliefs, and current infant care behaviors as well as satisfaction with the training. Three months after completion of all statewide trainings, the authors also conducted a follow-up survey. RESULTS: Nurses who participated in the training reported statistically significant improvements in knowledge and "Back to Sleep" adherent beliefs. Over 98% of participants (N=515) intended to place infants in back-only sleep positions following the training. Knowledge, attitudes, and practice intentions were significantly improved across both training formats. Additionally, follow-up survey respondents statewide (N=295) reported lasting improvements, including 63% of nurses reportedly using supine-only sleep position for infants after the first 24 h of life, compared to 28% in the original statewide survey. CONCLUSIONS: Further research is needed to determine the long-term impact of this intervention and assess its applicability beyond this initial implementation. Ultimately, the findings from the evaluation of this pilot intervention and nursing-specific "Back to Sleep" curriculum demonstrate that it has a promising effect on risk-reduction adherence in hospital settings where parent observations of safe sleep behavior first occur.     

The effects of absolute risks, relative risks, frequencies, and probabilities on decision quality

It is important to understand how the quality of people's decision making may be affected by the format used to present treatment benefits. Two experiments compared the accuracy of presenting the benefits of cancer screening tests or vaccines using either absolute or relative risk formats that included baseline risk information. Moreover, the absolute and/or baseline risks were presented using either natural frequencies or probabilities. In both experiments, accuracy was measured by the sensitivity of choices to differences in absolute rather than relative risks. Experiment 1 showed no significant differences in sensitivity between the relative and absolute risk formats when the risks were presented as natural frequencies. Sensitivity was, however, poor in both probability versions. Experiment 2 tested the natural frequency versions more stringently by presenting choices with different levels of difficulty. The author found that decision quality was significantly less affected by increases in difficulty in the absolute risk format. Presenting baseline risks using natural frequencies may help to reduce the biasing effects of relative risks but decision quality may not be on a par with the accuracy of decisions made when absolute risks are presented in natural frequency formats.     

Feasibility and reliability of an in-training assessment programme in an undergraduate clerkship

INTRODUCTION: Structured assessment, embedded in a training programme, with systematic observation, feedback and appropriate documentation may improve the reliability of clinical assessment. This type of assessment format is referred to as in-training assessment (ITA). The feasibility and reliability of an ITA programme in an internal medicine clerkship were evaluated. The programme comprised 4 ward-based test formats and 1 outpatient clinic-based test format. Of the 4 ward-based test formats, 3 were single-sample tests, consisting of 1 student-patient encounter, 1 critical appraisal session and 1 case presentation. The other ward-based test and the outpatient-based test were multiple sample tests, consisting of 12 ward-based case write-ups and 4 long cases in the outpatient clinic. In all the ITA programme consisted of 19 assessments. METHODS: During 41 months, data were collected from 119 clerks. Feasibility was defined as over two thirds of the students obtaining 19 assessments. Reliability was estimated by performing generalisability analyses with 19 assessments as items and 5 test formats as items. RESULTS: A total of 73 students (69%) completed 19 assessments. Reliability expressed by the generalisability coefficients was 0.81 for 19 assessments and 0.55 for 5 test formats. CONCLUSIONS: The ITA programme proved to be feasible. Feasibility may be improved by scheduling protected time for assessment for both students and staff. Reliability may be improved by more frequent use of some of the test formats.     

Comparison of Open and Closed Questionnaire Formats in Obtaining Demographic Information From Canadian General Internists

The objective of this study was to compare the impact of closed- versus open-ended question formats on the completeness and accuracy of demographic data collected in a mailed survey questionnaire. We surveyed general internists in five Canadian provinces to determine their career satisfaction. We randomized respondents to receive versions of the questionnaire in which 16 demographic questions were presented in a closed-ended or open-ended format. Two questions required respondents to make a relatively simple computation (ensuring that three or four categories of response added to 100%). The response rate was 1007/1192 physicians (80.0%). The proportion of respondents with no missing data for all 16 questions was 44.7% for open-ended and 67.0% for closed-ended formats (P < 0.001). The odds of having missing items remained higher for open-ended response options after adjusting for a number of respondent characteristics (2.67, 95% confidence interval 2.01 to 3.55). For the two questions requiring computations focused on professional activity and income, there were more missing data (P = 0.02, 0.02, respectively) but fewer inaccurate responses (P = 0.009, 0.20, respectively) for the open-ended compared to the closed-ended format. Investigators can achieve higher response rates for demographic items using closed format response options, but at the risk of increasing inaccuracy in response to questions requiring computation.     

Family-centred service: moving ideas into practice

BACKGROUND: With parents more involved in their child's day-to-day care, concepts of family-centred service (FCS) are increasingly adopted in children's health and rehabilitation service organizations. METHODS: In this paper, we report the results of a study to develop and evaluate educational materials for parents, service providers and health sciences students about FCS. The materials focus on the nature and philosophy of FCS, and the practical skills and systemic changes required for its implementation. RESULTS: Thirty-six participants (12 families, 12 service providers and 12 rehabilitation science students) were randomly assigned to receive one of the six FCS educational packages, each containing three FCS educational sheets. Participants' ratings of the format and content, and the impact of the FCS Sheets were very high, with overall means above 5.0 on a 7-point scale. Using a mixed model analysis, we found significant differences in participants' ratings of familiarity with the materials (students were less familiar than service providers). After statistical adjustment for familiarity, there were no significant differences between the groups or the packages on ratings of format and content or impact. CONCLUSIONS: There were no significant differences in the way in which the participant groups rated the impact of the FCS Sheets and the specific packages did not have an effect on the participants' ratings. The FCS educational materials, even those less familiar to participants, were rated highly on format and content, and impact. Results indicate that the material was perceived to be important to each group, and was formatted and written in a way that was easy to understand. This finding counters current recommendations in the knowledge transfer literature that suggest different versions should be written for different target groups.     

Enhanced killing effect on 5-bromodeoxyuridine labelled bacteriophage T1 by monoenergetic synchrotron X-ray at the energy of bromine K-shell absorption edge

5-Bromo-2'-deoxyuridine labelled bacteriophage T1 was irradiated with monoenergetic X-rays obtained from synchroton radiation with the energies at 13.51 keV and 12.40 keV, just above and below the K-shell absorption edge (13.41 keV) of bromine, respectively. Phage samples were irradiated under three conditions, wet N2 gas, dry N2 gas and in vacuum, with water contents (g H2O/g sample) of 60, 2.6 and 0%. At 12.40 keV the D0 in kGy were 1.1 ("wet"), 1.9 ("dry") and 2.5 ("vacuum") for the Br-labelled phages and 1.6 ("wet"), 2.4 ("dry") and 6.2 ("vacuum") for unlabelled phages. The results clearly demonstrated that the X-ray sensitivities decreased with decrease in water contents. Br-enhancement ratios, ER = D0 (unlabelled)/D0 (Br-labelled), varied from 1.4 ("wet", 12.40 keV) to 2.5 ("vacuum", 13.51 keV). Auger enhancements which were defined by energy-dependent enhancement factor, Fen = [ER(13.51 keV)-ER(12.40 keV)]/ER(12.40 keV), were 0.09 +/- 0.09, 0.29 +/- 0.07 and 0.02 +/- 0.03 under "wet", "dry" and "vacuum" conditions, respectively. The change in Auger enhancement under "dry" condition in comparison with "wet" condition could be explained due to less of water. However the Auger enhancement decreased sharply under "vacuum" condition as the water content was zero. The reason for the sharp decrease in Auger enhancement under "vacuum" condition is difficult to understand. A possible explanation is discussed in the text.     

Validation of Electronic Data Capture of the Irritable Bowel Syndrome—Quality of Life Measure, the Work Productivity and Activity Impairment Questionnaire for Irritable Bowel Syndrome and the EuroQol

OBJECTIVES: To assess the comparability, reliability, and subject acceptability of electronic data capture (EDC) versions of Irritable Bowel Syndrome-Quality of Life (IBS-QOL), EuroQoL (EQ-5D) and Work Productivity and Activity Impairment (WPAI:IBS) instruments. METHODS: Comparability of EDC and paper questionnaires was evaluated in 72 subjects with IBS who completed a baseline EDC or paper questionnaire, a crossover questionnaire 24 hours later, and a retest of the crossover version at 1 week. The EDC version was presented on a hand-held device. Comparability was assessed using paired t-test statistics, intraclass correlation coefficients (ICC) and tests for internal consistency (Cronbach's alpha). RESULTS: No significant differences were found between scores obtained by paper questionnaire and EDC at the baseline and crossover assessments. ICCs between baseline and crossover assessments ranged from 0.83 to 0.96 for the IBS-QOL scores, 0.82 to 0.96 for the WPAI:IBS scores, and 0.77 to 0.82 for the EQ-5D. Internal consistency was comparable for the two data collection methods for the IBS-QOL overall score (0.96) and subscales and the EQ-5D Index (0.70 vs. 0.74). Retest statistics (ICC) were generally comparable between the EDC and paper versions for all scores. Ease of use was comparable for the two modes of administration, but more patients preferred EDC (47.2%) than the paper questionnaire (23.6%). CONCLUSIONS: EDC versions of the IBS-QOL, EQ-5D, and WPAI:IBS are comparable to paper questionnaires in internal consistency and test-retest reliability, and have greater patient acceptability.     

A randomized comparison of alternative formats for clinical simulations.

Computer-based clinical simulations for medical education vary widely in structure and format, yet few studies have examined which formats are optimal for particular educational settings. This study is a randomized comparison of the same simulated case in three formats: a "pedagogic" format offering explicit educational support, a "high-fidelity" format attempting to model clinical reasoning in the real world, and a "problem-solving" format that requires students to express specific diagnostic hypotheses. Data were collected from rising third-year medical students using a posttest, attitudinal questionnaire, students' write-ups of the case, and log files of students' progress through the simulation. Student performances on all measures differed significantly by format. In general, students using the pedagogic format were more proficient but less efficient. They acquired more information but were able to do proportionately less with it. The results suggest that the format of computer-based simulations is an important educational variable.     

Measuring preferences for schizophrenia outcomes with the time tradeoff method

Measuring preferences for schizophrenia outcomes facilitates meaningful integration of multiple outcome measures and multiple perspectives on treatment outcomes. The Time Tradeoff (TTO) technique, specifically developed for measuring health state preferences, is used widely in health research, but some evidence suggests that the TTO may work less well with schizophrenia than with other health conditions. This study tested the hypotheses that tailoring the time frame of the standard TTO to the course of schizophrenia and simplifying its presentation format would improve its feasibility and efficiency. Forty clinicians provided TTO ratings using 1 of 4 combinations of time frame and presentation format. Numeric ratings and quantitative and qualitative measures of feasibility showed that while participants preferred the simpler format, none of the alterations improved feasibility. Participants' ratings were prone to logical inconsistencies and participants found all 4 versions of the TTO confusing and poorly suited to the context of schizophrenia treatment.     

Rapid assessment of psychosocial well-being: Are computers the way forward in a clinical setting?

Objective: To compare the agreement between anxiety, depression and supportive care needs data obtained using a touchscreen computer survey with traditional pen-and-paper surveys. Methods: The sample consisted of 1304 consecutive patients attending the medical or radiation oncology outpatient department in two public cancer treatment centres. Of the 357 eligible patients, 355 (99%) consented to participate and 350 (98%) completed both touchscreen computer and pen-and-paper versions of the Hospital Anxiety and Depression Scale (HADS) and short-form Supportive Care Needs Survey (SCNS-short). Results: Simple values indicated moderate agreement ( = 0.57) between the two modes of survey administration for most HADS items. Simple coefficients indicated only fair agreement ( = 0.28) when a simplified response option format was used in the computerised SCNS-short. When the paper and computerised survey used the same response format, simple kgr; coefficients increased and indicated moderate agreement ( = 0.44) for most SCNS-short items. Coefficients indicated at least moderate agreement ( > 0.41) in identifying patients with elevated levels of anxiety and depression; there were no significant differences in the proportion of patients identified with elevated levels of anxiety and depression according to computer survey compared to paper survey. Prevalence-adjusted bias-adjusted (PABAK) coefficients indicated at least moderate agreement ( = 0.79) in identifying participants with moderate/high levels of unmet needs. However, participants tended to report lower levels of unmet needs with a simplified response format in the computerised SCNS-short compared to the paper survey. This was not observed when the response format of the computerised SCNS-short replicated the paper survey. Conclusions: Despite the advantages that computerised surveys offer for simplifying survey presentation, current results suggest the need to exactly replicate the question and response option format of the original paper survey to ensure the data collected are equivalent. This finding is particularly important given the potential application of computerised surveys in the clinical setting to quickly assess and identify patients' concerns requiring intervention by health care providers.     

Comparison of the acceptability ratings of appetizers under laboratory, base level and high altitude field conditions

The relationship between laboratory and field ratings was investigated for six different appetizers, including four ready-to-reconstitute mixes and two ready-to-eat munches. Liking ratings on a 5-point hedonic scale were obtained from an Indian Army field study at base level as well as at an altitude of 11,500 ft above sea level and for the same appetizers in the laboratory. The field trials of the six products were conducted in two phases and results revealed that the products were more acceptable at altitude, with increased liking scores as compared to base level. Subjective ratings for hunger revealed that at altitude, appetizer consumption had stimulated the appetite of the soldiers. The ability of laboratory ratings to predict acceptability of foods consumed under realistic conditions appears to depend on the convenience of the appetizer as well as the environmental conditions and the psycho-physiological status of the participants. The appetizers received higher ratings at altitude because of the pungent and spicy nature of appetizer mixes as compared with base field and laboratory conditions. However, for all the appetizers the pungent and sweet taste of the appetizer munches was highly preferred.