Test-retest and inter-rater reliability of the Danish version of the management of everday technology assessment for use with older adults with and without COPD

Background: The trend towards telemedicine increasingly requires clients to manage everyday technology (ET) to access and use health services. The Management of Everday Technology Assessment (META) is an observation-based instrument developed to evaluate the ability to manage ET.

Aim: To examine test-retest (TRR) and inter-rater reliability (IRR) of the Danish translation of the META for older adults with and without COPD.

Method and Materials: 47 older adults with COPD (n?=?23) and without (n?=?24) were recruited. IRR was examined by four raters paired across 30 participants. TRR was examined for 21 participants by the same rater administering the META twice within four weeks. A rank-based method for paired ordinal data was used to calculate percentage agreement (PA) and measures of systematic disagreement and individual variability. Mann Whitney U tests were used to compare PA to health status (presence/absence of COPD).

Results: Inter-rater PA was acceptable across 10 of 11 items and test-retest PA across 8 of 11 items. Systematic disagreement was present for one item in TRR. No significant differences in PA were found regarding health status.

Conclusion: The Danish META generates reliable scores for this sample. However, conclusive statements cannot be made for all items.     

Making use of research: Clinical views on an evaluation of everyday technology use

Objective: The study aim was to investigate how and when an evaluation of perceived difficulty in use of everyday technology (Everyday Technology Use Questionnaire, ETUQ) could be used in clinical occupational therapy. Method: Eight focus-group interviews were undertaken with a total of 42 participants (occupational therapists), and data were analysed with a constant comparative approach. Results: The findings are presented in four main categories, including (i) appropriate purposes and contexts for using ETUQ, (ii) standardization versus individual flexibility, (iii) approaching everyday technology use and occupation as one whole, and (iv) synthesizing and documentation. Conclusions: In conclusion, the participants considered ability to use technology to be an important topic for occupational therapy, particularly in investigations of clients with subtle disabilities and in connection with discharge from hospital – but not in inpatient care. They had different views on how to integrate ETUQ with evaluations of occupational performance, and new ideas on how information about clients’ ability to use technology could be utilized in interventions. They held standardized evaluations in high regard, but a paradox appeared in that many of them would use ETUQ in a non-standardized way, while simultaneously asking for a standardized output to be used in clients’ medical files and to guide interventions.     

Construct validity of a revised version of the Occupational Balance Questionnaire

Abstract

Background: Occupational balance is fundamental to occupational therapy and occupational science. Therefore, the Occupational Balance Questionnaire (OBQ) was developed and has previously been found psychometrically valid according to classical test theory.

Aim: To investigate the internal construct validity of the OBQ using Rasch measurement theory.

Material and methods: Data from two general population samples were used to investigate the psychometric properties of the OBQ according to Rasch measurement theory.

Results: The analyses identified problems with the current response scale and multidimensionality of two items. As a result, a revised version, the OBQ11, was suggested and exhibited response categories that worked properly, good reliability (0.92), model fit and measurement invariance across age and gender groups. The hierarchical item ordering was in agreement with previous research.

Conclusion: The new OBQ11 satisfies the measurement criteria defined by the Rasch model. However, further studies of additional samples are needed to validate its generic properties.

Significance: The purpose of the OBQ11 is to measure occupational balance of individuals or groups, and to identify aspects of occupational balance in need of improvement. Our observations suggest that the OBQ11 is a valid and promising complement to other instruments.     

Participation after acquired brain injury: Associations with everyday technology and activities in daily life

Background: The development of the information society has led to increased use of everyday technology and changed the conditions for participation. Enabling participation in everyday life situations is an important rehabilitation goal after acquired brain injury (ABI). Identifying factors associated with individuals’ experienced participation and problems therein is therefore essential. Objective: This study aimed at exploring the relationship between perceived difficulty in everyday technology use, perceived ability in the activities of daily living (ADL), and perceived participation, and participation problems in persons with ABI. Methods: Eighty-one persons with ABI participated in the study and were assessed by the Impact on Participation and Autonomy questionnaire, the Everyday Technology Use Questionnaire, and the ADL taxonomy. Results: Findings showed that the combined model of difficulty in everyday technology (ET) use, ADL ability, and the interaction between them explained both participation in various domains of everyday life, and also overall level of perceived participation and the perceived problems. Conclusions: The findings underscore the importance of evaluating individuals’ ability in both ET use and ADL after ABI to increase the probability of explaining these persons’ participation in desired everyday life situations and, also, for rehabilitation design.     

Inter-rater and test-retest reliability of the Taiwanese Rehabilitation Functional Scale

The need of a standardized evaluation tool for clinical practice is acknowledged by occupational therapists worldwide. The purpose of this study was to examine the inter-rater and test-retest reliabilities of the Taiwanese Rehabilitation Functional Scale (TaRFS) developed in Taiwan. Seventy-five subjects with varying diagnoses were recruited to participate in the study. They came from an inpatient re-habilitation unit within a university-affiliated hospital. All participants were tested on a functional measure of their independence on activities of daily living and cognitive-social tasks, and of their sense of well-being. The Activity of Daily Living (ADL) and CognitiveSocial Skills (CSS) subscales of the TaRFS were found to be reliable across raters and times. But individual items on the Subjective Well Being subscale were not stable across time. The results indicated that the ADL subscale as well as most items in the CSS subscale of the TaRFS can be utilized reliably in clinical practice. The SWB subscale of the TaRFS may need further revision in order to be used stably across time. The development of the TaRFS and future revision of the scale can be beneficial for occupational therapists in Taiwan to provide evidence of their intervention efficacy.     

Inter-rater and test-retest reliability of three contingent valuation question formats in south-east Nigeria

This paper examines the inter-rater and test-retest reliability of willingness to pay (WTP) for insecticide-treated mosquito nets and net re-treatment using the bidding game (BG), binary with follow-up (BWFU) and a novel structured haggling technique (SH). Inter-rater reliability was evaluated by having two sets of interviewers administer questionnaires to 109 (BG), 110 (BWFU) and 103 (SH) randomly selected household heads. Test-retest reliability was investigated by repeating interviews on 146 (BG), 161 (BWFU) and 139 (SH) household heads one month after an initial survey. Data analysis used testing of means, Spearman's correlation and Pearson's correlation coefficient for test of reliability, while non-parametric analysis was used to determine factors causing a variation in WTP. The study was conducted in Southeast Nigeria. Inter-rater reliability coefficients were estimated for the individual's WTP for own nets, WTP for others and WTP for re-treatment. Using WTP for own nets as the best reliability estimate, the coefficients were high at values of 0.77 (C.I. 0.72-0.86), 0.75 (C.I. 0.64-0.81) and 0.74 (C.I. 0.63-0.82) in the BG, BWFU and SH, respectively. In test-retest reliability coefficients, the coefficients for WTP for own nets were low-to-moderate at values of 0.51 (C.I. 0.40-0.62), 0.41 (C.I. 0.28-0.53) and 0.56 (C.I. 0.41-0.65) for the BG, BWFU and SH groups, respectively. Factors such as gender, change in income, unplanned expenditures, stated WTP in first survey, time-to-think, external information, and subjecting respondents to more than one interview explained the lower test-retest reliability coefficients. We conclude that the CVM was reliable in the study area and the question formats had similar levels of reliability. The lower coefficients in the test-retest reliability were due to the influence of factors affecting demand that had changed in the intervening period. Standard formats for determining reliability within CVM should be developed for easy comparison of results from different studies.     

Return to work in people with acquired brain injury: association with observed ability to use everyday technology

Purpose: The aim of this study was to explore how the observed ability to use everyday technology (ET), intrapersonal capacities and environmental characteristics related to ET use contributes to the likelihood of return to work in people with ABI. The aim was also to explore whether these variables added to the likelihood of return to work to earlier defined significant variables in the group: age, perceived ADL ability and perceived ability in ET use.

Method: A cross-sectional study. The Management of Everyday Technology Assessment (META), the short version of the Everyday Technology Use Questionnaire (S-ETUQ) and a revised version of the ADL taxonomy were used to evaluate 74 people with ABI. Individual ability measures from all assessments were generated by Rasch analyses and used for additional statistical analysis.

Results: The univariate analyses showed that the observed ability to use ET, as well as intrapersonal capacities and environmental characteristics related to ET use were all significantly associated with returning to work. In the multivariate analyses, none of these associations remained.

Conclusion: The explanatory precision of return to work in people with ABI increased minimally by adding the observed ability to use ET and the variables related to ET use when age, perceived ability in ET use and ADL had been taken in account.     

The challenges of everyday technology in the workplace for persons with acquired brain injury

Abstract

Purpose: To explore and describe how persons with an acquired brain injury (ABI) managed the everyday technology (ET) that they needed to use in their workplace and how this use influences their opportunities to work. Methods: Nine persons with an ABI were interviewed and observed when managing ET in their workplace. The data were analysed qualitatively with a constant comparative method. Results: The main category, “The challenge of managing ET in the workplace”, consisted of three categories, all of which reflected different kinds of discrepancies between the participants' ability to manage ET and the demands that ET imposes on them in work: “Struggling with ET to be able to continue to work; “Depending on strategies to cope with ET to continue in a particular profession”, and “Managing ET at work but concerned about keeping up with the changes”. Conclusions: The result revealed discrepancies between the abilities of the persons with ABI to manage ET in relation to the demands that technology imposed on them in their work setting. This indicated that professionals need to consider the role of ET when designing interventions supporting a person's return to work after an ABI.     

Experiences of mobility device use over time: A multiple case study among very old Latvian women

Aim: To explore the experiences over time of using mobility devices (MDs) among very old women in the context of everyday life in Latvia. Methods: This study utilized a multiple case-study design that employed an explanatory mixed-methods approach. A combination of quantitative survey data on home and health and qualitative interview data for each participant were used to create three narratives to describe the experiences of MD use in everyday life over time, followed by a cross-case analysis. Results: The three cases illustrate that very old women accept and use MDs due to walking difficulties and related needs. Over time, functional decline combined with physical environmental barriers and changes in their social networks limited the supportive role that available MDs could offer these women. Conclusions: The findings contribute to the understanding of the complexity of MD use in everyday life among very old women in Latvia and the need for different kinds of MDs. These findings illustrate the importance of considering the interaction between people’s everyday life situation and their physical environment for their optimal use of MDs. This is important information for policy-makers who can optimize the services for old people in Latvia and support the need for occupational therapists’ professional competence for the planning and development of MD provision.     

Psychometric evaluation of the Spanish version of the Children's Assessment of Participation and Enjoyment and Preferences for Activities of Children

Culturally sensitive and valid assessments of participation in occupations are needed for Spanish‐speaking children to enhance services offered to this population. The purpose of this study was to assess the validity and reliability of the newly developed Spanish version of the Children's Assessment of Participation and Enjoyment and Preferences for Activities of Children, the Evaluación de Participacion y Disfrute de los Niños y Preferencias de las Actividades de los Niños (EPDN/PAN). The sample included 249 children with and without disabilities, aged 6 to 15 years. Statistically significant differences (p < 0.05) in the participation of children with disabilities and children without disabilities, and between younger and older children were found. No statistical significant differences were found between males and females in the dimensions of participation assessed. Adequate internal consistency (0.70–0.92) was found for the PAN scale and the Overall and Informal domain scores of the EPDN‐Intensity scale. Further study is needed to evaluate the formal domain scale and gender differences by activity types. Copyright © 2008 John Wiley & Sons, Ltd.     

Translating the Canadian Occupational Performance Measure to Danish,addressing face and content validity

Background: The Canadian Occupational Performance Measure (COPM) is a recognized assessment tool within Danish occupational therapy (OT), and translated versions of the COPM have been implemented in Danish OT practice. However, a rigorous translation and cross-cultural validation process has not been applied to these versions.

Aims: To produce a new Danish translation of the COPM evaluating its semantic, conceptual, operational and item equivalence while addressing its face and content validity.

Methods: An iterative, multistage translation process was undertaken with (1) forward translation involving professional translators and bilingual OTs, (2) pilot testing and cognitive debriefing interviews with 15 OTs and 37 clients which were analyzed quantitatively and qualitatively, and (3) finalization with adaptations, back translation and approval.

Results: Content validity of COPM in terms of appropriateness, acceptability, intelligibility and comprehensiveness was found in 78–100% of the cases, and its ability to ensure a client-centred focus in the assessment process was confirmed. However, issues concerning the administration and content of the COPM were identified. Subsequently, two recommendations regarding the administration of the COPM were added to the Danish version.

Conclusion: Semantic, conceptual, operational and aspects of item equivalence of the COPM into Danish were achieved and supported by face and content validity.     

Barriers to the use of assistive technology for children with multiple disabilities

Assistive technology has aided children with multiple disabilities to improve access and participation in their school and home environments. Effective educational outcomes from assistive technology use are dependent upon a coordinated assessment and implementation process. The literature on assistive technology with children was reviewed in order to identify current barriers to its effective integration within schools. These barriers were found to include lack of appropriate staff training and support, negative staff attitudes, inadequate assessment and planning processes, insufficient funding, difficulties procuring and managing equipment, and time constraints. A team model for assistive technology assessment and planning is proposed to optimize the educational goal achievement of children with multiple disabilities. Such a model can help target the allocation of occupational therapy resources in schools to best promote educational and broader functional outcomes from assistive technology use.     

Reliability and internal consistency of the Danish version of Loewenstein Occupational Therapy Cognitive Assessment 2nd Edition (LOTCA-II/D)

Abstract

Objective: To evaluate the intra-rater and inter-rater reliability and internal consistency of the Danish version of the Loewenstein Occupational Therapy Cognitive Assessment, second edition (LOTCA-II/D), an instrument designed to assess and evaluate cognitive functioning. Methods: A reliability study was conducted at neurological acute care and rehabilitation units at three Danish hospitals. Fifty-four neurological patients participated in the study. Patients were tested with LOTCA-II/D by occupational therapists, and video recordings were made for reliability assessments. Intra-rater assessments were performed at least two weeks after the first assessment, and all occupational therapists were blinded to previous assessments. Intra-rater and inter-rater reliability were evaluated using percentage agreement and weighted kappa. Internal consistency was evaluated using Cronbach’s alpha, and floor and ceiling effects were evaluated by calculating the proportion of observations in minimum and maximum categories of subtests. Results: The results showed a lack of intra-rater and inter-rater reliability and a large variation of agreement due to a substantial ceiling effect. Internal consistency was below the acceptable value in half of the cognitive areas. Conclusion: The study found that LOTCA-II/D is not reliable in acute clinical practice, and it could thus not be validated for use in this setting.     

Reliability of the Danish version of the McGill Ingestive Skills Assessment for observation-based measures during meals

Abstract

Aim: To establish measurement equivalence in terms of reliability of the Danish version of the Canadian McGill Ingestive Skills Assessment (MISA) for use by occupational therapists. Methods: A cross-sectional two-rater and test–retest design was applied. A total of 102 elderly medical patients were included consecutively, and were video-recorded during a meal. Raters were paired randomly for each video-case, which was re-scored within three to eight weeks. Reliability was evaluated with the intra-class correlation coefficients (ICC), the standard error of measurement (SEM), the smallest detectable change (SDC), and limits of agreement (LOA). Results: Inter-rater reliability was good to excellent (ICC _1.1 0.61–0.84) and intra-rater reliability was excellent (ICC _3.1 0.84–0.93). For the total scale, SEM was 7% between raters and 4% in repeated measurement by the same rater. For the absolute total scale range on 86 points, the SDC was 15.8 between raters and 10.3 in repeated measurement by the same rater. Conclusions: The reliability of the Danish MISA equals the original version and is suitable for clinical practice. When extending the evaluation of the reproducibility, weaker precision was evident when measurements are repeated by different raters than by the same rater. Therefore further investigation of rater effects is recommended.     

Developing a Danish version of the “Impact on Participation and Autonomy Questionnaire”

Abstract

Objective: To translate the “Impact on Participation and Autonomy Questionnaire” into Danish (IPAQ-DK), and estimate its internal consistency and test–retest reliability in order to promote participation-based interventions and research. Design: Translation and two successive reliability assessments through test–retest. Subjects: 137 adults with varying degrees of impairment; of these, 67 participated in the final reliability assessment. Methods: The translation followed guidelines set forth by the “European Group for Quality of Life Assessment and Health Measurement”. Internal consistency for subscales was estimated by Chronbach's alpha. Weighted kappa coefficients and intraclass correlation coefficients were calculated to assess the test–retest reliability at item and subscale level, respectively. Results: A preliminary reliability assessment revealed residual issues regarding the translation and cultural adaptation of the instrument. The revised version (IPAQ-DK) was subsequently subjected to a similar assessment demonstrating Chronbach's alpha values from 0.698 to 0.817. Weighted kappa ranged from 0.370 to 0.880; 78% of these values were higher than 0.600. The intraclass correlation coefficient covered values from 0.701 to 0.818. Conclusion: IPAQ-DK is a useful instrument for identifying person-perceived participation restrictions and satisfaction with participation. Further studies of IPAQ-DK's floor/ceiling effects and responsiveness to change are recommended, and whether there is a need for further linguistic improvement of certain items.     

Understanding the multiple realities of everyday life: Basic assumptions in focus-group methodology

In recent years, there has been a notable growth in the use of focus groups within occupational therapy. It is important to understand what kind of knowledge focus-group methodology is meant to acquire. The purpose of this article is to create an understanding of the basic assumptions within focus-group methodology from a theory of science perspective in order to elucidate and encourage reflection on the paradigm. This will be done based on a study of contemporary literature. To further the knowledge of basic assumptions the article will focus on the following themes: the focus-group research arena, the foundation and its core components; subjects, the role of the researcher and the participants; activities, the specific tasks and procedures. Focus-group methodology can be regarded as a specific research method within qualitative methodology with its own form of methodological criteria, as well as its own research procedures. Participants construct a framework to make sense of their experiences, and in interaction with others these experiences will be modified, leading to the construction of new knowledge. The role of the group leader is to facilitate a fruitful environment for the meaning to emerge and to ensure that the understanding of the meaning emerges independently of the interpreter. Focus-group methodology thus shares, in the authors’ view, some basic assumptions with social constructivism.     

Occupational therapists' skills and attitudes regarding use of computers and assistive technology

The purpose of this study was to examine the skills and attitudes of a sample of practicing occupational therapists regarding the use of computers, and to explore their agreement with the Foundation Level Technology Competencies proposed for adoption in the United States of America (American Occupational Therapy Association, 1991; reported in Hammel and Smith, 1993). Out of a sample of 250 subjects, 89 occupational therapists in the USA completed a questionnaire regarding their experiences with, and attitudes towards computers, and their agreement with the proposed Competencies. The results indicated that while the majority of therapists had had some exposure to computers, they wanted to learn more—particularly how a computer could aid management tasks. While 50% or more of the therapists ‘Agreed’ or ‘Strongly Agreed’ with 19 of the 21 Competencies, the level of therapist ‘Indecision’ or ‘Disagreement’ with several of the Competencies suggests that either these Competencies require revision, or that if the Competencies are nationally adopted, many therapists may require intensive continuing education programmes to bring their skills to a competent standard. Adoption of these Competencies has many implications for international standards of occupational therapy practice and levels of computer education. Copyright © 1997 Whurr Publishers Ltd.     

The validity and reliability of the modified version of the Role Checklist (M-RCL)

Abstract

Background: The Role Checklist assesses the number of roles a person performs, and the values ascribed to those roles. In this study, a modified version of the Role Checklist was created, which also measures role load. Objective: The purpose of this study is to examine the discriminant validity and test–retest reliability of the modified version. Method: 30 healthy women were divided into two groups: Mothers and women with no children. The original Role Checklist (RCL) and the modified version (M-RCL) were administered and compared. The modified questionnaire was administered again after two weeks. Results: Test–retest reliability was found for all but two roles, and ranged between 0.268–0.709; 0.248–0.629; and 0.336–0.648 for the past, present, and future respectively. Validity: The modified version and the original version showed strong correlation. Pearson correlations were high for the past (r = 0.923, p = 0.001), present (r = 0.1, p = 0.001), and future (r = 0.841, p = 0.001). In addition, significant statistical differences between mothers and non-mothers were found. Conclusion: The modified version of the RCL exhibits initial reliability and validity. Occupational therapists can use the M-RCL to learn about the individual's occupational roles and role load.