Evaluation in the design of health information systems: application of approaches emerging from usability engineering

This paper examines the role of evaluation in the design of health care information systems. A framework is presented for considering evaluation in the context of software development processes, in particular, the systems development life cycle (SDLC). Variations on standard design methodologies are then discussed, including methods based on rapid development and continual evaluation of prototype systems. Usability testing is presented as a key method for conducting evaluations during iterative system development. The emergence of design methodologies, where evaluation is viewed as a central part of the development cycle is also discussed. Evaluation methodologies are then considered along a continuum, ranging from studies involving a high degree of experimental control to observational approaches. A full cycle approach to evaluation of health care systems is argued for, involving deployment of new methods across the SDLC. Implications for future work exploring the integration of design and evaluation processes in health informatics are discussed.     

Navigation in the electronic health record: A review of the safety and usability literature

ObjectiveInefficient navigation in electronic health records has been shown to increase users’ cognitive load, which may increase potential for errors, reduce efficiency, and increase fatigue. However, navigation has received insufficient recognition and attention in the electronic health record (EHR) literature as an independent construct and contributor to overall usability. Our aims in this literature review were to (1) assess the prevalence of navigation-related topics within the EHR usability and safety research literature, (2) categorize types of navigation actions within the EHR, (3) capture relationships between these navigation actions and usability principles, and (4) collect terms and concepts related to EHR navigation. Our goal was to improve access to navigation-related research in usability.Materials and methodsWe applied scoping literature review search methods with the assistance of a reference librarian to identify articles published since 1996 that reported evaluation of the usability or safety of an EHR user interface via user test, analytic methods, or inspection methods. The 4336 references collected from MEDLINE, EMBASE, Engineering Village, and expert referrals were de-duplicated and screened for relevance, and navigation-related concepts were abstracted from the 21 articles eligible for review using a standard abstraction form.ResultsOf the 21 eligible articles, 20 (95%) mentioned navigation in results and discussion of usability evaluations. Navigation between pages of the EHR was the more frequently documented type of navigation (86%) compared to navigation within a single page (14%). Navigation actions (e.g., scrolling through a medication list) were frequently linked to specific usability heuristic violations, among which flexibility and efficiency of use, recognition rather than recall, and error prevention were most common.DiscussionDiscussion of navigation was prevalent in results across all types of evaluation methods among the articles reviewed. Navigating between multiple screens was frequently identified as a usability barrier. The lack of standard terminology created some challenges to identifying and comparing articles.ConclusionWe observed that usability researchers are frequently capturing navigation-related issues even in articles that did not explicitly state navigation as a focus. Capturing and synthesizing the literature on navigation is challenging because of the lack of uniform vocabulary. Navigation is a potential target for normative recommendations for improved interaction design for safer systems. Future research in this domain, including development of normative recommendations for usability design and evaluation, will be facilitated by development of a standard terminology for describing EHR navigation.     

The usability axiom of medical information systems

INTRODUCTION: In this article we begin by connecting the concept of simplicity of user interfaces of information systems with that of usability, and the concept of complexity of the problem-solving in information systems with the concept of usefulness. We continue by stating "the usability axiom" of medical information technology: information systems must be, at the same time, usable and useful. We then try to show why, given existing technology, the axiom is a paradox and we continue with analysing and reformulating it several times, from more fundamental information processing perspectives. DISCUSSION: We underline the importance of the concept of representation and demonstrate the need for context-dependent representations. By means of thought experiments and examples, we advocate the need for context-dependent information processing and argue for the relevance of algorithmic information theory and case-based reasoning in this context. Further, we introduce the notion of concept spaces and offer a pragmatic perspective on context-dependent representations. We conclude that the efficient management of concept spaces may help with the solution to the medical information technology paradox. Finally, we propose a view of informatics centred on the concepts of context-dependent information processing and management of concept spaces that aligns well with existing knowledge centric definitions of informatics in general and medical informatics in particular. In effect, our view extends M. Musen's proposal and proposes a definition of Medical Informatics as context-dependent medical information processing. SUMMARY: The axiom that medical information systems must be, at the same time, useful and usable, is a paradox and its investigation by means of examples and thought experiments leads to the recognition of the crucial importance of context-dependent information processing. On the premise that context-dependent information processing equates to knowledge processing, this view defines Medical Informatics as a context-dependent medical information processing which aligns well with existing knowledge centric definitions of our field.     

Internet-administered cognitive behavior therapy for health problems: a systematic review

Cognitive-behavioral interventions are the most extensively researched form of psychological treatment and are increasingly offered through the Internet. Internet-based interventions may save therapist time, reduce waiting-lists, cut traveling time, and reach populations with health problems who can not easily access other more traditional forms of treatments. We conducted a systematic review of twelve randomized controlled or comparative trials. Studies were identified through systematic searches in major bibliographical databases. Three studies focused on patients suffering from pain, three on headache, and six on other health problems. The effects found for Internet interventions targeting pain were comparable to the effects found for face-to-face treatments, and the same was true for interventions aimed at headache. The other interventions also showed some effects, although effects differed across target conditions. Internet-delivered cognitive-behavioral interventions are a promising addition and complement to existing treatments. The Internet will most likely assume a major role in the future delivery of cognitive-behavioral interventions to patients with health problems.     

Merits of usability testing for PACS selection

ObjectivesTo compare the usability of different Picture Archiving and Communication System (PACS) workstations, determine whether a usability test has added value with respect to the traditional way of comparing PACSs based on functional requirements, and to evaluate the appropriateness of a task-based methodology for a PACS usability test.MethodsA task-based usability test of four PACS workstations was performed. Radiologists’ subjective responses to the PACSs and their performance on the tasks were measured. To mimic the traditional PACS selection process, two functional requirements were defined which the PACSs met in varying degrees. The focus of the usability test was on the aspects of the PACS related to these requirements. The usability results were compared to the PACSs’ ability to meet the functional requirements.ResultsOne PACS outperformed the other PACSs both in terms of subjective preference and task performance, indicating its superior usability. There were differences in usability between PACSs with identical functionality. Also, a PACS with theoretically advantageous functionality for a given task did not necessarily have better usability for this task than a PACS without this functionality. There was a discrepancy between participants’ subjective preferences and their task performance, which indicates that it is vital to include performance measures in the usability assessment so that it accurately reflects the efficiency of interaction.ConclusionsThe differences in usability between PACSs with identical functionality indicate that functional requirements alone are insufficient to determine a PACS's overall quality. A usability test should therefore be used in addition to a functional requirement list in a PACS selection process to ensure that a hospital buys the PACS with the highest quality. A task-based usability evaluation methodology, which yields both subjective preference data and objective performance data of radiologists interacting with the PACS, is very suitable for such a usability test.     

The outcomes of regional healthcare information systems in health care: A review of the research literature

The resulting regional healthcare information systems were expected to have effects and impacts on health care procedures, work practices and treatment outcomes. The aim is to find out how health information systems have been investigated, what has been investigated and what are the outcomes. A systematic review was carried out of the research on the regional health information systems or organizations. The literature search was conducted on four electronic Cinahl Medline, Medline/PubMed and Cochrane. The common type of study design was the survey research and case study, and the data collection was carried out via different methodologies. They found out different types of regional health information systems (RHIS). The systems were heterogeneous and were in different phases of these developments. The RHIS outcomes focused on the five main areas: flow of information, collaboration, process redesign, system usability and organization culture. The RHIS improved the clinical data access, timely information, and clinical data exchange and improvement in communication and coordination within a region between professionals but also there was inadequate access to patient relevant clinical data. There were differences in organization culture, vision and expectations of leadership and consistency of strategic plan. Nevertheless, there were widespread participation by both healthcare providers and patients.     

The influence of online health information on health decisions: A systematic review

ObjectivesThis systematic review assessed the influence of online health information (OHI) search behaviour on health and medical decisions.MethodsEligible studies were identified by searching electronic databases PubMed, Scopus, and CINAHL in February 2020 for studies reporting OHI search behaviour and its influence on health decisions. Information was extracted pertaining to either consumers’ (self-reported) perceptions of the influence of OHI on decision-making or the association between online search behaviour and health decision-making.ResultsA total of 3995 articles were screened, with 48 included in the final analysis. The reviewed studies indicated that OHI assisted in making subsequent health related decisions such as asking questions during a consultation, increased professional visits, improved adherence to the advice of a physician, being more compliant with taking medication, and improved self-care.ConclusionConsumers largely used OHI to support information provided by their physicians. The strength of the patient-provider relationship was considered important in moderating the potential negative outcomes of OHI.Practice implicationsHealth care systems have a unique opportunity to direct OHI search behaviours towards empowering consumers to engage as an informed, active and joint decision-maker in their own health care.     

Examining the role of collaboration in studies of health information technologies in biomedical informatics: A systematic review of 25 years of research

PurposeOur objective was to identify and examine studies of collaboration in relation to the use of health information technologies (HIT) in the biomedical informatics field.MethodsWe conducted a systematic literature review of articles through PubMed searches as well as reviewing a variety of individual journals and proceedings. Our search period was from 1990–2015. We identified 98 articles that met our inclusion criteria. We excluded articles that were not published in English, did not deal with technology, and did not focus primarily on individuals collaborating.ResultsWe categorized the studies by technology type, user groups, study location, methodology, processes related to collaboration, and desired outcomes. We identified three major processes: workflow, communication, and information exchange and two outcomes: maintaining awareness and establishing common ground. Researchers most frequently studied collaboration within hospitals using qualitative methods.DiscussionBased on our findings, we present the “collaboration space model”, which is a model to help researchers study collaboration and technology in healthcare. We also discuss issues related to collaboration and future research directions.ConclusionWhile collaboration is being increasingly recognized in the biomedical informatics community as essential to healthcare delivery, collaboration is often implicitly discussed or intertwined with other similar concepts. In order to evaluate how HIT affects collaboration and how we can build HIT to effectively support collaboration, we need more studies that explicitly focus on collaborative issues.     

Systematic review and meta-analysis of audio-visual information aids for informed consent for invasive healthcare procedures in clinical practice

Objective To systematically review audio-visual (AV) interventions for promoting informed consent (IC) in clinical practice and to consider the impact of reading age adjustment.     

Tolerance of uncertainty: A systematic review of health and healthcare-related outcomes

Background

Uncertainty tolerance (UT) is thought to be a characteristic of individuals that influences various outcomes related to health, healthcare, and healthcare education. We undertook a systematic literature review to evaluate the state of the evidence on UT and its relationship to these outcomes.

Experiences of empathy training in healthcare: A systematic review of qualitative studies

ObjectiveTo synthesise qualitative studies reporting student, practitioner, or patient experiences of empathy-training in healthcare.MethodsWe included qualitative studies exploring (i) student or practitioner experiences of empathy training, or (ii) patient experiences of being treated by someone who has undergone empathy training. We used the Critical Appraisal Skills Programme (CASP) tool to assess study quality. Thematic synthesis was used to integrate findings from studies and to generate new insights.ResultsOur search yielded 2768 citations, of which 23 (1487 participants) met inclusion criteria. Two clusters of themes were identified from included studies. Firstly, themes related to practitioner/trainee professionalism and wellbeing, where the main finding was that participants experienced benefit from therapeutic empathy training. Secondly, themes related to the understanding and treatment of patients, where the main finding was practitioners’ deeper recognition of the positive impact of empathic care.ConclusionsThis review found that taking part in empathy-focused training can benefit practitioner/student personal growth and professional development, and benefits patient care. This review is limited by the difficulty in defining empathy and heterogeneity amongst included studies.Practice implicationsThese results support a rationale for empathy training and the development of a framework to ensure training is having the desired effect.     

A systematic search and review of the discrimination in health care measure,and its adaptations

BackgroundDiscrimination occurs in health care settings contributing to health inequities. Yet guidance on how best to measure discrimination in health care is still limited.ObjectivesWe sought to (1) identify and describe the characteristics of published studies that used the Discrimination in Health Care Measure, a scale first published in 2001; (2) review how the measure has been used or adapted and summarize the measure’s published psychometric properties and its variations; and (3) summarize associations between the measure and health-related variables.MethodsWe performed a systematic search and review of the measure by searching PsycINFO, PubMed, Sociological Abstracts, and Web of Science from January 1, 2001 through January 31, 2017. We screened 260 unique articles, identified 22 eligible articles, and completed a narrative synthesis.ResultsMost studies measured race or ethnicity-based discrimination. All studies made minor revisions to the measure, and most reported high reliabilities. Discrimination in health care, using this measure, was associated with adverse health outcomes.Discussion and practice implicationsStudy results indicate that the measure is easy to use and adapt. Researchers should consider using the Discrimination in Health Care Measure when designing studies that will examine individuals’ discriminatory experiences when receiving health care.     

Psychophysiological outcomes of health qigong for chronic conditions: A systematic review

We aimed to unravel the clinical benefits and the plausible underlying psychophysiological mechanism based on available randomized controlled trials (RCTs). Meta-analysis of 26 RCTs shortlisted from electronic databases from 1997 to 2006 shows that qigong had some effects on increasing the numbers of white blood cells and lymphocytes, stroke volume, peak early transmitral filling velocity, peak late transmitral filling velocity, forced vital capacity, and forced expiratory volume, and, conversely, lowering of total cholesterol, systolic blood pressure, diastolic blood pressure, and depressive mood scores. Explanatory pathways may pertain to stress reduction via nervous, endocrine, and immune systems. Limitations on methodology are discussed and directions for further studies are suggested. Because of its safety, minimal cost, and clinical benefit, health qigong can be advocated as an adjunctive exercise therapy for older people with chronic conditions.     

Communication practices in conversations about sexual health in medical healthcare settings: A systematic review

ObjectiveMany healthcare professionals experience difficulties in discussing sexual health with their patients. The aim of this review was to synthesize results of studies on communication practices in interactions about sexual health in medical settings, to offer healthcare professionals suggestions on how to communicate about this topic.MethodsWe searched for studies using five databases. Reference lists and specialist bibliographies were searched to identify additional studies. We included discourse analytic studies that used recordings of medical consultations.ResultsWe identified five studies that met the inclusion criteria. Findings were synthesized into seven categories of practices deployed by patients and healthcare professionals when talking about sexual health: avoiding delicate terms (1), delaying potentially delicate words and issues (2), using assumptive talk (3), generalized advice-giving (4), deploying patients’ talk (5), depersonalization (6), and patient-initiated advice (7).ConclusionPractices indicate the delicacy associated with discussing sexual health issues, but results also shed light on practices that can help professionals to deal with this delicacy, and to be responsive to patients’ needs and concerns.Practice implicationsFindings will assist healthcare professionals in broaching topics related to sexual health so they can help patients deal with challenges that affect their sexual health and overall well-being.     

User experiences with different regional health information exchange systems in Finland

PurposeThe interest in cross-organizational Health Information Exchange (HIE) is increasing at regional, national and cross-European levels. The purpose of our study was to compare user experiences (usability) of different regional health information exchange system (RHIE) types as well as the factors related to the experienced level of success of different RHIE system types.MethodsA web-based questionnaire was sent to 95% of Finnish physicians aged between 25 and 65 years of age. RHIE systems were mainly available in the public sector and only in certain regions. Those 1693 physicians were selected from the 3929 respondents of the original study, who met these criteria. The preferred means (paper/fax vs. electronic) of cross-organizational HIE, and replies to the 11 questions measuring RHIE success were used as the main dependent variables.ResultsTwo thirds (73%) of the primary care physicians and one third (33%) of the specialized care physicians replied using an electronic RHIE system rather than paper or fax as a primary means of cross-organizational HIE. Respondents from regions where a regional virtual EHR (type 3) RHIE system was employed had used electronic means rather than paper HIE to a larger extend compared to their colleagues in regions where a master patient index-type (type 1) or web distribution model (type 2) RHIE system was used. Users of three local EHR systems preferred electronic HIE to paper to a larger extend than users of other EHR systems. Experiences with an integrated RHIE system (type 3) were more positive than those with other types or RHIE systems.ConclusionsThe study revealed User preferences for the integrated virtual RHIE-system (type 3) over the master index model (type 1) or web distribution model (type 2). Success of individual HIE tasks of writing, sending and reading were impacted by the way these functionalities were realized in the EHR systems. To meet the expectations of increased efficiency, continuity, safety and quality of care, designers of HIE systems also need to take into account the different HIE needs of primary care clinicians and their secondary care colleagues in hospitals. Both national legislation and local interpretations of data protection possibly hinder the full use of any RHIE systems. These findings should be taken into account when designing usable HIE systems. More qualitative research is needed on specific features of those local systems that are associated with positive or negative user experience.     

Knowledge discovery in clinical decision support systems for pain management: A systematic review

ObjectiveThe occurrence of pain accounts for billions of dollars in annual medical expenditures; loss of quality of life and decreased worker productivity contribute to indirect costs. As pain is highly subjective, clinical decision support systems (CDSSs) can be critical for improving the accuracy of pain assessment and offering better support for clinical decision-making. This review is focused on computer technologies for pain management that allow CDSSs to obtain knowledge from the clinical data produced by either patients or health care professionals.Methods and materialsA comprehensive literature search was conducted in several electronic databases to identify relevant articles focused on computerised systems that constituted CDSSs and include data or results related to pain symptoms from patients with acute or chronic pain, published between 1992 and 2011 in the English language. In total, thirty-nine studies were analysed; thirty-two were selected from 1245 citations, and seven were obtained from reference tracking.ResultsThe results highlighted the following clusters of computer technologies: rule-based algorithms, artificial neural networks, nonstandard set theory, and statistical learning algorithms. In addition, several methodologies were found for content processing such as terminologies, questionnaires, and scores. The median accuracy ranged from 53% to 87.5%.ConclusionsComputer technologies that have been applied in CDSSs are important but not determinant in improving the systems’ accuracy and the clinical practice, as evidenced by the moderate correlation among the studies. However, these systems play an important role in the design of computerised systems oriented to a patient's symptoms as is required for pain management. Several limitations related to CDSSs were observed: the lack of integration with mobile devices, the reduced use of web-based interfaces, and scarce capabilities for data to be inserted by patients.     

Adverse health outcomes in vitamin D supplementation trials for depression: A systematic review

BackgroundVitamin D deficiency is a universal risk factor for adverse health outcomes. Since depression is consistently associated with low vitamin D levels as well as several adverse health outcomes, vitamin D supplementation may be especially relevant for depressed persons. This review examines the potential benefits of vitamin D for (somatic) health outcomes in randomised controlled supplementation trials for depression.MethodSystematic literature search to assess whether adverse health outcomes, such as frailty, falls, or cognitive functioning, were included in vitamin D supplementation trials for depression, and whether these outcomes were affected by supplementation. The revised Cochrane tool for assessing risk of bias in randomised trials was used.ResultsThirty-one trials were included. Adverse health outcomes were considered in five studies. Two studies reported some beneficial effect on an adverse health outcome.Conclusions and implicationsWhile depressed persons are at increased risk of vitamin D deficiency, supplementation trials hardly addressed the common negative health consequences of low vitamin D levels as secondary outcome measures. Well-designed trials of the effects of vitamin D supplementation in late-life depression should explore whether adverse health outcomes can be prevented or stabilised, and whether depression benefits from this improvement.     

The alignment of information systems with organizational objectives and strategies in health care

Purpose

The alignment of information systems with organizational objectives and strategies is a key, contemporary challenge to organizations in general and the health care industry in particular. Researchers and managers alike believe that the selection of new information systems to support objectives and strategies focuses the organization on accomplishing its objectives and realizing the value of the investments in the systems. The purpose of this study was to help understand alignment in health care so that health care information systems planners can better achieve it.

Methods

Structured interviews with 15 top information systems managers in health care organizations of various sizes and types inquired about organizational objectives and strategies, the process for choosing new information systems to support those objectives and strategies, and the concomitant facilitating and hindering managerial actions and organizational characteristics.

Results

In addition to identifying and elucidating specific objectives, strategies, processes for choosing new systems, and facilitating and hindering actions and characteristics, the study used the data to characterize a generalized process of alignment in health care organizations.

Conclusions

The study contributes by confirming that alignment is a significant issue in health care organizations, and that such organizations make deliberate efforts to achieve it. The study further contributes by providing tables of actions and characteristics that managers might use as checklists in current and future alignment efforts as well as in generally cultivating broad support for alignment. Finally, it contributes by suggesting future study of alignment's predictors and effects in health care organizations.     

Health at hand: A systematic review of smart watch uses for health and wellness

IntroductionSmart watches have the potential to support health in everyday living by: enabling self-monitoring of personal activity; obtaining feedback based on activity measures; allowing for in-situ surveys to identify patterns of behavior; and supporting bi-directional communication with health care providers and family members. However, smart watches are an emerging technology and research with these devices is at a nascent stage.MethodsWe conducted a systematic review of smart watch studies that engaged people in their use by searching PubMed, Embase, IEEE XPlore and ACM Digital libraries. Participant demographics, device features, watch applications and methods, and technical challenges were abstracted from included studies.ResultsSeventy-three studies were returned in the search. Seventeen studies published were included. Included studies were published from 2014 to 2016, with the exception of one published in 2011. Most studies employed the use of consumer-grade smart watches (14/17, 82%). Patient-related studies focused on activity monitoring, heart rate monitoring, speech therapy adherence, diabetes self-management, and detection of seizures, tremors, scratching, eating, and medication-taking behaviors. Most patient-related studies enrolled participants with few exclusion criteria to validate smart watch function (10/17, 58%). Only studies that focused on Parkinson’s disease, epilepsy, and diabetes management enrolled persons living with targeted conditions. One study focused on nursing work in the ICU and one focused on CPR training for laypeople.ConclusionConsumer-grade smart watches have penetrated the health research space rapidly since 2014. Smart watch technical function, acceptability, and effectiveness in supporting health must be validated in larger field studies that enroll actual participants living with the conditions these devices target.