Methodological framework for evaluating clinical processes: A cognitive informatics perspective

We propose a methodological framework for evaluating clinical cognitive activities in complex real-world environments that provides a guiding framework for characterizing the patterns of activities. This approach, which we refer to as a process-based approach, is particularly relevant to cognitive informatics (CI) research—an interdisciplinary domain utilizing cognitive approaches in the study of computing systems and applications—as it provides new ways for understanding human information processing, interactions, and behaviors. Using this approach involves the identification of a process of interest (e.g., a clinical workflow), and the contributing sequences of activities in that process (e.g., medication ordering). A variety of analytical approaches can then be used to characterize the inherent dependencies and relations within the contributing activities within the considered process. Using examples drawn from our own research and the extant research literature, we describe the theoretical foundations of the process-based approach, relevant practical and pragmatic considerations for using such an approach, and a generic framework for applying this approach for evaluation studies in clinical settings. We also discuss the potential for this approach in future evaluations of interactive clinical systems, given the need for new approaches for evaluation, and significant opportunities for automated, unobtrusive data collection.     

Use of a human factors approach to uncover informatics needs of nurses in documentation of care

PurposeThe success of health information technology implementations is often tied to the impact the technical system will have on the work of the clinicians using them. Considering the role of nurses in healthcare, it is shocking that there is a lack of evaluations of nursing information systems in the literature. Here we report on how a human factors approach can be used to address barriers and facilitators to use of the nursing information system (NIS). Human factors engineering (HFE) approaches provide the theoretical and methodological underpinning to address these socio-technical issues.MethodsThis study investigated the use of an NIS, a module of the electronic health record (EHR) previously implemented throughout the hospital system. The study took place in two hospitals (760 beds and 300 beds) within a three-hospital health system. Earlier in the year, the NIS was implemented throughout the health system. We applied a scenario-based evaluation technique in order to understand the barriers and facilitators to nurse use of the NIS as part of improving the healthcare delivery system. The scenarios were designed to have the nurses interact with the major components of the NIS. The research team developed the standardized scenarios to cover the major functions of the system.ResultsTwelve nurses completed the study and results show that documentation within the NIS was hindered by several aspects of the interface. This paper discusses the themes associated with the usability of the NIS interface analyzing them using usability heuristics. The team also identified facilitators to use and proposed avenues to support or enhance these facilitators.ConclusionsThis study examined the use of an NIS to standardize care and documentation in nursing. It used scenario-based usability testing, applying the “think-aloud” protocol technique to assess the use of the NIS in documenting patient care. This method of usability evaluation exposed an understanding of how nurses use the NIS and their perspective on the system. We hypothesize that this method will offer key insights into how the usability of the NIS not only impacts use but also informs redesign opportunities. In addition, this is one of the few rigorous studies of NIS and provides direction and recommendations for informaticians, developers and nurse decision makers.     

Cognitive engineering and health informatics: Applications and intersections

Cognitive engineering is an applied field with roots in both cognitive science and engineering that has been used to support design of information displays, decision support, human-automation interaction, and training in numerous high risk domains ranging from nuclear power plant control to transportation and defense systems. Cognitive engineering provides a set of structured, analytic methods for data collection and analysis that intersect with and complement methods of Cognitive Informatics. These methods support discovery of aspects of the work that make performance challenging, as well as the knowledge, skills, and strategies that experts use to meet those challenges. Importantly, cognitive engineering methods provide novel representations that highlight the inherent complexities of the work domain and traceable links between the results of cognitive analyses and actionable design requirements. This article provides an overview of relevant cognitive engineering methods, and illustrates how they have been applied to the design of health information technology (HIT) systems. Additionally, although cognitive engineering methods have been applied in the design of user-centered informatics systems, methods drawn from informatics are not typically incorporated into a cognitive engineering analysis. This article presents a discussion regarding ways in which data-rich methods can inform cognitive engineering.     

Designing and evaluating an electronic patient falls reporting system: Perspectives for the implementation of health information technology in long-term residential care facilities

Background and purposePatient falls are the leading cause of unintentional injury and death among older adults. In 2000, falls resulted in over 10,300 elderly deaths, costing the United States approximately $179 million in incidence and medical costs. Furthermore, non-fatal injuries caused by falls cost the United States $19 billion annually. Health information technology (IT) applications, specifically electronic falls reporting systems, can aid quality improvement efforts to prevent patient falls. Yet, long-term residential care facilities (LTRCFs) often do not have the financial resources to implement health IT, and workers in these settings are often not ready to adopt such systems. Additionally, most health IT evaluations are conducted in large acute-care settings, so LTRCF administrators currently lack evidence to support the value of health IT.MethodsIn this paper, we detail the development of a novel, easy-to-use system to facilitate electronic patient falls reporting within a LTRCF using off-the-shelf technology that can be inexpensively implemented in a wide variety of settings. We report the results of four complimentary system evaluation measures that take into consideration varied organizational stakeholders’ perspectives: (1) System-level benefits and costs, (2) system usability, via scenario-based use cases, (3) a holistic assessment of users’ physical, cognitive, and marcoergonomic (work system) challenges in using the system, and (4) user technology acceptance. We report the viability of collecting and analyzing data specific to each evaluation measure and detail the relative merits of each measure in judging whether the system is acceptable to each stakeholder.Results and conclusionsThe electronic falls reporting system was successfully implemented, with 100% reporting at 3-months post-implementation. The system-level benefits and costs approach showed that the electronic system required no initial investment costs aside from personnel costs and significant benefits accrued from user time savings. The usability analysis revealed several fixable design flaws and demonstrated the importance of scenario-based user training. The technology acceptance model showed that users perceived the reporting system to be useful and easy to use, even more so after implementation. Finally, the holistic human factors evaluation identified challenges encountered when nurses used the system as a part of their daily work, guiding further system redesign. The four-pronged evaluation framework accounted for varied stakeholder perspectives and goals and is a highly scalable framework that can be easily applied to health IT implementations in other LTRCFs.     

Expanding multi-disciplinary approaches to healthcare information technologies: what does information systems offer medical informatics?

The effective use of information technology (IT) is a crucial component for the delivery of effective services in health care. Current approaches to medical informatics (MI) research have significantly contributed to the success of IT use in health care but important challenges remain to be addressed. We believe that expanding the multi-disciplinary basis for MI research is important to meeting these research challenges. In this paper, we outline theories and methods used in information systems (IS) research that we believe can inform our understanding of health care IT applications and outcomes. To do so, we discuss some general differences in the focus and methods of MI and IS research to identify broad opportunities. We then review conceptual and methodological approaches in IS that have been applied in health care IT research. These include: technology-use mediation, collaborative work, genre theory, interpretive research, action research, and modeling. Examples of these theories and methods in healthcare IS research are illustrated.     

Usability in the real world: assessing medical information technologies in patients' homes

OBJECTIVE: This paper presents an approach to usability evaluation of computer-based health care systems designed for patient use in their homes. Although such devices are becoming more prevalent, there is very little known about their usability. DESIGN: The theoretical foundations for the methods are discussed. The approach incorporates a cognitive walkthrough usability evaluation and new methods for usability testing that can be conducted in patient's homes. The method was applied to the IDEATel intervention, a multi-institution randomized controlled trial of the feasibility, acceptability, and clinical utility of a home-based telemedicine system for diabetic Medicare population. The usability study was designed to assess barriers to optimal use of the system. The focus was both on dimensions of the interface and on dimensions of patient skills and competency. The usability field research involved testing 25 patients in their homes using the system. The analysis included a range of video-analytic methods of varying levels of granularity. RESULTS: The usability evaluation revealed aspects of the interface that were sub-optimal and impeded the performance of certain tasks. It also found a range of patient-related factors such as numeracy and psychomotor skills that constituted barriers to productive use. CONCLUSIONS: A multifaceted usability approach provided important insight regarding use of technology by an elderly chronic-care patient population and more generally, for understanding how home health initiatives can more effectively use such technology.     

Digital video analysis of health professionals’ interactions with an electronic whiteboard: A longitudinal,naturalistic study of changes to user interactions

As hospital departments continue to introduce electronic whiteboards in real clinical settings a range of human factor issues have emerged and it has become clear that there is a need for improved methods for designing and testing these systems. In this study, we employed a longitudinal and naturalistic method in the usability evaluation of an electronic whiteboard system. The goal of the evaluation was to explore the extent to which usability issues experienced by users change as they gain more experience with the system. In addition, the paper explores the use of a new approach to collection and analysis of continuous digital video recordings of naturalistic “live” user interactions. The method developed and employed in the study included recording the users’ interactions with system during actual use using screen-capturing software and analyzing these recordings for usability issues. In this paper we describe and discuss both the method and the results of the evaluation. We found that the electronic whiteboard system contains system-related usability issues that did not change over time as the clinicians collectively gained more experience with the system. Furthermore, we also found user-related issues that seemed to change as the users gained more experience and we discuss the underlying reasons for these changes. We also found that the method used in the study has certain advantages over traditional usability evaluation methods, including the ability to collect analyze live user data over time. However, challenges and drawbacks to using the method (including the time taken for analysis and logistical issues in doing live recordings) should be considered before utilizing a similar approach. In conclusion we summarize our findings and call for an increased focus on longitudinal and naturalistic evaluations of health information systems and encourage others to apply and refine the method utilized in this study.     

Incorporating ideas from computer-supported cooperative work

Many information systems have failed when deployed into complex health-care settings. We believe that one cause of these failures is the difficulty in systematically accounting for the collaborative and exception-filled nature of medical work. In this methodological review paper, we highlight research from the field of computer-supported cooperative work (CSCW) that could help biomedical informaticists recognize and design around the kinds of challenges that lead to unanticipated breakdowns and eventual abandonment of their systems. The field of CSCW studies how people collaborate with each other and the role that technology plays in this collaboration for a wide variety of organizational settings. Thus, biomedical informaticists could benefit from the lessons learned by CSCW researchers. In this paper, we provide a focused review of CSCW methods and ideas-we review aspects of the field that could be applied to improve the design and deployment of medical information systems. To make our discussion concrete, we use electronic medical record systems as an example medical information system, and present three specific principles from CSCW: accounting for incentive structures, understanding workflow, and incorporating awareness.     

Evaluation: salvation or nemesis of medical informatics?

The currently prevailing paradigms of evaluation in medical/health informatics are reviewed. Some problems with application of the objectivist approach to the evaluation of real-rather than simulated-(health) information systems are identified. The rigorous application of the objectivist approach, which was developed for laboratory experiments, is difficult to adapt to the evaluation of information systems in a practical real-world environment because such systems tend to be complex, changing rapidly over time, and often existing in a variety of variants. Practical and epistemological reasons for the consequent shortcomings of the objectivist approach are detailed. It is argued that insistence on the application of the objectivist principles to real information systems may hamper rather than advance insights and progress because of this. Alternatives in the form of the subjectivist approach and extensions to both the objectivist and subjectivist approaches that circumvent the identified problems are summarized. The need to include systems engineering approaches in, and to further extend, the evaluation methodology is pointed out.     

Usability in telemedicine systems—A literature survey

IntroductionThe rapid development of sensors and communication technologies enable the growth of new innovative services in healthcare, such as Telemedicine. An essential ingredient in the development of a telemedicine system and its final acceptance by end users are usability studies. The principles of usability engineering, evaluations and telemedicine are well established, and it may contribute to the adoption and eventually deployment of such systems and services. An in-depth usability analysis, including performance and attitude measures, requires knowledge about available usability techniques, and is depending on the amount of resources. Therefore it is worth investigating how usability methods are applied in developing telemedicine systems. Our hypothesis is: with increasing research and development of telemedicine systems, we expect that various usability methods are more equally employed for different end-user groups and applications.MethodA literature survey was conducted to find telemedicine systems that have been evaluated for usability or ease of use. The elements of the PICO framework were used as a basis for the selection criteria in the literature search. The search was not limited by year. Two independent reviewers screened all search results first by title, and then by abstract for inclusion. Articles were included up to May 2015.ResultsIn total, 127 publications were included in this survey. The number of publications on telemedicine systems significantly increased after 2008. Older adults and end-users with cardiovascular conditions were among largest target end-user groups. Remote monitoring systems were found the most, in 90 publications. Questionnaires are the most common means for evaluating telemedicine systems, and were found in 88 publications. Questionnaires are used frequently in studies focusing on cardiovascular diseases, Parkinson’s disease and older adult conditions. Interviews are found the most in publications related to stroke. In total 71% of the publications were trial-orientated and the remaining process orientated. An increase in telemedicine research, development and applications is found worldwide, with the majority of publications conducted in America.Discussion and conclusionMonitoring patients in their homes can lead to better healthcare at lower costs which implies an increased demand of new healthcare strategies like telemedicine. We expected that with the increase in telemedicine research and development, a greater range of usability methods would also be employed in the included publications. This is not the case. Researchers employed questionnaires as a preferred usability method for each type of telemedicine system and most end-users. However, in process-orientated studies a greater range of usability evaluations were applied, with fewer differences found in the amount of publications for each evaluation method. Questionnaires enable researchers to evaluate a system quickly on end users, as it requires less expertise on the evaluation method compared to the other methods. They are easily distributed and are customizable. The use of questionnaires is therefore an evaluation method of choice for a variety of telemedicine systems and end-users.     

Empirical distributional semantics: Methods and biomedical applications

Over the past 15 years, a range of methods have been developed that are able to learn human-like estimates of the semantic relatedness between terms from the way in which these terms are distributed in a corpus of unannotated natural language text. These methods have also been evaluated in a number of applications in the cognitive science, computational linguistics and the information retrieval literatures. In this paper, we review the available methodologies for derivation of semantic relatedness from free text, as well as their evaluation in a variety of biomedical and other applications. Recent methodological developments, and their applicability to several existing applications are also discussed.     

Advancing cognitive engineering methods to support user interface design for electronic health records

BackgroundDespite many decades of research on the effective development of clinical systems in medicine, the adoption of health information technology to improve patient care continues to be slow, especially in ambulatory settings. This applies to dentistry as well, a primary care discipline with approximately 137,000 practitioners in the United States. A critical reason for slow adoption is the poor usability of clinical systems, which makes it difficult for providers to navigate through the information and obtain an integrated view of patient data.ObjectiveIn this study, we documented the cognitive processes and information management strategies used by dentists during a typical patient examination. The results will inform the design of a novel electronic dental record interface.MethodsWe conducted a cognitive task analysis (CTA) study to observe ten general dentists (five general dentists and five general dental faculty members, each with more than two years of clinical experience) examining three simulated patient cases using a think-aloud protocol.ResultsDentists first reviewed the patient's demographics, chief complaint, medical history and dental history to determine the general status of the patient. Subsequently, they proceeded to examine the patient's intraoral status using radiographs, intraoral images, hard tissue and periodontal tissue information. The results also identified dentists’ patterns of navigation through patient's information and additional information needs during a typical clinician–patient encounter.ConclusionThis study reinforced the significance of applying cognitive engineering methods to inform the design of a clinical system. Second, applying CTA to a scenario closely simulating an actual patient encounter helped with capturing participants’ knowledge states and decision-making when diagnosing and treating a patient. The resultant knowledge of dentists’ patterns of information retrieval and review will significantly contribute to designing flexible and task-appropriate information presentation in electronic dental records.     

A development framework for semantically interoperable health information systems

Background

Semantic interoperability is a basic challenge to be met for new generations of distributed, communicating and co-operating health information systems (HIS) enabling shared care and e-Health. Analysis, design, implementation and maintenance of such systems and intrinsic architectures have to follow a unified development methodology.

Methods

The Generic Component Model (GCM) is used as a framework for modeling any system to evaluate and harmonize state of the art architecture development approaches and standards for health information systems as well as to derive a coherent architecture development framework for sustainable, semantically interoperable HIS and their components. The proposed methodology is based on the Rational Unified Process (RUP), taking advantage of its flexibility to be configured for integrating other architectural approaches such as Service-Oriented Architecture (SOA), Model-Driven Architecture (MDA), ISO 10746, and HL7 Development Framework (HDF).

Results

Existing architectural approaches have been analyzed, compared and finally harmonized towards an architecture development framework for advanced health information systems.

Conclusion

Starting with the requirements for semantic interoperability derived from paradigm changes for health information systems, and supported in formal software process engineering methods, an appropriate development framework for semantically interoperable HIS has been provided. The usability of the framework has been exemplified in a public health scenario.     

National efforts to improve health information system safety in Canada,the United States of America and England

ObjectiveIn this paper we review progress as well as challenges encountered in Canada, the United States and England with regard to ensuring safety of health information technology.MethodA review of major programs and initiatives for ensuring safety of health information technology in the three countries was conducted. Published literature and Web resources from national programs were reviewed for relevant information.ResultsIt was found that in all three countries the issue of technology-induced error has been recognized as being of critical importance. The three countries have developed approaches for dealing with the issue that have some commonalities; however, they are at varying different stages of maturity, with England having the longest standing and most well developed safety programs, while Canada and the United States are at earlier stages. The types of approaches employed have included work on developing standards related to usability and interface design, certifications, directives from regulatory bodies, educational initiatives in health information technology (HIT) safety as well as research into safer HIT design and implementation methods.ConclusionsHIT promises to lead to improved patient safety. However, it has become recognized that if not designed and deployed appropriately, such systems can lead to new types of errors. Based on this recognition, a variety of initiatives are being undertaken in Canada, the United States and England to promote the safe design, procurement and deployment of HIT. It is concluded that improved approaches to system design, testing, regulation, error reporting, safety education and cross-country collaboration will be needed to further promote safer HIT.     

Research-IQ: development and evaluation of an ontology-anchored integrative query tool

Investigators in the translational research and systems medicine domains require highly usable, efficient and integrative tools and methods that allow for the navigation of and reasoning over emerging large-scale data sets. Such resources must cover a spectrum of granularity from bio-molecules to population phenotypes. Given such information needs, we report upon the initial design and evaluation of an ontology-anchored integrative query tool, Research-IQ, which employs a combination of conceptual knowledge engineering and information retrieval techniques to enable the intuitive and rapid construction of queries, in terms of semi-structured textual propositions, that can subsequently be applied to integrative data sets. Our initial results, based upon both quantitative and qualitative evaluations of the efficacy and usability of Research-IQ, demonstrate its potential to increase clinical and translational research throughput.