With how many users should you test a medical infusion pump? Sampling strategies for usability tests on high-risk systems

Usability testing is recognized as an effective means to improve the usability of medical devices and prevent harm for patients and users. Effectiveness of problem discovery in usability testing strongly depends on size and representativeness of the sample. We introduce the late control strategy, which is to continuously monitor effectiveness of a study towards a preset target.A statistical model, the LNB_zt model, is presented, supporting the late control strategy. We report on a case study, where a prototype medical infusion pump underwent a usability test with 34 users. On the data obtained in this study, the LNB_zt model is evaluated and compared against earlier prediction models.The LNB_zt model fits the data much better than previously suggested approaches and improves prediction. We measure the effectiveness of problem identification, and observe that it is lower than is suggested by much of the literature. Larger sample sizes seem to be in order. In addition, the testing process showed high levels of uncertainty and volatility at small to moderate sample sizes, partly due to users’ individual differences. In reaction, we propose the idiosyncrasy score as a means to obtain representative samples. Statistical programs are provided to assist practitioners and researchers in applying the late control strategy.     

Digital video analysis of health professionals’ interactions with an electronic whiteboard: A longitudinal,naturalistic study of changes to user interactions

As hospital departments continue to introduce electronic whiteboards in real clinical settings a range of human factor issues have emerged and it has become clear that there is a need for improved methods for designing and testing these systems. In this study, we employed a longitudinal and naturalistic method in the usability evaluation of an electronic whiteboard system. The goal of the evaluation was to explore the extent to which usability issues experienced by users change as they gain more experience with the system. In addition, the paper explores the use of a new approach to collection and analysis of continuous digital video recordings of naturalistic “live” user interactions. The method developed and employed in the study included recording the users’ interactions with system during actual use using screen-capturing software and analyzing these recordings for usability issues. In this paper we describe and discuss both the method and the results of the evaluation. We found that the electronic whiteboard system contains system-related usability issues that did not change over time as the clinicians collectively gained more experience with the system. Furthermore, we also found user-related issues that seemed to change as the users gained more experience and we discuss the underlying reasons for these changes. We also found that the method used in the study has certain advantages over traditional usability evaluation methods, including the ability to collect analyze live user data over time. However, challenges and drawbacks to using the method (including the time taken for analysis and logistical issues in doing live recordings) should be considered before utilizing a similar approach. In conclusion we summarize our findings and call for an increased focus on longitudinal and naturalistic evaluations of health information systems and encourage others to apply and refine the method utilized in this study.     

胃癌移动医疗应用程序可用性评价的文献研究

目的通过检索国内外胃癌移动医疗应用程序的相关文献,了解该领域的研究现状和发展动态,并对可用性评价进行重点分析,以期为今后相关研究提供参考。方法检索2010年1月至2020年4月发表的胃癌移动医疗应用程序相关的中、英文文献,检索的中文数据库包括中国知网、万方、重庆维普数据库和中国生物医学文献数据库,英文数据库包括Web of Science、Pub Med、Embase、Medline、ScienceDirect、Springer Link及Cochrane Library等。结果经过文献筛选,本研究共纳入15篇英文文献。胃癌移动医疗应用程序的功能以自我监测、疼痛管理、健康教育、营养咨询和监测以及加强体育锻炼为主。应用程序的可用性评价以量性评价为主,评价方法多采用问卷调查的方式进行,评价对象大多针对目标患者用户,评价持续时间差异较大,可用性评价要素以可用性、可行性、有效性和满意度为主。结论胃癌移动医疗应用程序相关研究呈持续性增加趋势,仅适用于胃癌患者的移动医疗应用程序相对不足。应用程序设计时应在尽可能考虑胃癌患者全面需求的同时注重个性化服务,应用程序的可用性评价应贯穿应用程序研发的全过程,且如何规范实施移动医疗应用程序可用性评价应受到相关研究人员的重视。     

Usability flaws of medication-related alerting functions: A systematic qualitative review

IntroductionMedication-related alerting functions may include usability flaws that limit their optimal use. A first step on the way to preventing usability flaws is to understand the characteristics of these usability flaws. This systematic qualitative review aims to analyze the type of usability flaws found in medication-related alerting functions.MethodPapers were searched via PubMed, Scopus and Ergonomics Abstracts databases, along with references lists. Paper selection, data extraction and data analysis was performed by two to three Human Factors experts. Meaningful semantic units representing instances of usability flaws were the main data extracted. They were analyzed through qualitative methods: categorization following general usability heuristics and through an inductive process for the flaws specific to medication-related alerting functions.Main resultsFrom the 6380 papers initially identified, 26 met all eligibility criteria. The analysis of the papers identified a total of 168 instances of usability flaws that could be classified into 13 categories of usability flaws representing either violations of general usability principles (i.e. they could be found in any system, e.g. guidance and workload issues) or infractions specific to medication-related alerting functions. The latter refer to issues of low signal-to-noise ratio, incomplete content of alerts, transparency, presentation mode and timing, missing alert features, tasks and control distribution.Main conclusionThe list of 168 instances of usability flaws of medication-related alerting functions provides a source of knowledge for checking the usability of medication-related alerting functions during their design and evaluation process and ultimately constructs evidence-based usability design principles for these functions.     

The use of think-aloud and instant data analysis in evaluation research: Exemplar and lessons learned

While health information technologies have become increasingly popular, many have not been formally tested to ascertain their usability. Traditional rigorous methods take significant amounts of time and manpower to evaluate the usability of a system. In this paper, we evaluate the use of instant data analysis (IDA) as developed by Kjeldskov et al. to perform usability testing on a tool designed for older adults and caregivers. The IDA method is attractive because it takes significantly less time and manpower than the traditional usability testing methods. In this paper we demonstrate how IDA was used to evaluate usability of a multifunctional wellness tool, discuss study results and lessons learned while using this method. We also present findings from an extension of the method which allows the grouping of similar usability problems in an efficient manner. We found that the IDA method is a quick, relatively easy approach to identifying and ranking usability issues among health information technologies.     

Usability of Ventricular Assist Devices in Daily Experience: A Multicenter Study

In daily life, the safe, intuitive use of ventricular assist devices (VADs) and especially their peripheral components is not only a question of life quality, but also sometimes crucial for survival. To investigate the advantages and disadvantages of different systems and to get patient feedback on preferred features, a multicenter study was initiated. Based on previous single‐center studies, a questionnaire was developed to ascertain patients' experiences, difficulties with, and desires concerning use of the system. This questionnaire was provided both to ongoing patients and to new VAD patients after a minimum hospital discharge time of 6 weeks, at a regular checkup. Additionally, the patients completed a standardized questionnaire on life quality (Kansas City Cardiomyopathy Questionnaire). The centers that contributed to this study were Bad Oeynhausen, Berlin, Hannover, and Vienna. Three hundred fifty‐two completed questionnaires on eight different pump types were obtained. An important result is that 42% of those questioned dropped their controller bag at least once. Depending on the device, between 2 and 55% disconnected it unintentionally. Confidence in safe use of the system decreased significantly with age, from 80% at age 20–30 years to 33% at 70–80 years. In devices with an LCD display, 94% considered the readability sufficient. Ninety‐four percent considered the training adequate. Between 22 and 88% of the patients called the emergency telephone hotline, depending on the device, and 23–46% depending on the center. This first multicenter study on VAD usability reveals considerable differences among devices and centers. The comparative assessment aims to help optimize device design, patient management, and training.     

Advancing cognitive engineering methods to support user interface design for electronic health records

BackgroundDespite many decades of research on the effective development of clinical systems in medicine, the adoption of health information technology to improve patient care continues to be slow, especially in ambulatory settings. This applies to dentistry as well, a primary care discipline with approximately 137,000 practitioners in the United States. A critical reason for slow adoption is the poor usability of clinical systems, which makes it difficult for providers to navigate through the information and obtain an integrated view of patient data.ObjectiveIn this study, we documented the cognitive processes and information management strategies used by dentists during a typical patient examination. The results will inform the design of a novel electronic dental record interface.MethodsWe conducted a cognitive task analysis (CTA) study to observe ten general dentists (five general dentists and five general dental faculty members, each with more than two years of clinical experience) examining three simulated patient cases using a think-aloud protocol.ResultsDentists first reviewed the patient's demographics, chief complaint, medical history and dental history to determine the general status of the patient. Subsequently, they proceeded to examine the patient's intraoral status using radiographs, intraoral images, hard tissue and periodontal tissue information. The results also identified dentists’ patterns of navigation through patient's information and additional information needs during a typical clinician–patient encounter.ConclusionThis study reinforced the significance of applying cognitive engineering methods to inform the design of a clinical system. Second, applying CTA to a scenario closely simulating an actual patient encounter helped with capturing participants’ knowledge states and decision-making when diagnosing and treating a patient. The resultant knowledge of dentists’ patterns of information retrieval and review will significantly contribute to designing flexible and task-appropriate information presentation in electronic dental records.     

From expert-derived user needs to user-perceived ease of use and usefulness: A two-phase mixed-methods evaluation framework

Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of “the best available” comparison system to enable a cognitive walkthrough in phase 1 and a comparative effectiveness evaluation in phase 2. During phase 2, five quantitative and qualitative evaluation methods are mixed to assess usability: time-motion analysis, software log, questionnaires – System Usability Scale and the Unified Theory of Acceptance of Use of Technology, think-aloud protocols, and unstructured interviews. Each method contributes data for a unique measure (e.g., time motion analysis contributes task-completion-time; software log contributes action transition frequency). The measures are triangulated to yield complementary insights regarding user-perceived ease-of-use, functionality integration, anxiety during use, and workflow impact. To illustrate its use, we applied this framework in a formative evaluation of a software called Integrated Model for Patient Care and Clinical Trials (IMPACT). We conclude that this mixed-methods evaluation framework enables an integrated assessment of user needs satisfaction and user-perceived usefulness and usability of a novel design. This evaluation framework effectively bridges the gap between co-evolving user needs and technology designs during iterative prototyping and is particularly useful when it is difficult for users to articulate their needs for technology support due to the lack of a baseline.