A personalized mobile patient guide system for a patient-centered smart hospital: Lessons learned from a usability test and satisfaction survey in a tertiary university hospital

ObjectiveThe present study focused on the design, implementation, and evaluation of a personalized mobile patient guide system that utilizes smart phones, indoor navigation technology and a hospital information system (HIS) to address the difficulties that outpatients face in finding hospital facilities, recognizing their daily treatment schedule, and accessing personalized medical and administrative information.Materials and methodsThe present study was conducted in a fully digitized tertiary university hospital in South Korea. We developed a real-time location-based outpatient guide system that consists of Bluetooth access points (APs) for indoor navigation, an Android-based guide application, a guide server, and interfaces with the HIS. A total of 33 subjects and 43 outpatients participated in the usability test (UT) and the satisfaction survey, respectively.ResultsWe confirmed that the indoor navigation feature can be applied to outpatient departments with precision using a position error test. The participants in the UT completed each scenario with an average success rate of 67.4%. According to the results, we addressed the problems and made improvements to the user interface by providing users with context-based guidance information. The satisfaction rating of the system was high, with an average score of 4.0 out of 5.0, showing its utility as a patient-centered hospital service.ConclusionThe innovative mobile patient guide system for outpatients is feasible and can be successfully implemented to provide personalized information with high satisfaction. Additionally, the issues identified and lessons learned from our experiences regarding task scheduling, indoor navigation, and usability should be considered when developing the system.     

A study of user requests regarding the fully electronic health record system at Seoul National University Bundang Hospital: Challenges for future electronic health record systems

ObjectiveAlthough the adoption rates for Electronic Health Records (EHRs) are growing, significant opportunities for further advances in EHR system design remain. The goal of this study was to identify issues that should be considered in the design process for the successful development of future systems by analyzing end users’ service requests gathered during a recent three-year period after a comprehensive EHR system was implemented at Seoul National University's Bundang Hospital in South Korea.MethodsData on 11,400 service requests from end users of the EHR system made from 2008 through 2010 were used in this study. The requests were categorized as program modification/development, data request, insurance-fee identification/generation, patient-record merging, or other. The authors further subcategorized the requests for program modification/development into the following nine areas of concern: (1) indicators and statistics, (2) patient safety and quality of care, (3) special task-oriented functionalities, (4) ease of use and user interface, (5) system speed, (6) interoperability and integration, (7) privacy and security, (8) customer service, and (9) miscellaneous. The system users were divided into four groups—direct care, care support, administrative/insurance, and general management—to identify each group's needs and concerns.ResultsThe service requests for program modification/development, data request, insurance-fee identification/generation, patient-record merging, and other issues constituted approximately 49.2%, 33.9%, 11.4%, 4.0%, and 1.5% of the total data set, respectively. The number of data-request service requests grew over the three years studied. Different groups of users were found to have different concerns according to their activities and tasks. Within the program-modification/development category, end users were most frequently concerned with ease of use and user interface (38.1% of the total) and special task-oriented functionalities (29.3% of the total) in their use of the EHR system, with increasing numbers of requests in both categories over the three years. Users in the direct-care group differed from the other groups in that they most frequently submitted requests related to ease of use and user interface, followed by special functionalities, patient safety and quality care, and customer service, while users in other groups submitted requests concerning ease of use and user interface and special functionalities with a similarly high frequency.ConclusionsUsers have continued to make suggestions about their needs and requirements, and the EHR system has evolved to optimize ease of use and special functionalities for particular groups of users and particular subspecialties. Based on our experiences and the lessons we have learned in the course of maintaining full-EHR systems, we suggest that the key goals to be considered for future EHR systems include innovative new user-interface technologies; special extended functions for each user group's specific task-oriented requirements; powerful, easy-to-use functions to support research; new flexible system architecture; and patient-directed functions.     

Patients' opinions on health counseling in the Hartslag Limburg cardiovascular prevention project: perceived quality, satisfaction, and normative concerns

ObjectiveTo assess patients’ opinions on a new service in the Dutch health care system, viz., health counseling about behavioral risks and possible lifestyle changes. This was introduced within the framework of the Hartslag Limburg prevention project and was offered to patients at high risk of cardiovascular diseases.MethodsA questionnaire survey to measure quality, satisfaction, and normative concerns. Questionnaires were distributed among 1301 patients after they had completed the counseling and returned by 1056 of them (response rate 81.2%).ResultsPatients generally reported rather high levels of quality and satisfaction. They regarded the service as a welcome addition to usual care. However, the accompanying effect studies will have to provide more solid evidence for the effectiveness of the counseling.ConclusionIt was concluded that the service was highly acceptable to the patients, but that the health advisors should be careful not to induce less favorable normative concerns.Practice implicationsA quality and satisfaction survey should consist of well-validated instruments, context-specific items, and qualitative measures as well.     

Development,implementation and evaluation of an information model for archetype based user responsive medical data visualization

BackgroundWhen medical data have been successfully recorded or exchanged between systems there appear a need to present the data consistently to ensure that it is clearly understood and interpreted. A standard based user interface can provide interoperability on the visual level.ObjectivesThe goal of this research was to develop, implement and evaluate an information model for building user interfaces for archetype based medical data.MethodsThe following types of knowledge were identified as important elements and were included in the information model: medical content related attributes, data type related attributes, user-related attributes, device-related attributes. In order to support flexible and efficient user interfaces an approach that represents different types of knowledge with different models separating the medical concept from a visual concept and interface realization was chosen. We evaluated the developed approach using Guideline for Good Evaluation Practice in Health Informatics (GEP-HI).ResultsWe developed a higher level information model to complement the ISO 13606 archetype model. This enabled the specification of the presentation properties at the moment of the archetypes’ definition. The model allows realizing different users’ perspectives on the data. The approach was implemented and evaluated within a functioning EHR system. The evaluation involved 30 patients of different age and IT experience and 5 doctors. One month of testing showed that the time required reading electronic health records decreased for both doctors (from average 310 to 220 s) and patients (from average 95 to 39 s). Users reported a high level of satisfaction and motivation to use the presented data visualization approach especially in comparison with their previous experience.ConclusionThe introduced information model allows separating medical knowledge and presentation knowledge. The additional presentation layer will enrich the graphical user interface’s flexibility and will allow an optimal presentation of medical data considering the different users’ perspectives and different media used for data presentation.     

Website quality, expectation, confirmation, and end user satisfaction: the knowledge-intensive website of the Korean National Cancer Information Center

Background

The fact that patient satisfaction with primary care clinical practices and physician-patient communications has decreased gradually has brought a new opportunity to the online channel as a supplementary service to provide additional information.

Objective

In this study, our objectives were to examine the process of cognitive knowledge expectation-confirmation from eHealth users and to recommend the attributes of a “knowledge-intensive website.”. Knowledge expectation can be defined as users’ existing attitudes or beliefs regarding expected levels of knowledge they may gain by accessing the website. Knowledge confirmation is the extent to which user’s knowledge expectation of information systems use is realized during actual use. In our hypothesized research model, perceived information quality, presentation and attractiveness as well as knowledge expectation influence knowledge confirmation, which in turn influences perceived usefulness and end user satisfaction, which feeds back to knowledge expectation.

Methods

An empirical study was conducted at the National Cancer Center (NCC), Republic of Korea (South Korea), by evaluating its official website. A user survey was administered containing items to measure subjectively perceived website quality and expectation-confirmation attributes. A study sample of 198 usable responses was used for further analysis. We used the structural equation model to test the proposed research model.

Results

Knowledge expectation exhibited a positive effect on knowledge confirmation (beta = .27, P < .001). The paths from information quality, information presentation, and website attractiveness to knowledge confirmation were also positive and significant (beta = .24, P < .001; beta = .29, P < .001; beta = .18, P < .001, respectively). Moreover, the effect of knowledge confirmation on perceived usefulness was also positively significant (beta = .64, P < .001). Knowledge expectation together with knowledge confirmation and perceived usefulness also significantly affected end user satisfaction (beta = .22 P < .001; beta = .39, P < .001; beta = .25, P < .001, respectively).

Conclusions

Theoretically, this study has (1) identified knowledge-intensive website attributes, (2) enhanced the theoretical foundation of eHealth from the information systems (IS) perspective by adopting the expectation-confirmation theory (ECT), and (3) examined the importance of information and knowledge attributes and explained their impact on user satisfaction. Practically, our empirical results suggest that perceived website quality (ie, information quality, information presentation, and website attractiveness) is a core requirement for knowledge building. In addition, our study has also shown that knowledge confirmation has a greater effect on satisfaction than both knowledge expectation and perceived usefulness.     

A feasibility study to investigate the acceptability and potential effectiveness of a telecare service for older people with chronic obstructive pulmonary disease

AimsTo investigate the feasibility, acceptance and potential effectiveness of delivering a telecare service on the health outcomes and hospital service utilization of community-dwelling patients with chronic obstructive pulmonary disease.MethodsEligible participants were older people, with moderate or severe chronic obstructive pulmonary disease, and who had been admitted to hospital at least once for exacerbation during the previous year. The participants were randomly assigned to the intervention or control group. Participants in the intervention group received a telecare device kit and they were asked to monitor their oxygen saturation, pulse rate and respiration rate using the device and to transmit the data to an online network platform. A medication and purse-lip breathing reminder with a feedback function is also provided in the device kit. A community nurse monitors changes in the physiological parameters and takes immediate action to address the patients’ needs. Participants in the control group received no other extra care. Study outcomes include user satisfaction, health-related quality of life, pulmonary function, hospital re-admission and use of emergency room services.ResultsTwenty-two participants in the intervention group and 18 in the control group were included in the analysis. The mean age of all 40 participants was 72.93 years. Overall, the participants in the intervention group expressed satisfaction with the telecare service. Some patients reported difficulty in reading the screen of the mobile phone and manipulating the tiny key-in buttons. No significant differences were found between the two time points (baseline and post-test period) with regard to health-related quality of life. No significant differences in pulmonary function and in the number of emergency department visits and hospital re-admissions between the study groups were found.ConclusionThe high level of user satisfaction indicated the feasibility of conducting a large-scale randomized control trial to evaluate the effects of a telecare service on health outcomes of patients with chronic obstructive pulmonary disease.     

Inpatient satisfaction and usage patterns of personalized smart bedside station system for patient-centered service at a tertiary university hospital

ObjectiveBedside stations, also known as bedside terminals, are in place to enhance the quality and experience of a hospital’s healthcare service delivery. The purpose of this study was to identify information needs and overall satisfaction with the personalized patient bedside system, called Smart Bedside Station (SBS) system, embedded in a tertiary general university hospital.MethodsEnd-user responses on the satisfaction survey and system usage logs of the SBS system were collected and analyzed. For the user opinion survey, 156 nurses and 1914 patients, their family members, or caregivers participated during the evaluation period of 2013 to 2014 in this study. All working nurses in the SBS-installed ward were answered the paper-based evaluation, for complete enumeration survey. Inpatients were voluntary participated to deliver the online questionnaire on the SBS menu. We also explored system log data including page calls and usage time from December 2013 to 2015.ResultsRegarding the relationship of overall satisfaction of the SBS with patient’s characteristics, patient’s education status and degree of familiarity with the smart device were statistically significant. From the analysis of system logs, Personalized My Menu(28.0%) was the most frequently used menu item (except for TV and Internet entertainment service use of 62.7%),it provides individual health information, such as laboratory test results, hospital fee check, message logs, daily medication information, and meal information. Next frequently used menus were information support(4.9%) which deliver hospital guide and health information and convenience service ordering(4.4%) such as meal order, bed sheet change. Satisfaction survey results and log data results show that the personalized service enhances the user satisfaction during hospital admission.ConclusionsOur post-implementation experience and subsequent assessment of SBS system is capable of providing insights into improving the hospital information system and service contents for patient-centered services. Further research should be directed at developing sophisticated patient-centered services as a communication tool between the hospital and the patient.     

Electronic health records implementation: An evaluation of information system impact and contingency factors

ObjectiveThis paper provides a review of EHR (electronic health record) implementations around the world and reports on findings including benefits and issues associated with EHR implementation.Materials and methodsA systematic literature review was conducted from peer-reviewed scholarly journal publications from the last 10 years (2001–2011). The search was conducted using various publication collections including: Scopus, Embase, Informit, Medline, Proquest Health and Medical Complete. This paper reports on our analysis of previous empirical studies of EHR implementations. We analysed data based on an extension of DeLone and McLean's information system (IS) evaluation framework. The extended framework integrates DeLone and McLean's dimensions, including information quality, system quality, service quality, intention of use and usage, user satisfaction and net benefits, together with contingent dimensions, including systems development, implementation attributes and organisational aspects, as identified by Van der Meijden and colleagues.ResultsA mix of evidence-based positive and negative impacts of EHR was found across different evaluation dimensions. In addition, a number of contingent factors were found to contribute to successful implementation of EHR.LimitationsThis review does not include white papers or industry surveys, non-English papers, or those published outside the review time period.ConclusionThis review confirms the potential of this technology to aid patient care and clinical documentation; for example, in improved documentation quality, increased administration efficiency, as well as better quality, safety and coordination of care. Common negative impacts include changes to workflow and work disruption. Mixed observations were found on EHR quality, adoption and satisfaction. The review warns future implementers of EHR to carefully undertake the technology implementation exercise. The review also informs healthcare providers of contingent factors that potentially affect EHR development and implementation in an organisational setting. Our findings suggest a lack of socio-technical connectives between the clinician, the patient and the technology in developing and implementing EHR and future developments in patient-accessible EHR. In addition, a synthesis of DeLone and McLean's framework and Van der Meijden and colleagues’ contingent factors has been found useful in comprehensively understanding and evaluating EHR implementations.     

Finding the Patient’s Voice Using Big Data: Analysis of Users’ Health-Related Concerns in the ChaCha Question-and-Answer Service (2009–2012)

BackgroundThe development of effective health care and public health interventions requires a comprehensive understanding of the perceptions, concerns, and stated needs of health care consumers and the public at large. Big datasets from social media and question-and-answer services provide insight into the public’s health concerns and priorities without the financial, temporal, and spatial encumbrances of more traditional community-engagement methods and may prove a useful starting point for public-engagement health research (infodemiology).ObjectiveThe objective of our study was to describe user characteristics and health-related queries of the ChaCha question-and-answer platform, and discuss how these data may be used to better understand the perceptions, concerns, and stated needs of health care consumers and the public at large.MethodsWe conducted a retrospective automated textual analysis of anonymous user-generated queries submitted to ChaCha between January 2009 and November 2012. A total of 2.004 billion queries were read, of which 3.50% (70,083,796/2,004,243,249) were missing 1 or more data fields, leaving 1.934 billion complete lines of data for these analyses.ResultsMales and females submitted roughly equal numbers of health queries, but content differed by sex. Questions from females predominantly focused on pregnancy, menstruation, and vaginal health. Questions from males predominantly focused on body image, drug use, and sexuality. Adolescents aged 12–19 years submitted more queries than any other age group. Their queries were largely centered on sexual and reproductive health, and pregnancy in particular.ConclusionsThe private nature of the ChaCha service provided a perfect environment for maximum frankness among users, especially among adolescents posing sensitive health questions. Adolescents’ sexual health queries reveal knowledge gaps with serious, lifelong consequences. The nature of questions to the service provides opportunities for rapid understanding of health concerns and may lead to development of more effective tailored interventions.     

Reliability and Validity of Assessing User Satisfaction With Web-Based Health Interventions

BackgroundThe perspective of users should be taken into account in the evaluation of Web-based health interventions. Assessing the users’ satisfaction with the intervention they receive could enhance the evidence for the intervention effects. Thus, there is a need for valid and reliable measures to assess satisfaction with Web-based health interventions.ObjectiveThe objective of this study was to analyze the reliability, factorial structure, and construct validity of the Client Satisfaction Questionnaire adapted to Internet-based interventions (CSQ-I).MethodsThe psychometric quality of the CSQ-I was analyzed in user samples from 2 separate randomized controlled trials evaluating Web-based health interventions, one from a depression prevention intervention (sample 1, N=174) and the other from a stress management intervention (sample 2, N=111). At first, the underlying measurement model of the CSQ-I was analyzed to determine the internal consistency. The factorial structure of the scale and the measurement invariance across groups were tested by multigroup confirmatory factor analyses. Additionally, the construct validity of the scale was examined by comparing satisfaction scores with the primary clinical outcome.ResultsMultigroup confirmatory analyses on the scale yielded a one-factorial structure with a good fit (root-mean-square error of approximation =.09, comparative fit index =.96, standardized root-mean-square residual =.05) that showed partial strong invariance across the 2 samples. The scale showed very good reliability, indicated by McDonald omegas of .95 in sample 1 and .93 in sample 2. Significant correlations with change in depressive symptoms (r=−.35, P<.001) and perceived stress (r=−.48, P<.001) demonstrated the construct validity of the scale.ConclusionsThe proven internal consistency, factorial structure, and construct validity of the CSQ-I indicate a good overall psychometric quality of the measure to assess the user’s general satisfaction with Web-based interventions for depression and stress management. Multigroup analyses indicate its robustness across different samples. Thus, the CSQ-I seems to be a suitable measure to consider the user’s perspective in the overall evaluation of Web-based health interventions.     

Patient and practitioner priorities and concerns about primary healthcare interactions for osteoarthritis: A meta-ethnography

ObjectiveTo explore primary care practitioners’ (PCPs) and patients’ priorities and concerns for healthcare interactions for osteoarthritis (OA) in primary care.MethodsWe searched Embase, CINAHL, Medline, PsychInfo (1990 to present) for primary qualitative and mixed methods studies with findings concerning healthcare interactions for OA symptoms. Patient and PCP perceptions were analysed separately then inter-related using a ‘line of argument’ synthesis.ResultsTwenty-six studies reporting qualitative data from 557 patients and 199 PCPs were synthesised. Our findings suggest that therapeutic interactions for OA can be based on discordant priorities and concerns; some patients perceive that PCPs hold negative attitudes about OA and feel their concerns about impact are not appreciated; some PCPs feel patients have misconceptions about prognosis, and hold pessimistic views about outcomes; and both tend to de-prioritise OA within consultations.ConclusionGreater working in partnership could build mutual trust, facilitate tailored provision of information, and foster a shared understanding of OA upon which to build realistic goals for management.Practice implicationsDeveloping a better shared understanding of OA has the potential to improve the quality of healthcare interactions for both patients and PCPs. The significant impact of OA on everyday life means it should be given higher priority in primary care consultations.     

Impacts of mobile tablet computing on provider productivity,communications, and the process of care

ObjectiveHealth information technology investments continue to increase while the value derived from their implementation and use is mixed. Mobile device adoption into practice is a recent trend that has increased dramatically and formal studies are needed to investigate consequent benefits and challenges. The objective of this study is to evaluate practitioner perceptions of improvements in productivity, provider-patient communications, care provision, technology usability and other outcomes following the adoption and use of a tablet computer connected to electronic health information resources.MethodsA pilot program was initiated in June 2013 to evaluate the effect of mobile tablet computers at one health provider organization in the southeast United States. Providers were asked to volunteer for the evaluation and were each given a mobile tablet computer. A total of 42 inpatient and outpatient providers were interviewed in 2015 using a survey style questionnaire that utilized yes/no, Likert-style, and open ended questions. Each had previously used an electronic health record (EHR) system a minimum of one year outside of residency, and were regular users of personal mobile devices. Each used a mobile tablet computer in the context of their practice connected to the health system EHR.ResultsThe survey results indicate that more than half of providers perceive the use of the tablet device as having a positive effect on patient communications, patient education, patient's perception of the provider, time spent interacting with patients, provider productivity, process of care, satisfaction with EHR when used together with the device, and care provision. Providers also reported feeling comfortable using the device (82.9%), would recommend the device to colleagues (69.2%), did not experience increased information security and privacy concerns (95%), and noted significant reductions in EHR login times (64.1%). Less than 25% of participants reported negative impacts on any of these areas as well as on time spent on order submission, note completion time, overall workload, patient satisfaction with care experience and patient outcomes. Gender, number of years in practice, practice type (general practitioner vs. specialist), and service type (inpatient/outpatient) were found to have a significant effect on perceptions of patient satisfaction, care process, and provider productivity.ConclusionsProviders found positive gains from utilizing mobile devices in overall productivity, improved communications with their patients, the process of care, and technology efficiencies when used in combination with EHR and other health information resources. Demographic and health care work environment play a role in how mobile technologies are integrated into practice by providers.     

The success of a management information system in health care – A case study from Finland

PurposeThe purpose of this article is to describe perspectives on information availability and information use among users of a management information system in one specialized health care organization. The management information system (MIS) is defined as the information system that provides management with information about financial and operational aspects of hospital management.MethodsThe material for this qualitative case study was gathered by semi-structured interviews. The interviewees were purposefully selected from one specialized health care organization. The organization has developed its management information system in recent years. Altogether 13 front-line, middle and top-level managers were interviewed. The two themes discussed were information availability and information use. The data were analyzed using inductive content analysis using ATLAS.ti computer program.ResultsThe main category “usage of management information system” consisted of four sub-categories: (1) system quality, (2) information quality, (3) use and user satisfaction and (4) development of information culture.ConclusionsThere were many organizational and cultural aspects which influence the use of MIS in addition to factors concerning system usability and users. The connection between information culture and information use was recognized and the managers proposed numerous ways to increase the use of information in management work. The implementation and use of management information system did not seem to be planned as an essential tool in strategic information management in the health care organization studied.     

Medical equipment libraries: implementation,experience and user satisfaction

Abstract

The hospital-wide pooling and sharing of certain types of medical equipment can lead to both significant improvements in patient safety and financial advantages when compared with a department or ward-level equipment ownership system. In September 2003, a Medical Equipment Loan Service (MELS) was established, focusing initially on infusion pumps. The aims and expected benefits included; improving availability of equipment for both patients and clinical users, managing and reducing clinical risk, reducing equipment diversity, improving equipment management and reducing the overall cost of equipment provision. A user survey was carried out in 2005 and repeated in 2011. The results showed wide and continued satisfaction with the service. The process and difficulties of establishing the service and its development to include additional types of equipment are described. The benefits of managing medical equipment which is in widespread general use, through a MELS as part of a Clinical Engineering Department, are presented.     

Gaining actionable knowledge to improve local health-promoting capacities in long-term care support settings for people with intellectual disabilities

ObjectivePeople with intellectual disabilities (ID) are largely dependent on their environment to live healthily and, in this, ID-support organizations play a vital role. An environmental asset mapping tool for ID-support settings has been developed. This study aims to provide insight into whether or not the tool can provide a comprehensive view on assets in the system and actionable knowledge to improve health-promoting capacities in ID-support settings.MethodsFifty-seven users from four setting completed the tool on availability, user satisfaction, and dreams regarding social, physical, organizational, and financial assets.ResultsThe findings provide a comprehensive view of available assets. Together with user satisfaction and dreams for improvements, they provide actionable knowledge for improving the health-promoting capacities of the settings, including: (1) how use of available assets can be improved, (2) the type of assets that should be enriched, and (3) the assets that can be added to the system.ConclusionThe asset mapping tool provides a comprehensive view on assets in the system and actionable knowledge to improve health-promoting capacities in ID-support settings.Practice implicationsID-support organizations can use the tool to generate actionable bottom-up knowledge for priority setting and implementing interventions to improve their health-promoting capacities.     

Streamline triage and manage user expectations: lessons from a qualitative study of GP out-of-hours services

Background

Several models of GP out-of-hours provision exist in the UK but there is little detail about their effectiveness to meet users'' needs and expectations.

Aim

To explore users'' needs, expectations, and experiences of out-of-hours care, and to identify proposals for service redesign.

Setting

Service providers in urban (GP cooperative), mixed (hospital based), rural (private) locations in Wales.

Participants

Sixty recent service users or carers (20 in each location).

Method

Semi-structured telephone interviews; thematic analysis.

Results

Users'' concerns were generally consistent across the three different services. Efficiency was a major concern, with repetitive triage procedures and long time delays at various stages in the process being problematic. Access to a doctor when required was also important to users, who perceived an obstructive gatekeeping function of preliminary contacts. Expectations moderated the relationship between user concerns and satisfaction. Where expectations of outcome were unfulfilled, participants reported greater likelihood of reconsulting with the same or alternative services for the same illness episode. Accurate expectations concerning contacts with the next administrative, nursing, or medical staff professional were managed by appropriate information provision.

Conclusion

Users require more streamlined and flexible triage systems. Their expectations need to be understood and incorporated into how services advise and provide services for users, and actively managed to meet the aims of both enhancing satisfaction and enabling users to cope with their condition. Better information and education about services are needed if users are to derive the greatest benefit and satisfaction. This may influence choices about using the most appropriate forms of care.     

A study of the influence of task familiarity on user behaviors and performance with a MeSH term suggestion interface for PubMed bibliographic search

PurposePrevious research has shown that information seekers in biomedical domain need more support in formulating their queries. A user study was conducted to evaluate the effectiveness of a metadata based query suggestion interface for PubMed bibliographic search. The study also investigated the impact of search task familiarity on search behaviors and the effectiveness of the interface.MethodsA real user, user search request and real system approach was used for the study. Unlike tradition IR evaluation, where assigned tasks were used, the participants were asked to search requests of their own. Forty-four researchers in Health Sciences participated in the evaluation – each conducted two research requests of their own, alternately with the proposed interface and the PubMed baseline. Several performance criteria were measured to assess the potential benefits of the experimental interface, including users’ assessment of their original and eventual queries, the perceived usefulness of the interfaces, satisfaction with the search results, and the average relevance score of the saved records.ResultsThe results show that, when searching for an unfamiliar topic, users were more likely to change their queries, indicating the effect of familiarity on search behaviors. The results also show that the interface scored higher on several of the performance criteria, such as the “goodness” of the queries, perceived usefulness, and user satisfaction. Furthermore, in line with our hypothesis, the proposed interface was relatively more effective when less familiar search requests were attempted.ConclusionsResults indicate that there is a selective compatibility between search familiarity and search interface. One implication of the research for system evaluation is the importance of taking into consideration task familiarity when assessing the effectiveness of interactive IR systems.     

A four stage approach for ontology-based health information system design

ObjectiveTo describe and illustrate a four stage methodological approach to capture user knowledge in a biomedical domain area, use that knowledge to design an ontology, and then implement and evaluate the ontology as a health information system (HIS).Methods and materialsA hybrid participatory design-grounded theory (GT-PD) method was used to obtain data and code them for ontology development. Prototyping was used to implement the ontology as a computer-based tool. Usability testing evaluated the computer-based tool.ResultsAn empirically derived domain ontology and set of three problem-solving approaches were developed as a formalized model of the concepts and categories from the GT coding. The ontology and problem-solving approaches were used to design and implement a HIS that tested favorably in usability testing.ConclusionsThe four stage approach illustrated in this paper is useful for designing and implementing an ontology as the basis for a HIS. The approach extends existing ontology development methodologies by providing an empirical basis for theory incorporated into ontology design.     

More than just tracking time: Complex measures of user engagement with an internet-based health promotion intervention

BackgroundThere has been a rise in internet-based health interventions without a concomitant focus on new methods to measure user engagement and its effect on outcomes. We describe current user tracking methods for internet-based health interventions and offer suggestions for improvement based on the design and pilot testing of healthMpowerment.org (HMP).MethodsHMP is a multi-component online intervention for young Black men and transgender women who have sex with men (YBMSM/TW) to reduce risky sexual behaviors, promote healthy living and build social support. The intervention is non-directive, incorporates interactive features, and utilizes a point-based reward system. Fifteen YBMSM/TW (age 20–30) participated in a one-month pilot study to test the usability and efficacy of HMP. Engagement with the intervention was tracked using a customized data capture system and validated with Google Analytics. Usage was measured in time spent (total and across sections) and points earned.ResultsAverage total time spent on HMP was five hours per person (range 0–13). Total time spent was correlated with total points earned and overall site satisfaction.ConclusionMeasuring engagement in internet-based interventions is crucial to determining efficacy. Multiple methods of tracking helped derive more comprehensive user profiles. Results highlighted the limitations of measures to capture user activity and the elusiveness of the concept of engagement.