An evaluation of two commercial deep learning-based information retrieval systems for COVID-19 literature

The COVID-19 pandemic has resulted in a tremendous need for access to the latest scientific information, leading to both corpora for COVID-19 literature and search engines to query such data. While most search engine research is performed in academia with rigorous evaluation, major commercial companies dominate the web search market. Thus, it is expected that commercial pandemic-specific search engines will gain much higher traction than academic alternatives, leading to questions about the empirical performance of these tools. This paper seeks to empirically evaluate two commercial search engines for COVID-19 (Google and Amazon) in comparison with academic prototypes evaluated in the TREC-COVID task. We performed several steps to reduce bias in the manual judgments to ensure a fair comparison of all systems. We find the commercial search engines sizably underperformed those evaluated under TREC-COVID. This has implications for trust in popular health search engines and developing biomedical search engines for future health crises.     

MIRASS: Medical Informatics Research Activity Support System Using Information Mashup Network

The advancement of information technology has facilitated the automation and feasibility of online information sharing. The second generation of the World Wide Web (Web 2.0) enables the collaboration and sharing of online information through Web-serving applications. Data mashup, which is considered a Web 2.0 platform, plays an important role in information and communication technology applications. However, few ideas have been transformed into education and research domains, particularly in medical informatics. The creation of a friendly environment for medical informatics research requires the removal of certain obstacles in terms of search time, resource credibility, and search result accuracy. This paper considers three glitches that researchers encounter in medical informatics research; these glitches include the quality of papers obtained from scientific search engines (particularly, Web of Science and Science Direct), the quality of articles from the indices of these search engines, and the customizability and flexibility of these search engines. A customizable search engine for trusted resources of medical informatics was developed and implemented through data mashup. Results show that the proposed search engine improves the usability of scientific search engines for medical informatics. Pipe search engine was found to be more efficient than other engines.     

Comparative Analysis of Medical Web Search Using Generalized vs. Niche Technologies

One increasingly popular solution to the problem of information overload is the niche search engine, designed as a customized knowledge management tool to meet the needs of information seekers with similar interests. By limiting web crawling to a specific subject area, the niche engine is able to crawl deeper and more discreetly than commercial counterparts. This exploratory technology assessment study sought to examine differences in information search and retrieval patterns between commercial and niche search engines in areas of debated or uncertain healthcarer treatment. Findings suggest that while information found within the niche search is generally more trustworthy, niche searches leave out many reliable sources which are retrievable through commercial search engines. Also, consumer-friendly (non-technical) resources were found to be more prevalent using commercial search engines, suggesting greater relevance for the layperson within this domain.     

Is Relevance Relevant? User Relevance Ratings May Not Predict the Impact of Internet Search on Decision Outcomes

Objective

A common measure of Internet search engine effectiveness is its ability to find documents that a user perceives as ‘relevant’. This study sought to test whether user provided relevance ratings for documents retrieved by an Internet search engine correlate with the decision outcome after use of a search engine.

Design

227 university students were asked to answer four randomly assigned consumer health questions, then to conduct an Internet search on one of two randomly assigned search engines of different performance, and to again answer the question.

Measurements

Participants were asked to provide a relevance score for each document retrieved as well as a pre and post search answer to each question.

Results

User relevance rankings had little or no predictive power. Relevance rankings were unable to predict whether the user of a search engine could correctly answer a question after search and could not differentiate between two search engines with statistically different performance in the hands of users. Only when users had strong prior knowledge of the questions, and the decision task was of low complexity, did relevance appear to have modest predictive power.

Conclusions

User provided relevance rankings taken in isolation seem to be of limited to no value when designing a search engine that will be used in a general-purpose setting. Relevance rankings may have a place in situations in which experts provide rankings, and decision tasks are of complexity commensurate with the abilities of the raters. A more natural metric of search engine performance may be a user''s ability to accurately complete a task, as this removes the inherent subjectivity of relevance rankings, and provides a direct and repeatable outcome measure which directly correlates with the performance of the search technology in the hands of users.     

网络信息资源组织中的医学搜索引擎

本文阐述了网络信息资源、网络信息资源组织、搜索引擎的定义、特点以及彼此之间的相互关系，从医学专业的特殊性和专深性说明了医学搜索引擎产生的必要性；着重介绍了医学搜索引擎的类型及目前比较常用的中英文医学搜索引擎，提出医学搜索引擎将发展成为最具前途的医学网络信息资源组织方式。     

Seeking health information online: does limited healthcare access matter?

Consumers facing barriers to healthcare access may use online health information seeking and online communication with physicians, but the empirical relationship has not been sufficiently analyzed. Our study examines the association of barriers to healthcare access with consumers’ health-related information searching on the internet, use of health chat groups, and email communication with physicians, using data from 27 210 adults from the 2009 National Health Interview Survey. Individuals with financial barriers to healthcare access, difficulty getting timely appointments with doctors, and conflicts in scheduling during clinic hours are more likely to search for general health information online than those without these access barriers. Those unable to get timely appointments with physicians are more likely to participate in health chat groups and email physicians. The internet may offer a low-cost source of health information and could help meet the heightened demand for health-related information among those facing access barriers to care.     

Health information on the Internet: accessibility, quality, and readability in English and Spanish

CONTEXT: Despite the substantial amount of health-related information available on the Internet, little is known about the accessibility, quality, and reading grade level of that health information. OBJECTIVE: To evaluate health information on breast cancer, depression, obesity, and childhood asthma available through English- and Spanish-language search engines and Web sites. DESIGN AND SETTING: Three unique studies were performed from July 2000 through December 2000. Accessibility of 14 search engines was assessed using a structured search experiment. Quality of 25 health Web sites and content provided by 1 search engine was evaluated by 34 physicians using structured implicit review (interrater reliability >0.90). The reading grade level of text selected for structured implicit review was established using the Fry Readability Graph method. MAIN OUTCOME MEASURES: For the accessibility study, proportion of links leading to relevant content; for quality, coverage and accuracy of key clinical elements; and grade level reading formulas. RESULTS: Less than one quarter of the search engine's first pages of links led to relevant content (20% of English and 12% of Spanish). On average, 45% of the clinical elements on English- and 22% on Spanish-language Web sites were more than minimally covered and completely accurate and 24% of the clinical elements on English- and 53% on Spanish-language Web sites were not covered at all. All English and 86% of Spanish Web sites required high school level or greater reading ability. CONCLUSION: Accessing health information using search engines and simple search terms is not efficient. Coverage of key information on English- and Spanish-language Web sites is poor and inconsistent, although the accuracy of the information provided is generally good. High reading levels are required to comprehend Web-based health information.     

Assessing Health Consumerism on the Web: A Demographic Profile of Information-Seeking Behaviors

The growing diversity of the online health information community is increasingly cited as a limiting factor related to the potential of the Internet as an effective health communication channel and information resource. Public-access Internet portals and decreasing costs of personal computers have created a consensus that unequal access to information, or a “Digital Divide,” presents a like problem specific to health care consumers. Access to information, however, is an essential part of the consumer-centric framework outlined in the recently proposed U.S. National Health Information Infrastructure (NHII) and Health Architecture initiatives. To date little research has been done to differentiate the types of health information sought on the Web by different subgroups, linking user characteristics and health-seeking behaviors. Data from a study of consumer Web search activity in a post-intervention era serves as a natural experiment, and can identify whether a “digitally underserved group” persists in the United States. Such an environment would serve to exclude traditionally underserved groups from the benefits of the planned national heath information infrastructure. This exploratory technology assessment study seeks to differentiate and delineate specific behaviors, or lack of desired behaviors, across targeted health care subgroups. Doing so allows the design of more effective strategies to promote the use of the Web as a health education and health promotion tool, under the envisioned shared decision-making, consumer-centric health information model.     

Comparison of consumers’ views on electronic data sharing for healthcare and research

New models of healthcare delivery such as accountable care organizations and patient-centered medical homes seek to improve quality, access, and cost. They rely on a robust, secure technology infrastructure provided by health information exchanges (HIEs) and distributed research networks and the willingness of patients to share their data. There are few large, in-depth studies of US consumers’ views on privacy, security, and consent in electronic data sharing for healthcare and research together.Objective This paper addresses this gap, reporting on a survey which asks about California consumers’ views of data sharing for healthcare and research together.Materials and Methods The survey conducted was a representative, random-digit dial telephone survey of 800 Californians, performed in Spanish and English.Results There is a great deal of concern that HIEs will worsen privacy (40.3%) and security (42.5%). Consumers are in favor of electronic data sharing but elements of transparency are important: individual control, who has access, and the purpose for use of data. Respondents were more likely to agree to share deidentified information for research than to share identified information for healthcare (76.2% vs 57.3%, p < .001).Discussion While consumers show willingness to share health information electronically, they value individual control and privacy. Responsiveness to these needs, rather than mere reliance on Health Insurance Portability and Accountability Act (HIPAA), may improve support of data networks.Conclusion Responsiveness to the public’s concerns regarding their health information is a pre-requisite for patient-centeredness. This is one of the first in-depth studies of attitudes about electronic data sharing that compares attitudes of the same individual towards healthcare and research.     

Racial disparities in health information access: resilience of the digital divide

Policy initiatives of the late 1990s were believed to have largely eliminated the information “Digital Divide.” For healthcare consumers, access to information is an essential part of the consumer-centric framework outlined in the recently proposed national health information initiative. This study sought to examine how racial/ethnic characteristics are associated with Internet use and online health information. Using a cross-sectional nationwide study of reported Internet use and information search in 2000 and 2002, we studied a stratified sample of computer users from the Pew Internet & American Life Project surveys. Adjusted estimates of race/ethnicity and income effects on Internet use and search behaviors were derived from generalized estimating equations. Results show wide gaps in the use of computers between Hispanics and Whites (OR = 0.593 [0.440, 0.798]) and between African-Americans and Whites (OR = 0.554 [0.427, 0.720]) in 2000 significantly narrowed in 2002 (OR of Hispanic to white = 1.250 [0.874, 1.789]; OR of African-American to Whites = (0.793 [0.551, 1.141]). Gaps in access to the Internet, however, remained consistent between 2000–2002. Differences in health information seeking between Hispanics and Whites existed in both 2000 and 2002. 56% of White Internet users at some time searched for online health information, whereas 42% of Hispanic Internet users did so in 2000. By 2002, these percentages had increased to 13.4 and 15.8%, respectively. Data highlight the persistence of “Digitally Underserved Groups,” despite recent Divide reduction strategies.     

Sociotechnical challenges to developing technologies for patient access to health information exchange data

Background

Providing patients with access to their medical data is widely expected to help educate and empower them to manage their own health. Health information exchange (HIE) infrastructures could potentially help patients access records across multiple healthcare providers. We studied three HIE organizations as they developed portals to give consumers access to HIE data previously exchanged only among healthcare organizations.

Objective

To follow the development of new consumer portal technologies, and to identify barriers and facilitators to patient access to HIE data.

Methods

Semistructured interviews of 15 key informants over a 2-year period spanning the development and early implementation of three new projects, coded according to a sociotechnical framework.

Results

As the organizations tried to develop functionality that fully served the needs of both providers and patients, plans were altered by technical barriers (primarily related to data standardization) and cultural and legal issues surrounding data access. Organizational changes also played an important role in altering project plans. In all three cases, patient access to data was significantly scaled back from initial plans.

Conclusions

This prospective study revealed how sociotechnical factors previously identified as important in health information technology success and failure helped to shape the evolution of three novel consumer informatics projects. Barriers to providing patients with seamless access to their HIE data were multifactorial. Remedies will have to address technical, organizational, cultural, and other factors.     

A personally controlled electronic health record for Australia

Objective

On July 1, 2012 Australia launched a personally controlled electronic health record (PCEHR) designed around the needs of consumers. Using a distributed model and leveraging key component national eHealth infrastructure, the PCEHR is designed to enable sharing of any health information about a patient with them and any other health practitioner involved in their care to whom the patient allows access. This paper discusses the consumer-facing part of the program.

Method

Design of the system was through stakeholder consultation and the development of detailed requirements, followed by clinical design assurance.

Results

Patients are able to access any posted information through a web-accessible ‘consumer portal.’ Within the portal they are able to assert access controls on all or part of their record. The portal includes areas for consumers to record their own personal information.

Discussion

The PCEHR has the potential to transform the ability of patients to actively engage in their own healthcare, and to enable the emerging partnership model of health and healthcare in medicine. The ability to access health information traditionally kept within the closed walls of institutions also raises challenges for the profession, both in the language clinicians choose and the ethical issues raised by the changed roles and responsibilities.

Conclusions

The PCEHR is aimed at connecting all participants and their interventions, and is intended to become a system-wide activity.     

基于特征分析的公共健康网络信息资源获取策略研究

基于公共健康网络信息资源的分布特征和内容特征，分析其获取存在的障碍，针对性地提出改进策略，包括利用一站式平台、专业搜索引擎、公益性健康信息咨询机构、关注卫生机构和地方政府健康信息公告等方面。     

获取网上医学英文期刊免费全文的方法

介绍获取网上医学英文期刊免费全文的5种方法。它们是因特网免费全文期刊信息的网站，医学搜索引擎，期刊主办机构或出版机构的网站，搜索引擎查出期刊的网络版或电子版的网址，网络上的信息服务机构如图书馆对电子期刊的收集、整理、链接等。     

A Unified Architecture for Biomedical Search Engines Based on Semantic Web Technologies

There is a huge growth in the volume of published biomedical research in recent years. Many medical search engines are designed and developed to address the over growing information needs of biomedical experts and curators. Significant progress has been made in utilizing the knowledge embedded in medical ontologies and controlled vocabularies to assist these engines. However, the lack of common architecture for utilized ontologies and overall retrieval process, hampers evaluating different search engines and interoperability between them under unified conditions. In this paper, a unified architecture for medical search engines is introduced. Proposed model contains standard schemas declared in semantic web languages for ontologies and documents used by search engines. Unified models for annotation and retrieval processes are other parts of introduced architecture. A sample search engine is also designed and implemented based on the proposed architecture in this paper. The search engine is evaluated using two test collections and results are reported in terms of precision vs. recall and mean average precision for different approaches used by this search engine.     

Community-based health insurance schemes in India: a review

There is an increasing inclination among multinational agencies--including the World Bank, World Health Organization and International Labour Organization--to advocate community-based health insurance (CBHI) schemes as part of a comprehensive solution to improving access for healthcare services in India. This paper reviews the experience of Indian CBHI schemes, their impact on health system goals, such as access to hospitalization and protection from indebtedness, and the factors--particularly scheme design and management--that may contribute to success. The CBHI schemes in India are extremely diverse in terms of their designs, sizes and target populations. While some of the schemes are run by non-governmental organization (NGO) providers, there is an increasing trend towards collaboration with the Government Insurance Company (GIC). In its partnership with NGOs, the GIC seems to have provided favourable group plans compared to the individual Mediclaim and Jan Arogya policies. We have little information on the impact of existing CBHI schemes--most importantly, in terms of access and protection from indebtedness--and even less on factors that make for a successful scheme. This review suggests that there is a demand for health insurance services among the poor. To date, there is little evidence to suggest that these schemes can include the poorest of the poor or improve access to inpatient care. Furthermore, the schemes have done little to address the issue of low/variable quality of healthcare services. Empirically derived data on the existing schemes in India are extremely limited, making this fertile ground for future research.     

Seeking Health Information Online: Does Wikipedia Matter?

Objective

To determine the significance of the English Wikipedia as a source of online health information.

Design

The authors measured Wikipedia's ranking on general Internet search engines by entering keywords from MedlinePlus, NHS Direct Online, and the National Organization of Rare Diseases as queries into search engine optimization software. We assessed whether article quality influenced this ranking. The authors tested whether traffic to Wikipedia coincided with epidemiological trends and news of emerging health concerns, and how it compares to MedlinePlus.

Measurements

Cumulative incidence and average position of Wikipedia^® compared to other Web sites among the first 20 results on general Internet search engines (Google^®, Google UK^®, Yahoo^®, and MSN^®), and page view statistics for selected Wikipedia articles and MedlinePlus pages.

Results

Wikipedia ranked among the first ten results in 71-85% of search engines and keywords tested. Wikipedia surpassed MedlinePlus and NHS Direct Online (except for queries from the latter on Google UK), and ranked higher with quality articles. Wikipedia ranked highest for rare diseases, although its incidence in several categories decreased. Page views increased parallel to the occurrence of 20 seasonal disorders and news of three emerging health concerns. Wikipedia articles were viewed more often than MedlinePlus Topic (p = 0.001) but for MedlinePlus Encyclopedia pages, the trend was not significant (p = 0.07-0.10).

Conclusions

Based on its search engine ranking and page view statistics, the English Wikipedia is a prominent source of online health information compared to the other online health information providers studied.     

Triggers and Monitoring in Intelligent Personal Health Record

Although Web-based personal health records (PHRs) have been widely deployed, the existing ones have limited intelligence. Previously, we introduced expert system technology and Web search technology into the PHR domain and proposed the concept of an intelligent PHR (iPHR). iPHR provides personalized healthcare information to facilitate users' daily activities of living. The current iPHR is passive and follows the pull model of information distribution. This paper introduces triggers and monitoring into iPHR to make iPHR become active. Our idea is to let medical professionals pre-compile triggers and store them in iPHR's knowledge base. Each trigger corresponds to an abnormal event that may have potential medical impact. iPHR keeps collecting, processing, and analyzing the user's medical data from various sources such as wearable sensors. Whenever an abnormal event is detected from the user's medical data, the corresponding trigger fires and the related personalized healthcare information is pushed to the user using natural language generation technology, expert system technology, and Web search technology.