通用搜索引擎与医学搜索引擎的比较研究

设计有代表性的检索提问,将搜索引擎分为A组（医学搜索引擎,n=9),B组（通用搜索引擎,n＝9）,用SPSS8．0统计软件对检索结果进行分析;文献计量学方法用于分析“发热处理”检索结果。通过比较,明确通用搜索引擎和医学搜索引擎在检索不同提问时获得网页内容相关性,网页重复,新颖性及无效链接等方面有无显著性差异;从用户角度对两组搜索引擎查寻专题信息“发热处理”的检索结果进行分析评价。结果表明,医学搜索引擎检出结果的内容相关性不如通用搜索引擎,无效链接百比高于通用搜索引擎;索及网页的更新时效性比通用搜索引擎强,网页重复率较通用搜索引擎低;医学搜索引擎提供“发热处理”信息的质量和完整性不如通用搜索引擎;二者提供的有效信息均较少。以上结果提示,多种搜索引擎联合检索有助于获得较多的有用信息;医学搜索引擎及网页的质量,数据库规模有待改进和提高。     

Tracing medical information over the Internet

The Internet became with do doubt a huge and valuable source of information for researchers. The wealth of information on the Internet is second to none and medical information is no exception. Yet with the vast expansion of the Internet and the World Wide Web in specie, to find the kind of information one is looking for, he/she needs to browse thousands of web sites and the experience would be like digging into a stack of hay looking for a needle. That's why search engines and subject indexes, as means to overcome this problem, were introduced and grew so rapidly. In general, there are three approaches to retrieve data from the World Wide Web; the subject directories, search engines and detailed subject indexes. However, there is no single comprehensive search engine or directory and it is recommended to use more than one with different keywords and synonymous.     

合肥市社区居民对互联网健康信息的使用状况调查

目的：了解社区居民对互联网健康信息的使用现状及互联网健康信息的使用与健康行为之间的联系。方法：采用分层整群抽样的方法,在安徽省合肥市7个区县中抽取10个社区,每个社区选择40户社区居民,采用调查问卷对社区居民获取互联网健康信息的途径、主要关注的健康信息类型、对互联网健康信息的信任程度、互联网健康信息对健康行为的影响进行调查。结果：83．9％的被调查者会经常使用网络来查看健康与疾病相关信息;网页和搜索引擎为社区居民获取互联网健康信息的主要方式;随年龄增长,会有更多的人通过网络获取健康信息;食品安全、养生保健、疾病信息是社区居民感兴趣的健康信息;49．6％的人对互联网上健康信息的真实性持中立态度,35．2％的人比较相信,13．6％的人不太相信;53．0％的人认为健康信息会对其健康行为产生影响。结论：互联网已成为社区居民获取健康信息的一种重要途径,多数人对互联网上的健康信息持较为信任的态度,互联网上的健康信息会对部分社区居民的健康行为产生一定影响。     

Assessing Domains of Uncertainty in Critical Web-Based Healthcare Communities

Search engines and Web-based directories play a central role in facilitating access to online health information. In this environment consumers often find and act upon health information of variable quality, with little input from health professionals. As an added concern, when consumers access health information on the Web via a search engine they often fail to discuss the information with a healthcare provider, especially in areas of perceived medical uncertainty. For many health consumers, the Internet may be the primary (or even sole) source of health information, especially where consumers lack access to, or trust in, trained providers. To date the corresponding role of search engine technology use and efficacy has received relatively little attention, however. This study serves as an exploratory technology assessment that explains the application of keyword effectiveness indexing (KEI) analysis in estimating the ability of commercial search engines to return relevant results for healthcare consumers. Findings suggest that KEI can serve as a useful health search profiling tool, though its applicability may vary across domains. This study was provided support from the Center for Technology Assessment, State College, PA.     

MIRASS: Medical Informatics Research Activity Support System Using Information Mashup Network

The advancement of information technology has facilitated the automation and feasibility of online information sharing. The second generation of the World Wide Web (Web 2.0) enables the collaboration and sharing of online information through Web-serving applications. Data mashup, which is considered a Web 2.0 platform, plays an important role in information and communication technology applications. However, few ideas have been transformed into education and research domains, particularly in medical informatics. The creation of a friendly environment for medical informatics research requires the removal of certain obstacles in terms of search time, resource credibility, and search result accuracy. This paper considers three glitches that researchers encounter in medical informatics research; these glitches include the quality of papers obtained from scientific search engines (particularly, Web of Science and Science Direct), the quality of articles from the indices of these search engines, and the customizability and flexibility of these search engines. A customizable search engine for trusted resources of medical informatics was developed and implemented through data mashup. Results show that the proposed search engine improves the usability of scientific search engines for medical informatics. Pipe search engine was found to be more efficient than other engines.     

Content and design attributes of antivaccination web sites

Context  Individuals searching the Internet for vaccine information may find antivaccination Web sites. Few published studies have systematically evaluated these sites. Objectives  To examine antivaccination Web site attributes and to delineate the specific claims and concerns expressed by antivaccination groups. Design and Setting  In late 2000, using a metasearch program that incorporates 10 other search engines, we reviewed and analyzed 772 links to find 12 Web sites that promulgated antivaccination information. Analyzing links from these 12 sites yielded another 10 sites, producing a total of 22 sites for study. Using a standardized form, 2 authors (R.M.W., L.K.S.) systematically evaluated these sites based on specific content and design attributes. Main Outcome Measures  Presence or absence of 11 Web site content attributes (antivaccination claims) and 10 Web site design attributes. Results  The most commonly found content claims were that vaccines cause idiopathic illness (100% of sites), vaccines erode immunity (95%), adverse vaccine reactions are underreported (95%), and vaccination policy is motivated by profit (91%). The most common design attributes were the presence of links to other antivaccination sites (100%of sites), information for legally avoiding immunizations (64%), and the use of emotionally charged stories of children who had allegedly been killed or harmed by vaccines (55%). Conclusion  Antivaccination Web sites express a range of concerns related to vaccine safety and varying levels of distrust in medicine. The sites rely heavily on emotional appeal to convey their message.      

Use of the Web for medical information by a gastroenterology clinic population

CONTEXT: Surveys have shown that 60 million persons in the United States searched for health information online in 1998. However, lack of sampling from a clinic population limits the generalizability of these surveys to clinical practice. OBJECTIVES: To determine gastroenterology patients' access to and use of the Web as a medical information resource, to identify for what information patients search, and to determine how often physicians recommend that patients search the Web. DESIGN, SETTING, AND PARTICIPANTS: Cross-sectional survey of 1006 gastroenterology outpatients in Durham, NC, and Rockford, Ill, conducted in August 1999. MAIN OUTCOME MEASURES: Patient characteristics and education level, access to the Web, use of the Web as a medical information resource, search methods, and plans for future Web use. RESULTS: A total of 924 patients (92%) completed the questionnaire. Median age was 53 years, 41% were men, and the median education level was having completed some college. Fifty percent (462/924) reported having access to the Web. Of the 462 with access, 235 (51%) had searched the Web for medical information within the previous 12 months. Therefore, 25.5% of all patients surveyed had searched the Web for medical information within the previous year. Sixty percent of patients intended to use the Web as a medical information resource in the future. Only 35 (4%) of 825 had ever been referred to the Web by a physician. CONCLUSIONS: In this clinic setting, more than one quarter of gastroenterology outpatients reported having obtained medical information from the Web within the previous year. More than two thirds of patients stated they would use the Web as a medical information resource in the future. JAMA. 2000;284:1962-1964.     

Bat-Inspired Algorithm Based Query Expansion for Medical Web Information Retrieval

With the increasing amount of medical data available on the Web, looking for health information has become one of the most widely searched topics on the Internet. Patients and people of several backgrounds are now using Web search engines to acquire medical information, including information about a specific disease, medical treatment or professional advice. Nonetheless, due to a lack of medical knowledge, many laypeople have difficulties in forming appropriate queries to articulate their inquiries, which deem their search queries to be imprecise due the use of unclear keywords. The use of these ambiguous and vague queries to describe the patients’ needs has resulted in a failure of Web search engines to retrieve accurate and relevant information. One of the most natural and promising method to overcome this drawback is Query Expansion. In this paper, an original approach based on Bat Algorithm is proposed to improve the retrieval effectiveness of query expansion in medical field. In contrast to the existing literature, the proposed approach uses Bat Algorithm to find the best expanded query among a set of expanded query candidates, while maintaining low computational complexity. Moreover, this new approach allows the determination of the length of the expanded query empirically. Numerical results on MEDLINE, the on-line medical information database, show that the proposed approach is more effective and efficient compared to the baseline.     

Seeking Health Information Online: Does Wikipedia Matter?

Objective

To determine the significance of the English Wikipedia as a source of online health information.

Design

The authors measured Wikipedia's ranking on general Internet search engines by entering keywords from MedlinePlus, NHS Direct Online, and the National Organization of Rare Diseases as queries into search engine optimization software. We assessed whether article quality influenced this ranking. The authors tested whether traffic to Wikipedia coincided with epidemiological trends and news of emerging health concerns, and how it compares to MedlinePlus.

Measurements

Cumulative incidence and average position of Wikipedia^® compared to other Web sites among the first 20 results on general Internet search engines (Google^®, Google UK^®, Yahoo^®, and MSN^®), and page view statistics for selected Wikipedia articles and MedlinePlus pages.

Results

Wikipedia ranked among the first ten results in 71-85% of search engines and keywords tested. Wikipedia surpassed MedlinePlus and NHS Direct Online (except for queries from the latter on Google UK), and ranked higher with quality articles. Wikipedia ranked highest for rare diseases, although its incidence in several categories decreased. Page views increased parallel to the occurrence of 20 seasonal disorders and news of three emerging health concerns. Wikipedia articles were viewed more often than MedlinePlus Topic (p = 0.001) but for MedlinePlus Encyclopedia pages, the trend was not significant (p = 0.07-0.10).

Conclusions

Based on its search engine ranking and page view statistics, the English Wikipedia is a prominent source of online health information compared to the other online health information providers studied.     

A critical review of nutrition resources for General Practitioners focusing on healthy diet, including seafood

Background

General practitioners (GPs) are considered a trusted and reliable source of health-related information including nutritional advice. Preliminary investigation found that GPs wanted evidence-based nutrition resources that could be used within a 10 minute consultation.

Aim

The aim of the study was to identify and critically review current resources available to GPs that promote seafood consumption within a healthy diet, as a preventative or treatment measure for common lifestyle or medical conditions.

Methods

English language resources currently available to GPs in 2008 were sourced through multiple avenues including: individual organisations; medical service networks; health information services and internet search engines. Assessment included critical review of: format; appropriateness for target groups; reference to seafood and supporting evidence; credibility; readability; and suitability for use by practitioners in a short consultation.

Results

One hundred and twenty resources were identified. The majority (88.4%, n=106) of identified resource were available Electronically. Just over half (57.5%, n=69) of the resources were targeted at specific audiences. All of the resources made reference to the health benefits of regular consumption of fish (100%, n=120), 22.5% (n=27) made reference to seafood in general and 5% (n=6) made reference to fish oil. Only 15% (n=18) of the identified resources were suitable for use with the general Australian population at or below the recommended reading level of Year Eight. The majority (87.5%, n=105) of the identified resources were associated with credible sources of information about the health benefits of regular consumption of seafood.

Conclusions

This study found that the majority of resources available to GPs were not suitable for use with the general Australian population at the recommended reading level of Year 8 or lower. Whilst it is acknowledged that written health information alone cannot change health behaviours, it can provide accurate information to assist in making changes to behaviours with support from appropriate health care professionals.     

应对突发公共卫生事件的医学信息服务工作探析

结合工作实践，提出建立应对突发公共卫生事件的医学信息服务保障体系，包括加强人才与信息资源的建设，健全信息服务网络，确保网络的畅通，建立馆藏目录数据库，配置必要的现代化设备，建立与所在地的医学院校、医院、疾控中心的疫情信息联系反馈通道，开展相关课题研究；同时，整合、编辑专题性题录、文献摘要、网页专题；运用计算机网络技术，开发利用相关信息，如联网数据库、网络搜索引擎、相关网站，使其在应对突发公共卫生事件过程中发挥最大效用。     

网络获取技术在循证护理信息资源检索中的应用研究

目的 加强循证护理信息资源的获取与利用，促进循证护理的发展。方法 文献复习与信息资源检索。结果生物医学文献数据库、搜索引擎、元搜索引擎、电子期刊、重要网站都能获取网络循证护理信息资源。结论 生物医学文献数据库、搜索引擎、元搜索引擎、电子期刊、重要网站都是获取循证护理信息资源的非常重要的途径，传授这些信息资源获取的方法和技巧，是医学信息人员的重要责任。加强护理人员与医学图书情报人员的合作将有利于推动循证护理实践的开展。     

城市社区慢性非传染性疾病综合防控建设示范

目的通过城市慢性非传染性疾病综合防治示范社区建设探索以社区为平台,社区卫生服务机构适宜技术用于控制城市社区慢性非传染性疾病发生发展,促进社区参与和提高健康管理水平。方法利用社区卫生服务可及性、连续性、家庭医生服务模式,以社区为基础的一级教育、二级随访、三级治疗分级干预流程和网络;进行信息化条件下社区病例综合管理。结果50～60岁中老年人健康体检结论回顾性评价居民生物危险因素降低了2．87％,其中高血压发生降低了10．99％：高血糖发生降低了2．48％;高血压、糖尿病慢性病新发病例筛查登记下降了17．1％;问卷调查自我血压水平知晓率达到80．4％。结论慢性病的发生和快速发展有效控制除环境因素和生活方式改变外,基层社区卫生服务机构的干预能力是重要的因素,社区的参与配合能够使慢病防控工作做到实处。     

Assessing the quality,suitability and readability of internet-based health information about warfarin for patients

Background

Warfarin is a high-risk medication where patient information may be critical to help ensure safe and effective treatment. Considering the time constraints of healthcare providers, the internet can be an important supplementary information resource for patients prescribed warfarin. The usefulness of internet-based patient information is often limited by challenges associated with finding valid and reliable health information. Given patients'' increasing access of the internet for information, this study investigated the quality, suitability and readability of patient information about warfarin presented on the internet.

Method

Previously validated tools were used to evaluate the quality, suitability and readability of patient information about warfarin on selected websites.

Results

The initial search yielded 200 websites, of which 11 fit selection criteria, comprising seven non-commercial and four commercial websites. Regarding quality, most of the non-commercial sites (six out of seven) scored at least an ‘adequate’ score. With regard to suitability, 6 of the 11 websites (including two of the four commercial sites) attained an ‘adequate’ score. It was determined that information on 7 of the 11 sites (including two commercial sites) was written at reading grade levels beyond that considered representative of the adult patient population with poor literacy skills (e.g. school grade 8 or less).

Conclusion

Despite the overall ‘adequate’ quality and suitability of the internet derived patient information about warfarin, the actual usability of such websites may be limited due to their poor readability grades, particularly in patients with low literacy skills.     

Improving breast health education for Hispanic women

Hispanic women are less likely to receive screening mammograms than are white or African American women, even though breast cancer is the leading cause of cancer-related deaths in the population. Between 1990 and 2000, the Hispanic population of Indianapolis, Indiana (Marion County) grew by approximately 300%: from 8450 to 33 290. The present project was undertaken to provide educational outreach in Spanish to the new Hispanic residents of Indianapolis. A bilingual outreach coordinator visited community centers, churches, and health clinics and presented information to Hispanic women and men about a variety of cancers for which Hispanic women are at particularly high risk, including breast and cervical cancer. The number of Hispanic women obtaining mammograms rose by more than 200% during the first 6 months of the program. The authors conclude that culturally appropriate educational outreach presented by a Spanish-speaking woman using a Spanish-language videotape about mammography can improve breast cancer screening in Latinas.     

A User-centered Model for Web Site Design: Needs Assessment, User Interface Design, and Rapid Prototyping

As the Internet continues to grow as a delivery medium for health information, the design of effective Web sites becomes increasingly important. In this paper, the authors provide an overview of one effective model for Web site design, a user-centered process that includes techniques for needs assessment, goal/task analysis, user interface design, and rapid prototyping. They detail how this approach was employed to design a family health history Web site, Health Heritage <www.healthheritage.net>. This Web site helps patients record and maintain their family health histories in a secure, confidential manner. It also supports primary care physicians through analysis of health histories, identification of potential risks, and provision of health care recommendations. Visual examples of the design process are provided to show how the use of this model resulted in an easy-to-use Web site that is likely to meet user needs. The model is effective across diverse content arenas and is appropriate for applications in varied media.The design of effective health-related Web sites is becoming increasingly important as the Internet continues to grow as a delivery medium for health information. As these sites are developed, the importance of a user-centered approach to design is recognized by informaticians:

Groups producing information materials must start with needs defined by patients, give treatment information based on rigorous systematic reviews, and involve multidisciplinary teams (including patients) in developing and testing the material. ¹

In this paper, we present a user-centered model for this type of Web site design. The model includes techniques for needs assessment, goal/task analysis, user interface design, and rapid prototyping. Each of these techniques can be used to produce effective solutions across multiple content arenas.To show how these techniques may be used, we detail their application in the design of a family health history Web site, Health Heritage (http://www.HealthHeritage.net). Funded by the Robert Wood Johnson Foundation, the purposes of this Web site are:

• To assist patients in recording and maintaining their family health histories in a secure, confidential manner, and
• To support primary care physicians, through analysis of family health histories, identification of potential risks, and provision of preventive or treatment recommendations.

     

A Comparative Evaluation of Full-text,Concept-based,and Context-sensitive Search

Objectives

Study comparatively (1) concept-based search, using documents pre-indexed by a conceptual hierarchy; (2) context-sensitive search, using structured, labeled documents; and (3) traditional full-text search. Hypotheses were: (1) more contexts lead to better retrieval accuracy; and (2) adding concept-based search to the other searches would improve upon their baseline performances.

Design

Use our Vaidurya architecture, for search and retrieval evaluation, of structured documents classified by a conceptual hierarchy, on a clinical guidelines test collection.

Measurements

Precision computed at different levels of recall to assess the contribution of the retrieval methods. Comparisons of precisions done with recall set at 0.5, using t-tests.

Results

Performance increased monotonically with the number of query context elements. Adding context-sensitive elements, mean improvement was 11.1% at recall 0.5. With three contexts, mean query precision was 42% ± 17% (95% confidence interval [CI], 31% to 53%); with two contexts, 32% ± 13% (95% CI, 27% to 38%); and one context, 20% ± 9% (95% CI, 15% to 24%). Adding context-based queries to full-text queries monotonically improved precision beyond the 0.4 level of recall. Mean improvement was 4.5% at recall 0.5. Adding concept-based search to full-text search improved precision to 19.4% at recall 0.5.

Conclusions

The study demonstrated usefulness of concept-based and context-sensitive queries for enhancing the precision of retrieval from a digital library of semi-structured clinical guideline documents. Concept-based searches outperformed free-text queries, especially when baseline precision was low. In general, the more ontological elements used in the query, the greater the resulting precision.     

Using Internet technology to deliver a behavioral weight loss program

CONTEXT: Rapid increases in access to the Internet have made it a viable mode for public health intervention. No controlled studies have evaluated this resource for weight loss. OBJECTIVE: To determine whether a structured Internet behavioral weight loss program produces greater initial weight loss and changes in waist circumference than a weight loss education Web site. DESIGN: Randomized, controlled trial conducted from April to December 1999. SETTING AND PARTICIPANTS: Ninety-one healthy, overweight adult hospital employees aged 18 to 60 years with a body mass index of 25 to 36 kg/m(2). Analyses were performed for the 65 who had complete follow-up data. INTERVENTIONS: Participants were randomly assigned to a 6-month weight loss program of either Internet education (education; n = 32 with complete data) or Internet behavior therapy (behavior therapy; n = 33 with complete data). All participants were given 1 face-to-face group weight loss session and access to a Web site with organized links to Internet weight loss resources. Participants in the behavior therapy group received additional behavioral procedures, including a sequence of 24 weekly behavioral lessons via e-mail, weekly online submission of self-monitoring diaries with individualized therapist feedback via e-mail, and an online bulletin board. MAIN OUTCOME MEASURES: Body weight and waist circumference, measured at 0, 3, and 6 months, compared the 2 intervention groups. RESULTS: Repeated-measures analyses showed that the behavior therapy group lost more weight than the education group (P =.005). The behavior therapy group lost a mean (SD) of 4.0 (2.8) kg by 3 months and 4.1 (4.5) kg by 6 months. Weight loss in the education group was 1.7 (2.7) kg at 3 months and 1.6 (3.3) kg by 6 months. More participants in the behavior therapy than education group achieved the 5% weight loss goal (45% vs 22%; P =.05) by 6 months. Changes in waist circumference were also greater in the behavior therapy group than in the education group at both 3 months (P =.001) and 6 months (P =.005). CONCLUSIONS: Participants who were given a structured behavioral treatment program with weekly contact and individualized feedback had better weight loss compared with those given links to educational Web sites. Thus, the Internet and e-mail appear to be viable methods for delivery of structured behavioral weight loss programs.     

Indicators of Accuracy of Consumer Health Information on the Internet: A Study of Indicators Relating to Information for Managing Fever in Children in the Home

Objectives: To identify indicators of accuracy for consumer health information on the Internet. The results will help lay people distinguish accurate from inaccurate health information on the Internet.Design: Several popular search engines (Yahoo, AltaVista, and Google) were used to find Web pages on the treatment of fever in children. The accuracy and completeness of these Web pages was determined by comparing their content with that of an instrument developed from authoritative sources on treating fever in children. The presence on these Web pages of a number of proposed indicators of accuracy, taken from published guidelines for evaluating the quality of health information on the Internet, was noted.Main Outcome Measures: Correlation between the accuracy of Web pages on treating fever in children and the presence of proposed indicators of accuracy on these pages. Likelihood ratios for the presence (and absence) of these proposed indicators.Results: One hundred Web pages were identified and characterized as “more accurate” or “less accurate.” Three indicators correlated with accuracy: displaying the HONcode logo, having an organization domain, and displaying a copyright. Many proposed indicators taken from published guidelines did not correlate with accuracy (e.g., the author being identified and the author having medical credentials) or inaccuracy (e.g., lack of currency and advertising).Conclusions: This method provides a systematic way of identifying indicators that are correlated with the accuracy (or inaccuracy) of health information on the Internet. Three such indicators have been identified in this study. Identifying such indicators and informing the providers and consumers of health information about them would be valuable for public health care.Millions of people now use the Internet to gather health information.¹ It is important for the health and well-being of these people, and for the societies to which they belong, that the information they acquire in this way be, on the whole, accurate.² Much of the consumer health information on the Internet is accurate. However, studies have shown that some of this information is not accurate.^3–6 The present study is part of a project to help lay people distinguish accurate from inaccurate consumer health information on the Internet.^7,8In a recent study, Impicciatore et al.⁴ looked at the accuracy of information on the treatment of fever in children. According to them, “only a few of the [41] Web pages we reviewed gave complete and accurate information for such a common and widely discussed condition.” Thus, Impicciatore et al. established that there is a problem with inaccurate consumer health information on the Internet.To help Internet users to avoid this inaccurate information, a number of authors and organizations have published guidelines for evaluating the quality of health information on the Internet.^9–11 These guidelines typically include lists of indicators that are intended to help Internet users determine the accuracy of Web sites. For example, the medical credentials of the author of a Web site are supposed to be an indicator of accuracy. Also, an out-of-date Web site and a lack of advertising on a Web site are supposed to be indicators of inaccuracy.Of course, these guidelines can only help Internet users to avoid inaccurate information if the indicators really are correlated with accuracy (or inaccuracy). Unfortunately, there is currently no empirical data to support the claim that these indicators are correlated with accuracy (or inaccuracy). In fact, a recent study¹² casts doubt on the reliability of at least one of the proposed indicators of accuracy (namely, positive ratings from services that evaluate medical information on the Internet).The aim of the current study is to test empirically several of the proposed indicators of accuracy. Following Impicciatore et al., we looked at Web sites that discuss the treatment of fever in children. However, in addition to assessing the accuracy of these sites, we looked for the presence of several of the proposed indicators. With this data (which were collected from March to November 2000), we were able to determine that some of the proposed indicators are correlated with accuracy, but many are not.     

An evaluation of two commercial deep learning-based information retrieval systems for COVID-19 literature

The COVID-19 pandemic has resulted in a tremendous need for access to the latest scientific information, leading to both corpora for COVID-19 literature and search engines to query such data. While most search engine research is performed in academia with rigorous evaluation, major commercial companies dominate the web search market. Thus, it is expected that commercial pandemic-specific search engines will gain much higher traction than academic alternatives, leading to questions about the empirical performance of these tools. This paper seeks to empirically evaluate two commercial search engines for COVID-19 (Google and Amazon) in comparison with academic prototypes evaluated in the TREC-COVID task. We performed several steps to reduce bias in the manual judgments to ensure a fair comparison of all systems. We find the commercial search engines sizably underperformed those evaluated under TREC-COVID. This has implications for trust in popular health search engines and developing biomedical search engines for future health crises.